SmartBody - a character animation system for Ogre

[arishapiro]

All,



Update 11/26/13
We now have downloadable SDK versions for Windows, Mac and Linux platforms here for download using the LGPLv3 license:
http://sourceforge.net/projects/smartbody/files/



My team has been developing the SmartBody character animation system for a number of years at the Institute of Creative Technologies, part of the University of Southern California. The goal is to create an extensive set of capabilities that are used by most virtual characters.

SmartBody provides the following capabilities:
* locomotion
* steering
* gazing
* lip synchronization to speech
* reaching/grasping/touching/pointing
* gesturing
* head movements
* blinking and eye saccades
* breathing
* facial animation
* physics
* constraints
* animation blending
* online retargeting
* autorigging
* autoskinning

In addition, we include an automated retargeting capability to transfer data automatically to any skeleton.
SmartBody is written in portable C++, runs on Windows, Linux and OSx, as well as Android and iOS platforms.
The system can be controlled or configured either via C++ or through our Python API.
The license is LGPL. The system is being actively supported and developed and is hosted at SourceForge.

The SmartBody website is located at:
http://smartbody.ict.usc.edu

and I have posted a number of videos showing SmartBody capabilities on YouTube. (http://www.youtube.com/playlist?list=PL38CB86DA1BC151F7)

Here are a few of interest:

Example-based locomotion: http://www.youtube.com/watch?feature=pl ... DA1BC151F7
Gazing: http://www.youtube.com/watch?feature=pl ... DA1BC151F7
Automated speech, gesture and head movements: http://www.youtube.com/watch?feature=pl ... DA1BC151F7
Constraints: http://www.youtube.com/watch?feature=pl ... DA1BC151F7
Eye saccades: http://www.youtube.com/watch?feature=pl ... DA1BC151F7
Interactive reaching and grasping: http://www.youtube.com/watch?feature=pl ... DA1BC151F7
Full body IK: http://www.youtube.com/watch?feature=pl ... DA1BC151F7
Lip syncing: http://smartbody.ict.usc.edu/HTML/video ... IG2013.mp4
Pointing: http://smartbody.ict.usc.edu/other/pointing.mp4

Ogre was the first game engine that we integrated. We are happy to share our code and capability with the Ogre community.
Please feel free to post in our forum:

http://smartbody.ict.usc.edu/forum

or to contact me directly at:

shapiro@ict.usc.edu

if you have any questions about SmartBody.

Regards,

Ari Shapiro, Ph.D.
Research Scientist
Institute for Creative Technologies

[Jabberwocky]

Wow. Extremely nice work. That's an incredibly useful feature list.

[bstone]

Awesome.

[areay]

Wow Ari, that's really great. I watched all the vids on your youtube channel and I gotta say that it looks spectacular. The clips that impressed me the most were the 'hands on table constraint', 'gesturing' and 'retargeting locomotion', also, the 'pertubation' vid made me lol.

Haven't looked at your code yet but I imagine the complexity is staggering based on what you've achieved.

Huge kudos for releasing it LGPL too, and a big congratulations to your team.

[Mind Calamity]

When I saw this - my reaction was: This is going to be way out of my reach... And then I checked out your website where I was pleasantly surprised by your licensing.

You guys did an amazingly good job at this system, and I'm sure glad you decided to support Ogre as well.

[drwbns]

Amazing work, wow

[f00bar]

Wow, what a first post
This looks like it could save me a huge truckload of work. Building now...
Thanks a lot.

[DavlexDesign]

Now that's a first post !!!!

I've been toying around with Mocap driven ragdoll constraints for something similar, but looks like this will help allot.
Kudos to you guys for this.

Alex

[Transporter]

Have a look at the CMU Graphics Lab Motion Capture Database for free motion capturings. Could be useful for this interesting tool.

[TheSHEEEP]

This is really awesome.

Where can we see that Ogre integration? I mean the actual code of it. Is it part of the source code?

[arishapiro]

This is really awesome.

Where can we see that Ogre integration? I mean the actual code of it. Is it part of the source code?

As part of the distribution, there is a project called OgreViewer, which runs SmartBody using Ogre and the SmartBody DLL. Specifically, you can look at the SBListener.h/cpp and the OgreFrameListener.cpp:FrameStarted()
I have also included some instructions for integration in the documentation as well:

http://smartbody.ict.usc.edu/documentat ... Manual.pdf

The integration is mainly a matter of setting up the SmartBody world, updating the step, copying the SmartBody character state values onto the corresponding Ogre character, and responding to a few event callbacks (character created, deleted, etc.)

The current DLL offers a simplified interface that allows you to access state information via C++, and the rest of the internal aspects of SmartBody via embedded Python. I have a richer interface that allows direct access to all the SmartBody internal structures via C++, but I haven't yet packaged it in to a dynamic library yet. If you wanted to use that interface, you would have to link directly to the source code.

Please let me know if you think there is a way to make this more accessible to the Ogre community. My intention was to create the equivalent for character animation what Ogre has done for rendering engines. I'd be happy to have my team spend some time to make it easy for Ogre users to experiment with it.

Regards,

Ari Shapiro

[f00bar]

[Edit:] sorry, missed Ari's post.

Where can we see that Ogre integration? I mean the actual code of it. Is it part of the source code?

Look at their site or the SVN. Complete ogre is contained in their SVN. "Running SmartBody in the Ogre Game Engine" is a chapter in the included manual , ogre example application is also provided. I did not test the ogre stuff till now (still exploring the default GUI), but it looks like they just transfer bone data to the skeleton with the python API, so I assume you can use all the features with just some lines of integration code from the sample. Of course tight integration with your own game world, physics and events is another story, but engine dependent anyway.

There are two different C/C++ APIs that can be used: a simplified interface where instructions to SmartBody are sent as
Python commands via the VHMessage system (smarbody-dll) and a comprehensive API that allows you to access, query,
and change all of SmartBody's internal structures (smarbody-lib). In addition, there is way to integrate SmartBody into a
game engine using a network protocol called BoneBus which is language-independent.

You can also build an android and IPhone example application. As OpenGL version or as ogre based version!

SmartBody can be run in a number of different ways:
As a standalone application
Embedded within a game engine such as Ogre or Unity
On an Android or iPhone mobile device
Embedded as a Python library

[Blender+C++]

Trully awesomeness!!!!
the Ogre community appreciates that

[f00bar]

Hi Ari,

So far it works great. I would have some questions, only one ogre related though:

1) Is there a specific license for the data (models, skeletons, animation data, behaviors, ...)? Is it something free like CC0 or CC-by?

2) I open default scene in sbm-fltk (I must always open default.py by hand, thought it should auto load). Then I attach default OgreViewer by BoneBus. I only see character brad, once for doctor and once for elder, real brad and utah are missing in ogre.
I get log messages like this:
16:54:37: OGRE EXCEPTION(5:ItemIdentityException): Child node named 254578270 does not exist. in Node::getChild at ..\..\..\..\OgreMain\src\OgreNode.cpp (line 733)
(and unrelated:)
16:54:30: OGRE EXCEPTION(5:ItemIdentityException): OverlayElement with name Core/AverageFps not found. in OverlayManager::getOverlayElementImpl at ..\..\..\..\OgreMain\src\OgreOverlayManager.cpp (line 621)

From your forum I understood that all 4 characters are available as ogre models. Is this normal at the moment or is something wrong with my build?

3) Is walking on terrain working (in sbm-fltk)? I tried like described here: http://www.mail-archive.com/smartbody-s ... 03746.html
setBoolAttribute("steering.terrainMode", True)
and set the following for all blends:
blend.setIncrementWorldOffsetY(True)

Also enabled collision system and tried some random character settings. But terrain seems to be ignored.

4) It looks like the characters can get stuck in certain behaviors. I assume this is because I send it commands that do not mix well (like stacking stuff that does not end automatically) or me setting unreasonable parameters (just playing around). Is there an easy way to reset a characters behavior and remove gazes/constraints/reaches or whatever I did to break it? (resetting to PseudoIdle in BlendViewer->RunTimeEditor does not help)

5) Can we assume that the default.py of sbm-fltk will load the best and most stable behavior config currently available? Or are there specific examples that show specific features best?

The only complaint with the current default is that it uses the single step animation for distances that are too large for it so it looks odd (if walking 1-2m from idle). For me more than 2 single steps in a row look very odd. I think it would look much better to allow at most 1 single step. That would also require backwards continuous locomotion animation which I did not find in the example.


Thanks again for releasing this and the support. Very much appreciated. Sorry if some of these are obvious, sill overwhelmed by it.

[arishapiro]

Hi Ari,

So far it works great. I would have some questions, only one ogre related though:

1) Is there a specific license for the data (models, skeletons, animation data, behaviors, ...)? Is it something free like CC0 or CC-by?

2) I open default scene in sbm-fltk (I must always open default.py by hand, thought it should auto load). Then I attach default OgreViewer by BoneBus. I only see character brad, once for doctor and once for elder, real brad and utah are missing in ogre.
I get log messages like this:
16:54:37: OGRE EXCEPTION(5:ItemIdentityException): Child node named 254578270 does not exist. in Node::getChild at ..\..\..\..\OgreMain\src\OgreNode.cpp (line 733)
(and unrelated:)
16:54:30: OGRE EXCEPTION(5:ItemIdentityException): OverlayElement with name Core/AverageFps not found. in OverlayManager::getOverlayElementImpl at ..\..\..\..\OgreMain\src\OgreOverlayManager.cpp (line 621)

From your forum I understood that all 4 characters are available as ogre models. Is this normal at the moment or is something wrong with my build?

3) Is walking on terrain working (in sbm-fltk)? I tried like described here: http://www.mail-archive.com/smartbody-s ... 03746.html
setBoolAttribute("steering.terrainMode", True)
and set the following for all blends:
blend.setIncrementWorldOffsetY(True)

Also enabled collision system and tried some random character settings. But terrain seems to be ignored.

4) It looks like the characters can get stuck in certain behaviors. I assume this is because I send it commands that do not mix well (like stacking stuff that does not end automatically) or me setting unreasonable parameters (just playing around). Is there an easy way to reset a characters behavior and remove gazes/constraints/reaches or whatever I did to break it? (resetting to PseudoIdle in BlendViewer->RunTimeEditor does not help)

5) Can we assume that the default.py of sbm-fltk will load the best and most stable behavior config currently available? Or are there specific examples that show specific features best?

The only complaint with the current default is that it uses the single step animation for distances that are too large for it so it looks odd (if walking 1-2m from idle). For me more than 2 single steps in a row look very odd. I think it would look much better to allow at most 1 single step. That would also require backwards continuous locomotion animation which I did not find in the example.


Thanks again for releasing this and the support. Very much appreciated. Sorry if some of these are obvious, sill overwhelmed by it.

1) Regarding the use of the data outside of SmartBody, that's a good question. Let me get back to you about that.

2) I just checked in a fix for the BoneBus mode as you describe (r3799) - you should see all four models now.

3) We've been playing around with different ways to handle terrain mode. One way is to add an additional dimension to our example-based locomotion, which would require four dimensions and we haven't altered the code to handle that. An alternative was to replace the strafing dimension with the up/down dimension, which would then allow movement on the terrain; that is what the terrainMode attribute does. I don't think the data set that I distribute includes the up/down motions, however, so without them, terrain won't work properly. In the next week or so I'm going to update the data set to include this.

4) Which behaviors get 'stuck'? You can recall a reaching behavior by adding another reach command with the attribute: sbm:finish="true". You can cancel an entire BML command

5) default.py (which loads ../../../../data/sbm-common/scripts/default-init.py, same thing) shows a sample configuration with a number of characters. Certainly, you can load different characters, animations, faces, blends, whatever. I've tried to provide enough functionality to test out the majority of behaviors. I'd like to add a set of examples showing what you describe, but I haven't had time to do that. Maybe I'll get my intern to do it

The animation set in the sample isn't ideal - the idle pose does not match the pose in the blends, so when transitioning between locomotion and the idle pose, you will notice the feet sliding. This is even worse when the stepping is used one at a time, since every step causes a slide from the idle pose to the step pose, then another from the step pose back to the idle pose. If you run the blend editor and do the allStep blend, this doesn't happen. If you decided to run the single step posture as the idle posture like this:

<body posture="ChrUtah_Idle001"/>

then you wouldn't see the footsliding problem anymore. Also, the locomotion I included in this sample was created for Utah, and hasn't been retargeted to Brad, the Elder and the Doctor, so it'll look best on Utah.

Hopefully you are overwhelmed in a good way.

Regards,

Ari Shapiro

[al2950]

WOW!!!!

If your system is as half as good as it looks it will be amazing! The shear number of features you have crammed in is incredible. Congratulations to you and your team and thank you for releasing it under an open license. I cant wait to start playing with it .

I had a quick scan of your documentation and I noticed you mentioned about supporting PhysX in the future and I was wondering what you think the effort might be and point me in the direction of the relevant code as I would like to help

[arishapiro]

Not sure if I was clear enough in my original post, but SmartBody has a retargeting system that automatically determines a skeleton mapping and can quickly retarget a set of motions onto your character. I have a video of using Mixamo and 3dsMax skeletons here:

http://people.ict.usc.edu/~shapiro/mig1 ... _Final.mp4

So you could, for example, download a skeleton + rig, then retarget the locomotion, reaching, gestures, etc. onto your character.
Over time, I am trying to build a useful set of behavior libraries that can be quickly used by new characters.

Regards,

Ari Shapiro

[VJoe]

Wow, great work.
Another great project for the community.

Thanks

[arishapiro]

WOW!!!!

If your system is as half as good as it looks it will be amazing! The shear number of features you have crammed in is incredible. Congratulations to you and your team and thank you for releasing it under an open license. I cant wait to start playing with it .

I had a quick scan of your documentation and I noticed you mentioned about supporting PhysX in the future and I was wondering what you think the effort might be and point me in the direction of the relevant code as I would like to help

Regarding the PhysX integration, you should be able to substitute the ODE-specific parts for PhysX-specific parts.There's a base class called SbmPhysicsSim, which should be overriden by the specific engine (thus we have a SbmPhysicsSimODE class that extends the SbmPhysicsSim class). These tie into the SBPhysicsManager, and we also use ODE's collision detection in the SBCollisionManager. If you search for those terms, you should find the relevant code.
So, the process should be to implement a class called SbmPhysicsSimPhysX that extends SbmPhyiscsSim and override all those functions such as 'addPhysicsObj', 'addPhysicsCharacter', etc.

Let me know if you need more specific information than this.

Ari

[LBDude]

I can't believe this is real! This seems like just what I need for my game.

[arishapiro]

Hi Ari,

So far it works great. I would have some questions, only one ogre related though:

1) Is there a specific license for the data (models, skeletons, animation data, behaviors, ...)? Is it something free like CC0 or CC-by?
...

You are free to use the models, skeletons and animation data in your application or for use as long as you are using it in conjunction with SmartBody, but the data itself is not CC-0. I'll have to look more closely into CC-by to see if that could apply.

If you have a specific use of the data that falls outside of that, you are welcome to contact me directly; I will try to accommodate your needs.

Regards,

Ari Shapiro

[f00bar]

You are free to use the models, skeletons and animation data in your application or for use as long as you are using it in conjunction with SmartBody, but the data itself is not CC-0. I'll have to look more closely into CC-by to see if that could apply.

Hi Ari,

Thanks a lot for the detailed answers. CC-by would mean people have to say that they are using your animations. With none of the CC licenses you could enforce usage of smartbody imho.

I have too many ideas to realize, but I try to use/create stuff that can be of use for all of them. I can think of some usages that are not compatible with the stated license:

- generic game engine: uses smartbody for locomotion component, but also has simpler animation component for far away characters and performance. User of this engine creates game but only uses simple component. That means the game the user wants to distribute does not use smartbody. That means that animations for simple animation component cannot be derived from the smartbody based locomotion component. So smartbody can only be used in this case if all smartbody animations are recreated by engine creator, or having all base animations twice, once per component.

- cut scene creator: exposes smartbody functionality to direct your scene. Problem: this is only one part of a pipeline, next part wants to use result, but does not use smartbody. Runtime part on client maybe also does not use smartbody.

- asset conditioning: user loads not-game ready 3d art, allows to tune materials and geometry and to retarget animations (using custom animations or the smartbody animations). User maybe wants to sell creations on a 3d marketplace.

- commercial plugin for 3d package: exposes all smartbody functionality 1 to 1, lets you save created animations

- competing product to smartbody: uses all smartbody data but different code

- just selling the smartbody animations on a 3d marketplace


I can understand if you want to limit these usages (especially the final 3). However, wording of custom licenses can be a problem for certain projects (law department is not sure what "in conjunction with SmartBody" means). And some projects cannot commit to a technology without simple ways out (start using smartbody, retarget smartbody data to all assets, fine tune everything to smartbody, find out you need something simpler than smartbody => all retargeting and fine tuning lost if data from smartbody cannot be used anylonger).


So I hope you will find CC-by acceptable too (data can be used for everything, but users must state that they use smartbody data). I think it would also have a positive effect on smartbody adoption, because as stated certain projects are afraid of such wording. Having open source and open data makes the project accessible to all, otherwise some potential users are not (or cannot be) sure if their final output will comply with the license and not all of them are able or willing to create their own base animation sets.

Whatever you decide, having what we have now is already awesome.

(did not have more time to play with smartbody, but will soon)

[arishapiro]

You are free to use the models, skeletons and animation data in your application or for use as long as you are using it in conjunction with SmartBody, but the data itself is not CC-0. I'll have to look more closely into CC-by to see if that could apply.

Hi Ari,

Thanks a lot for the detailed answers. CC-by would mean people have to say that they are using your animations. With none of the CC licenses you could enforce usage of smartbody imho.

I have too many ideas to realize, but I try to use/create stuff that can be of use for all of them. I can think of some usages that are not compatible with the stated license:

- generic game engine: ...

- cut scene creator: ...

- asset conditioning: ...

- commercial plugin for 3d package: ...

- competing product to smartbody: ...

- just selling the smartbody animations on a 3d marketplace

...

You make a good point - that may be a better way to license the assets. I'll need to consult with some others in my group, but we might end up going that way. I'll post again on this topic soon.

Ari

[Slappy]

There features are really killing. very nice!

[arishapiro]

I am thinking of ways to better support the Ogre community with SmartBody. So far, here's what I have come up with:

1) Native support for Ogre skeleton formats
2) Native support for Ogre animation formats
3) Examples using standard Ogre assets, such as standard Ogre characters (do these exist?)
4) Binaries for Ogre (no compilation needed) on standard platforms: Windows, Linux, ...?

Is there anything else that someone can think of that I have overlooked?

Regards,

Ari Shapiro