SmartBody - a character animation system for Ogre

[arishapiro]

I haven't read the entire topic, but what is the price for a non-LGPL licensed version of SmartBody?

<edit>
I know that your are intentionally being vague about this, so consider this a a request for a more transparent licensing option for those people who wants to use the library on mobile devices.
</edit>

Actually, I'm not trying to be vague. If you want a non-LGPL license, you can ask me (or more specifically, the Institute for Creative Technologies) for a license and there is no cost to this. We mainly want to keep track of who is using SmartBody, and would prefer to retain changes and improvements that are made to SmartBody (whether the code is linked statically or dynamically is not really that important). Even if you wish to keep changes or improvements to SmartBody private or proprietary, we are open to this as well. I would like to keep this project funded, and it is easier to make a case for this when there is usage and acceptance of it in the gaming, simulation and academic communities. So the more people that use it, the better chance I can retain (or enlarge) my team to support it.

To your point, my team would be thrilled if you were to use (and help improve...) the mobile capabilities - we just need to know who you are and what you intend on using it for.

Ari

[jacmoe]

Thanks, man. That clears it up.

[arishapiro]

...
After some time using SB, I have another round of how-to questions for you Ari
1) Using the code you have on my repository, I cannot get locomotion animations to work. The character moves in the 3D space, but no animation is played. No visible error in logs. I'm starting the locomotion using:

Code: Select all

<locomotion manner="run" target="1000 0 400 -1000 0 1000 10 -10" />

I'm setting up the locomotion animations the same way you do in Python (check SmartBodyTools.cpp and SmartBodyManager::createCharacter, I mimic'd a few of the python scripts in C++ with the native interface), except I'm not retargetting the motions as I'm using Utah and Utah's animations directly (no prefix).

The locomotion system can either be 'meat-hook' ("basic") animation (the character is dragged around without any animation) or it can use the example-based system ("example"), which uses several states consisting of many animations that control forward, sideways, turning and other movements (like you've got in the code). If all the states that are needed are present and the steering is set to 'example', then the example-based animation should work. So on line 403 of SmartBodyTools.cpp, change "basic" to "example". Also, make sure that the steering is enabled (steerManager->setEnable(true);) after all the characters have been configured. Let me know if that fixes your problem.

2) I also cannot get lipsyncing to work. Same as above, https://github.com/xplodwild/OgreSmartB ... r.cpp#L170 is used to setup the character, and I'm starting a speech using this (took from SB manual):

Code: Select all

sbmgr->executeBML(character->getName(), "<speech type=\"application/ssml+xml\">" \
		"<sync id=\" T0\"  time=\" .1\" />hello" \
		"<sync id=\" T1\"  time=\" .2\" />" \
		"<sync id=\" T2\"  time=\" .35\" />my" \
		"<sync id=\" T3\"  time=\" .4\" />" \
		"<sync id=\" T4\"  time=\" .6\" />name" \
		"<sync id=\" T5\"  time=\" .72\" />" \
		"<sync id=\" T6\"  time=\" .9\" />is" \
		"<sync id=\" T7\"  time=\" .1.07\" />" \
		"<sync id=\" T8\"  time=\" 1.4\" />Utah" \
		"<sync id=\" T9\"  time=\" 1.8\" />" \
		"<lips viseme=\" _\"  articulation=\" 1.0\"  start=\" 0\"  ready=\" 0.0132\"  relax=\" 0.0468\"  end=\" 0.06\" />" \
		"<lips viseme=\" Z\"  articulation=\" 1.0\"  start=\" 0.06\"  ready=\" 0.0952\"  relax=\" 0.1848\"  end=\" 0.22\" />" \
		"<lips viseme=\" Er\"  articulation=\" 1.0\"  start=\" 0.22\"  ready=\" 0.2442\"  relax=\" 0.3058\"  end=\" 0.33\" />" \
		"<lips viseme=\" D\"  articulation=\" 1.0\"  start=\" 0.33\"  ready=\" 0.3586\"  relax=\" 0.4314\"  end=\" 0.46\" />" \
		"<lips viseme=\" OO\"  articulation=\" 1.0\"  start=\" 0.46\"  ready=\" 0.4644\"  relax=\" 0.4756\"  end=\" 0.48\" />" \
		"<lips viseme=\" oh\"  articulation=\" 1.0\"  start=\" 0.48\"  ready=\" 0.4888\"  relax=\" 0.5112\"  end=\" 0.52\" />" \
		"</speech>");

But no lips are moving unfortunately. I don't plan to use a TTS, but rather an externally played sound file, I might be missing a config line somewhere.

There are two different lip syncing schemes; a lower quality one that activates a mouth shape on every viseme, and higher-quality one that accounts for combinations of visemes (the diphone-based method). Utah can only use the lower quality one, since he doesn't have the mouth shapes needed for the higher quality one (which are the same face shapes that FaceFX uses - the Brad and Rachel characters have those lip shapes). So first and foremost, you have to put Utah in lower quality lip syncing mode by setting the bool attribute 'useDiphone' to false (SmartBodyManager.cpp:171).

Having said that, in the past there are two ways to power the lip sync; via TTS, or embedded within a file. The command that is being sent actually isn't BML, but rather our internal format that gives instructions on how to accomplish the lip syncing. Up until now, we haven't had the need to send this information directly to the character (it's always been gathered from a file, or send from the TTS). To make this work you can do the following:

1) Set Utah to lower quality lip sync mode: character->setBoolAttribute("useDiphone", false);
2) Place the contents of the XML above in a file called 'myspeech.bml'.
3) Include an audio file called 'myspeech.wav' in the same directory
4) Tell Utah to read audio information from a file: character->setStringAttribute("voice", "audiofile");
5) Tell Utah where to find his sound files: character->setStringAttribute("voiceCode", ".");
6) Tell SmartBody where to find the audio files: scene->addAssetPath("audio", "path/to/my/directory")
7) run BML like this: sbmgr->executeBML(character->getName(), "<speech type=\"application/ssml+xml\" ref=\"myspeech\"></speech>");

Does that work for you?

Ari

[Xplodwild]

Great! They work fine now!

1) It turned out I had originally "example" set, but it said that "example" couldn't be turned on and it defaulted back to basic. I didn't really know why, but after taking a look at the sources it appears that in SbmCharacter::checkExamples(), there are LOG lines commented out, telling what motions are missing for 'example' to work. Turning them on allowed me to see that I was just having a wrong statePreFix (it should be the Character's name). After fixing that, everything went fine. I think those LOG lines should be enabled, it might help a lot.

2) After following what you did, lipsync worked fine. One question though, why does it needs the wav file, since it doesn't actually play it, but read phonemes from the sbm/xml file?

Here's a short demo of everything together: (gazing, locomotion, (quick) lipsync, pose, online retargetting, head nod and toss).

[youtube]M8aK-LQFhcc[/youtube]


EDIT: Final question: How to reset gazing or reach? Doing <gaze /> gives an error.

Thanks again for this wonderful library

[arishapiro]

Great! They work fine now!

1) It turned out I had originally "example" set, but it said that "example" couldn't be turned on and it defaulted back to basic. I didn't really know why, but after taking a look at the sources it appears that in SbmCharacter::checkExamples(), there are LOG lines commented out, telling what motions are missing for 'example' to work. Turning them on allowed me to see that I was just having a wrong statePreFix (it should be the Character's name). After fixing that, everything went fine. I think those LOG lines should be enabled, it might help a lot.

2) After following what you did, lipsync worked fine. One question though, why does it needs the wav file, since it doesn't actually play it, but read phonemes from the sbm/xml file?

Here's a short demo of everything together: (gazing, locomotion, (quick) lipsync, pose, online retargetting, head nod and toss).

[youtube]M8aK-LQFhcc[/youtube]


EDIT: Final question: How to reset gazing or reach? Doing <gaze /> gives an error.

Thanks again for this wonderful library

Fantastic! Regarding your comments
(1) I'll add in the debug information for locomotion, and
(2) you should hear the sound playing through the internal audio player, which you can enable by setting the "internalAudio" bool attribute on the scene: scene->setBoolAttribute("internalAudio", true);

Gazes are automatically removed when a new gaze comes into effect that uses the same joints (you can limit the gaze to the neck and head only which works better with locomotion like this: <gaze sbm:joint-range="NECK EYES" target="whatever"/>.
To remove a gaze, its best to give the gaze a name:
<gaze sbm:handle="mygaze" target="whatever"/>
and then later you can fade out that gaze:
<gaze sbm:handle="mygaze" sbm:fade-out="1"/>

reach uses a similar mechanism, with reach-finish:

<sbm:reach sbm:reach-finish="true"/>

All this BML needs to be simplified at some point.

Ari

[Lee04]

Ari wrote "Yes, my team has constructed the basic interface for using different physics engines, but we haven't made them completely separately yet, nor tested the interface. "

There was a physics wrapper for Ogre called Gangsta, That looked prommising but the guy how developed it got other things to do, his nick is/was Monster.

Maybe his work is worth while to look into,

[arishapiro]

a lot of file missing on the ios version of smartbody-lib proj like :

nvbg.cpp
nvbg.h
SBDebuggerClient.cpp
.........

We've fixed the iOS build and the instructions for how to build the library are in the manual.

Ari

[arishapiro]

Ari wrote "Yes, my team has constructed the basic interface for using different physics engines, but we haven't made them completely separately yet, nor tested the interface. "

There was a physics wrapper for Ogre called Gangsta, That looked prommising but the guy how developed it got other things to do, his nick is/was Monster.

Maybe his work is worth while to look into,

Character physics is a little more temperamental than generalized rigid body physics in that the parameters need to be highly tuned in order work properly (for example, if you had a controller that allowed the character to balance, it might work with physics engine A, but not with physics engine B without altering the configuration parameters). But this is also a problem when you want the character physics and the non-character physics to work in sync with each other; you either use the same simulator for both, with different tuning parameters, or two different simulators, and have a proxy representation of one in the other's context (a character's hand can have a physics representation that allows it to interact with other objects in the environment).

Ari

[arishapiro]

All,

The SourceForge site that hosts the SmartBody project has forced us to upgrade to its latest platform which has a different URL than in the past.
So if you've already downloaded SmartBody, you should relocate the code to the new URL by going into the trunk/ or top-level SmartBody directory and doing the following:

svn switch --relocate https://smartbody.svn.sourceforge.net/s ... body/trunk https://svn.code.sf.net/p/smartbody/code/trunk

You can also do this on Windows using TortoiseSVN by right-clicking on the folder, choose Tortoise SVN -> Relocate... and then enter:

https://svn.code.sf.net/p/smartbody/code/trunk

in the input field.

Ari

[Lee04]

Yes, I am aware of the physics parameter
differences.

How do you resolve that in your wrapper?

[arishapiro]

Yes, I am aware of the physics parameter
differences.

How do you resolve that in your wrapper?

We haven't. Our character physics uses ODE. We have abstractions for rigid bodies, forces, geometric surfaces, and so forth, but we haven't tried another physics engine with it yet.

Ari

[f00bar]

Another physics abstraction layer: PAL
http://www.adrianboeing.com/pal/index.html

[neopangaia]

Bullet Physics is a well-adopted physics system (an OGRE specific implementation is here http://www.ogre3d.org/tikiwiki/tiki-ind ... ogrebullet) i like the fact that it can be hardware accelerated, for more complex simulations offloaded from the CPU in a server side situation. It also has soft-body dynamics, should that ever become an element of Virtual Human Toolkit (SmartBody)

[arishapiro]

All,

We've added a phoneme scheduling tool which allows you to record a .wav file, type in the transcription, then automatically create a phoneme schedule which allows SmartBody to automatically lip sync to that audio.

SmartBody has always had that capability, but we've used a proprietary tool on our own projects that couldn't be included in the SmartBody distribution.

The scripts do the following:
1) Given some text, creates the phonemes (word sounds) associated with those words.
2) Align the phonemes with their presence in the audio
3) Convert the phonemes into SmartBody's format

These tools are located in the distribution in SmartBody/tools/CSLUPhonemeScheduler and in the manual under the section 'Configuring Speech'.

Regards,

Ari Shapiro

[thios]

This project is amazing! Thank your for making it available!

Xplodwild, I just started to play with smartbody-lib myself, so if you made any further progress with your bindings and would be willing to share, that would be great!

[arishapiro]

I'll be releasing a built version of the SmartBody library soon. Is it a good idea to include sample Ogre code integration with this as well?

If so, are there any other integration areas that are useful to the Ogre community?

What would be the easiest format for integration into your Ogre-based systems?

Ari

[thios]

I'm only speaking for myself but it would be great to have a working example of c++/smartbody-lib showing online/offline retargetting and locomotion on sinbad. Ideally in the style of ogre tutorials, but an ogreviewer-like example would also work!

[holocronweaver]

I'm only speaking for myself but it would be great to have a working example of c++/smartbody-lib showing online/offline retargetting and locomotion on sinbad. Ideally in the style of ogre tutorials, but an ogreviewer-like example would also work!

I second this suggestion. Demoing the main features of SmartBody in an Ogre example would be extremely useful for beginners.

[thios]

So I've followed in Xplodwild's footsteps and attempted to convert OgreDemo.py to c++, using smartbody-lib.

So far I got head movements and steering working. I'm facing a couple of problems, however, which are hopefully apparent in the embedded video (of sinbad shaking his head and walking between two points):
1)locomotion is choppy, as if an animation is trying to play
2)stance and locomotion animations are in fact not playing

I'm basing my example in BasicTutorial6. Am I making some obvious mistake(s) here?

Furthermore, the actual project to which I'm trying to incorporate smartbody is extending python3 through cython (and ogre). I'm assuming the best way to do this is to use smartbody-lib, and rewrite any bindings as needed. Is this accurate, or is there going to be python3 support eventually (or am I missing something else entirely)?

Thanks!

[youtube]pXPN6Qs2P5k[/youtube]

BasicTutorial6.h

BasicTutorial6.cpp

[arishapiro]

Hi,

I am attaching versions of that code with use Python (Xplodwild didn't want Python integrated, so his code assumes that there is no ability to script the commands, and instead, all of those scripted commands are instead embedded within the C++ code). I'll have to take a closer look to see why his code doesn't seem to work, but the basics are there.

Place the BasicTutorial6.py in the smartbody/data/ directory and change the code at line 176 to the root of your SmartBody directory.

Here's a video of the working code:

[youtube]chp9tOHNd60[/youtube]


Regards,

Ari Shapiro

[arishapiro]

Alos, just for fun, you can change line 44 from:

scene.run('BehaviorSetMaleLocomotion.py')

to

scene.run('BehaviorSetMaleMocapLocomotion.py')

to use the mocapped locomotion set, and you can see the difference between the animations in the two videos. This one is using data from a motion capture session so the dynamics and energy of real movement is seen.

[youtube]lBWDUEAHhFM[/youtube]

Regards,

Ari Shapiro

[Xplodwild]

Hi everyone,

I had to move on to another project for some time, so unfortunately I didn't have time to update my wrapper on GitHub. I'll update it ASAP when I get back on it.
I'm planning on scripting SmartBody using AngelScript, so through custom bindings (as well as some other types of integration in our level editor), thus having to rely on a python backend as well sounds like too much layer to me (though it could very well work, but I prefer to reduce the amount of dependencies).


@arishapiro: The mocap looks cool! Was it captured using Kinect or a pro gear? If Kinect, did you clean it afterwards?

[arishapiro]

The mocap was done through a PhaseSpace system, which has some advantages over Vicon-like systems, but still requires cleanup.
We are starting to experiment with the iPiSoft software that uses 2 Kinects to capture data.

Ari

[thios]

Thanks ..., that's exactly what I had in mind (apart from the no python bit)! I suggest it should make the wiki.

[arishapiro]

All,

We've enhanced the lip syncing algorithm and have some new results that will be published in the upcoming Motion in Games conference in Dublin this year:

http://smartbody.ict.usc.edu/HTML/video ... IG2013.mp4

Also, we've added an IK controller that reduces foot sliding artifacts, and will be releasing an SDK for Windows and OSx very soon (about 200 mb download).

Regards,

Ari Shapiro