SmartBody - a character animation system for Ogre

[arishapiro]

Thanks for the quick reply.

I'd be happy to help you with the Ogre wrapper. A great open source library like this deserves this kind of users contributions. I think it should work in a similar way to physics libraries bindings, hiding most of the internal classes and Interacting directly with ogres classes.

If you could tell me how to get started with the API (initialization, etc), I can dig through the code and start working on a wrapper/binding.

I've been refactoring the code a bit in order to clean up the API, but the main set of classes for use are located in the /sb/ directory here:
smartbody/core/smartbody/smartbody-lib/src/sb. I would stay away from the other directories (sbm/, sk/ controllers/, etc. ) as much as possible.

The basic object types in /sb/ are SBCharacter, SBAsset, SBScene, SBSkeleton, SBJoint, SBMotion, and so forth. There is an example of using them in the simplesmartbody project

I think the goal would be some kind of plug-and-play capability for Ogre. The OgreViewer project already has an integrated example with Ogre, where SmartBody characters are created and then instantiated with an Ogre character proxy, then updates to the Ogre character are done per time step from SmartBody. Keep in mind that the OgreViewer example uses a 'simpler' API that doesn't allow access to the main SmartBody object types.

Ari

[Xplodwild]

Thanks. I started making a small wrapper inside my game engine, keeping it as generic as possible so I can extract it later on and keep it standalone for other people.

I have one question though: there is a parameter "meshScale" for pawns and "deformableMeshScale" for characters, which I guess scale all axis at the same time. What if I have an object scaled at different values on X/Y/Z? (especially some pawns)

[arishapiro]

Thanks. I started making a small wrapper inside my game engine, keeping it as generic as possible so I can extract it later on and keep it standalone for other people.

I have one question though: there is a parameter "meshScale" for pawns and "deformableMeshScale" for characters, which I guess scale all axis at the same time. What if I have an object scaled at different values on X/Y/Z? (especially some pawns)

hmm...I didn't think of that. I suppose I could change 'meshScale' to a 3-tuple for x,y,z values. What kind of mesh would receive a non-uniform scale?

Ari

[Xplodwild]

Well unless I'm mistaken, pawns are all the static geometry the characters may interact with, right?
There is a huge possibility artists, in level editors, change the scale of some meshes in a non-uniform way to fit a particular case without having to go back to the modeling software. It's rare, but it may happen. For example with objects like trees/rocks, which they rotate and scale a lot in multiple ways to make cheap variations quickly.

[arishapiro]

Well unless I'm mistaken, pawns are all the static geometry the characters may interact with, right?
There is a huge possibility artists, in level editors, change the scale of some meshes in a non-uniform way to fit a particular case without having to go back to the modeling software. It's rare, but it may happen. For example with objects like trees/rocks, which they rotate and scale a lot in multiple ways to make cheap variations quickly.

Ok, I see. It's a little more nuanced than that: right now there is viewable geometry that is chiefly controlled by the renderer (say, Ogre), and collision geometry that SmartBody is aware of and can interact with. On a pawn, there is a 'collisionShape' attribute and a 'collisionShapeScale' attribute (which has xyz values) which creates a rigid collision surface for a pawn, but that doesn't necessarily match the 'mesh', which is how the object is displayed in the 3D scene. I haven't experimented with collisions between meshes and the characters yet (for example, a character touching the surface of a tree) but that is certainly something that is doable. I think Ogre supports some kind of collision detection, and it might be better to get that kind of information directly from Ogre.

Ari

[Xplodwild]

I still have a long way to go, I haven't checked the full doc in detail yet But thanks for the explanation about viewable geometry and collision mesh. This distinction should be made clearer in the manual I think, as you can see it's a bit confusing at first, especially when implementing in an engine (where you wouldn't care about rendering at all). Does the position/orientation controls also only affects render, or it is used for SmartBody interactions too? Do we need to read the position from SB when using locomotion for instance?

Also, I quickly checked the doc, and it looks like collisionShape is limited to capsule/boxes/spheres? Is there any way to have a more complex collision mesh? Is there some interaction possible with an external physics/collision engine to provide SmartBody the collisions checks it needs? (I haven't checked the physics interface yet, so excuse me if it's happening there, just trying to grab as much info as possible to make things smoother)

[yaxinhoo]

a lot of file missing on the ios version of smartbody-lib proj like :

nvbg.cpp
nvbg.h
SBDebuggerClient.cpp
.........

[Xplodwild]

a lot of file missing on the ios version of smartbody-lib proj like :

nvbg.cpp
nvbg.h
SBDebuggerClient.cpp
.........

It's a problem on your side... It's fine here.



@Ari: Is it normal that Brad's ogre mesh bones names are named after the standard SmartBody skeleton (base/pelvis1/...), but the FBX, SK and other skeleton description files refers to JkBase, JkPelvis1? (ie. the names mismatches)
EDIT: Nvm, digged through the python scripts and found the Zebra2 mapping

Also, in my test app I still get the "bind() failed" with the same error 10048.

[yaxinhoo]

......
It's a problem on your side... It's fine here.

did you , do you can find

Code: Select all

sbm/SBPhysicsSimODE.cpp
sbm/SBPhysicsSimODE.h
sbm/steering/SteeringAgent.cpp
sbm/steering/SteeringAgent.h
sbm/Event.cpp
sbm/Event.h
sbm/mcontrol_util.cpp
sbm/mcontrol_util.h
sbm/Resource.cpp
sbm/Resource.h
sbm/resource_cmds.cpp
sbm/resource_cmds.h
sbm/ResourceManager.cpp
sbm/ResourceManager.h
sbm/sbm_character.cpp
sbm/sbm_character.hpp
sbm/sbm_pawn.cpp
sbm/sbm_pawn.hpp
sbm/SbmCharacterListener.h
sbm/SBDebuggerServer.cpp
sbm/SBDebuggerServer.h
sbm/viseme_map.hpp

i missed those file , i can't find in any folder of the dir of smartbody. what about u ?

can u upload you "smartbody-lib/src" dir's and smartbody-lib.xcodeproj .

[Xplodwild]

......
It's a problem on your side... It's fine here.

did you , do you can find

Code: Select all

sbm/SBPhysicsSimODE.cpp
sbm/SBPhysicsSimODE.h
sbm/steering/SteeringAgent.cpp
sbm/steering/SteeringAgent.h
sbm/Event.cpp
sbm/Event.h
sbm/mcontrol_util.cpp
sbm/mcontrol_util.h
sbm/Resource.cpp
sbm/Resource.h
sbm/resource_cmds.cpp
sbm/resource_cmds.h
sbm/ResourceManager.cpp
sbm/ResourceManager.h
sbm/sbm_character.cpp
sbm/sbm_character.hpp
sbm/sbm_pawn.cpp
sbm/sbm_pawn.hpp
sbm/SbmCharacterListener.h
sbm/SBDebuggerServer.cpp
sbm/SBDebuggerServer.h
sbm/viseme_map.hpp

i missed those file , i can't find in any folder of the dir of smartbody. what about u ?

can u upload you "smartbody-lib/src" dir's and smartbody-lib.xcodeproj .

My bad - I skipped the "iOS" in your post. I miss those files too, but I'm not working on iOS platform.

[arishapiro]

did you , do you can find

Code: Select all

sbm/SBPhysicsSimODE.cpp
sbm/SBPhysicsSimODE.h
sbm/steering/SteeringAgent.cpp
sbm/steering/SteeringAgent.h
sbm/Event.cpp
sbm/Event.h
sbm/mcontrol_util.cpp
sbm/mcontrol_util.h
sbm/Resource.cpp
sbm/Resource.h
sbm/resource_cmds.cpp
sbm/resource_cmds.h
sbm/ResourceManager.cpp
sbm/ResourceManager.h
sbm/sbm_character.cpp
sbm/sbm_character.hpp
sbm/sbm_pawn.cpp
sbm/sbm_pawn.hpp
sbm/SbmCharacterListener.h
sbm/SBDebuggerServer.cpp
sbm/SBDebuggerServer.h
sbm/viseme_map.hpp

i missed those file , i can't find in any folder of the dir of smartbody. what about u ?

can u upload you "smartbody-lib/src" dir's and smartbody-lib.xcodeproj .[/quote]


My bad - I skipped the "iOS" in your post. I miss those files too, but I'm not working on iOS platform.[/quote]

The iOS build is probably not up to date (it's last in line for fixes, since it takes a little extra work to build it compared to the other platforms). I'll try to get it up to date today.

Ari

[Xplodwild]

Carried on with my implementation, and hurting a first problem, check the attached screenshot. That's basically the one and only pose I can get Brad (I hope it doesn't hurt him!). The character is totally static, as if no animation is running. I've searched for a few hours, but couldn't find what's wrong. There is nothing particular in logs (except that it cannot set the mesh).

Here's what I'm doing:
1) Initialization

Code: Select all

m_pScene = SmartBody::SBScene::getScene();

	// Load assets paths
	addAssetPath("motion", "Data/SmartBody/ChrBrad");
	addAssetPath("mesh", "Data/SmartBody/mesh");
	addAssetPath("motion", "Data/SmartBody/sbm-common/common-sk");
	addAssetPath("script", "Data/SmartBody/sbm-common/scripts");

	std::cout << "Loaded " << m_pScene->getNumMotions() << " motions, " << m_pScene->getNumScripts() << " scripts." << std::endl;

	m_pScene->start();

(addAssetPath is a method that adds and load a path)
I get 356 motions loaded and 0 scripts, which is to be expected as I built the lib without Python support (SB_NO_PYTHON).


2) Creation of a character

Code: Select all

	SmartBody::SBGestureMapManager* gmm = m_pScene->getGestureMapManager();
	SmartBody::SBGestureMap* gMap = gmm->createGestureMap("ChrBrad");

	gMap->addGestureMapping("ChrBrad@Idle01_YouLf01", "DEICTIC", "YOU", "LEFT_HAND", "", "ChrBrad@Idle01");

	SmartBody::SBCharacter* character = m_pScene->createCharacter("ChrBrad", "");

	// Mapping from Zebra2 skeleton to SmartBody skeleton
	SmartBody::SBJointMapManager* jmm = m_pScene->getJointMapManager();
	SmartBody::SBJointMap* zebra2Map = jmm->createJointMap("zebra2");

	// Core
	zebra2Map->setMapping("JtRoot", "base");
	zebra2Map->setMapping("JtSpineA", "spine1");
	[...]
	SmartBody::SBSkeleton* skel = m_pScene->createSkeleton("ChrBrad.sk");
	zebra2Map->applySkeleton(skel);
	zebra2Map->applyMotionRecurse("Data/SmartBody/ChrBrad");
	zebra2Map->applyMotion(m_pScene->getMotion("ChrBrad@Idle01"));

	character->setSkeleton(skel);
	character->createStandardControllers();
	character->setStringAttribute("gestureMap", "ChrBrad");
	character->setStringAttribute("deformableMesh", "ChrBrad");
	
	m_pScene->getBmlProcessor()->execBML("ChrBrad", "<body posture=\"ChrBrad@Idle01\"/>");

3) Update loop

Code: Select all

	SmartBody::SBSimulationManager* sim = m_pScene->getSimulationManager();
	sim->updateTimer(_timeDelta);
	
	// loop for each character => loop for each joint
	bone->setManuallyControlled(true);
	bone->setOrientation(quatDelta);
	// end loop

What am I doing wrong/missing ?

[arishapiro]

You should have:

sim->setTime(timeInSeconds);
m_pScene->->update();

somewhere in your loop to update the SmartBody world, assuming that you are setting the time manually and not using the realtime internal clock. Does that change your results?

The bind() error is related to a connection to the SmartBody debugger (remote connection to the application that lets you examine the state of the system) that needs to be rearranged a bit, so ignore that for now.

Ari

[Xplodwild]

You should have:

sim->setTime(timeInSeconds);
m_pScene->->update();

somewhere in your loop to update the SmartBody world, assuming that you are setting the time manually and not using the realtime internal clock. Does that change your results?

The bind() error is related to a connection to the SmartBody debugger (remote connection to the application that lets you examine the state of the system) that needs to be rearranged a bit, so ignore that for now.

Ari

Okay, works better with the setTime and update, now the character moves, but the pose is still very weird.
Do I need to iterate on joints children too?

I forgot to mention what is quatDelta in my previous post:

Code: Select all

Ogre::Quaternion quatDelta(joint->quat()->value().w,joint->quat()->value().x,joint->quat()->value().y,joint->quat()->value().z);

[arishapiro]

Okay, works better with the setTime and update, now the character moves, but the pose is still very weird.
Do I need to iterate on joints children too?

I forgot to mention what is quatDelta in my previous post:

Code: Select all

Ogre::Quaternion quatDelta(joint->quat()->value().w,joint->quat()->value().x,joint->quat()->value().y,joint->quat()->value().z);

[/quote]

Yes, you need to set the orientation on each of the joints as well. The idea is that a SmartBody character is running in its own context, and its state gets copied over to your Ogre character.

Ari

[Xplodwild]

Yes, I was looping over all the Joints given by Skeleton as manual indicates. I was asking if I needed to recursively apply the joints' children orientation too.

Anyway, that's what I did, and I get pretty much the same result...

[arishapiro]

Yes, I was looping over all the Joints given by Skeleton as manual indicates. I was asking if I needed to recursively apply the joints' children orientation too.

Anyway, that's what I did, and I get pretty much the same result...

Can you either post all your code here, or send it to me at: shapiro@ict.usc.edu?

Ari

[Xplodwild]

Okay, after a short email conversation with Ari, it turned out the Brad skeleton isn't the same as the ChrBrad.sk file. Using common.sk fixed the issue.
Retargetting works fine, here's Utah rocking!

I'll be working on the wrapper this weekend, and will upload it on github once it works fine

[arishapiro]

Great! You even got the online retargeting working!
In theory, any character should work with SmartBody, as long as there is a mapping between its skeleton and the standard SmartBody skeleton. and as long as the character's skeleton is in a format that SmartBody can read (for example, using Ogre's XML skeleton standard).
Without building in the Python interface, some configuration tasks would have to be done in the C++ code (which isn't as convenient to change if it were in a script file) or a separate format (like XML) would need to be written. Is there a scripting language that many Ogre users prefer instead of Python?

Ari

[Xplodwild]

From what I read around here, people tend to use LUA. I personally use AngelScript for its flexibility.

I pushed my work-in-progress for my SmartBody wrapper. It's useable if you want to run a quick test in your game, you'll just need to replace a few includes as I pulled the files right from my engine, and remove C++11 keywords if you don't have them. I'll clean this once the wrapper will be complete, but it should fit for most cases.
https://github.com/xplodwild/OgreSmartBody

There is a README.md included for more details and example of use. Lib uses the same license as SmartBody, LGPL.

After some time using SB, I have another round of how-to questions for you Ari
1) Using the code you have on my repository, I cannot get locomotion animations to work. The character moves in the 3D space, but no animation is played. No visible error in logs. I'm starting the locomotion using:

Code: Select all

<locomotion manner="run" target="1000 0 400 -1000 0 1000 10 -10" />

I'm setting up the locomotion animations the same way you do in Python (check SmartBodyTools.cpp and SmartBodyManager::createCharacter, I mimic'd a few of the python scripts in C++ with the native interface), except I'm not retargetting the motions as I'm using Utah and Utah's animations directly (no prefix).

2) I also cannot get lipsyncing to work. Same as above, https://github.com/xplodwild/OgreSmartB ... r.cpp#L170 is used to setup the character, and I'm starting a speech using this (took from SB manual):

Code: Select all

sbmgr->executeBML(character->getName(), "<speech type=\"application/ssml+xml\">" \
		"<sync id=\" T0\"  time=\" .1\" />hello" \
		"<sync id=\" T1\"  time=\" .2\" />" \
		"<sync id=\" T2\"  time=\" .35\" />my" \
		"<sync id=\" T3\"  time=\" .4\" />" \
		"<sync id=\" T4\"  time=\" .6\" />name" \
		"<sync id=\" T5\"  time=\" .72\" />" \
		"<sync id=\" T6\"  time=\" .9\" />is" \
		"<sync id=\" T7\"  time=\" .1.07\" />" \
		"<sync id=\" T8\"  time=\" 1.4\" />Utah" \
		"<sync id=\" T9\"  time=\" 1.8\" />" \
		"<lips viseme=\" _\"  articulation=\" 1.0\"  start=\" 0\"  ready=\" 0.0132\"  relax=\" 0.0468\"  end=\" 0.06\" />" \
		"<lips viseme=\" Z\"  articulation=\" 1.0\"  start=\" 0.06\"  ready=\" 0.0952\"  relax=\" 0.1848\"  end=\" 0.22\" />" \
		"<lips viseme=\" Er\"  articulation=\" 1.0\"  start=\" 0.22\"  ready=\" 0.2442\"  relax=\" 0.3058\"  end=\" 0.33\" />" \
		"<lips viseme=\" D\"  articulation=\" 1.0\"  start=\" 0.33\"  ready=\" 0.3586\"  relax=\" 0.4314\"  end=\" 0.46\" />" \
		"<lips viseme=\" OO\"  articulation=\" 1.0\"  start=\" 0.46\"  ready=\" 0.4644\"  relax=\" 0.4756\"  end=\" 0.48\" />" \
		"<lips viseme=\" oh\"  articulation=\" 1.0\"  start=\" 0.48\"  ready=\" 0.4888\"  relax=\" 0.5112\"  end=\" 0.52\" />" \
		"</speech>");

But no lips are moving unfortunately. I don't plan to use a TTS, but rather an externally played sound file, I might be missing a config line somewhere.

[TheSHEEEP]

From what I read around here, people tend to use LUA. I personally use AngelScript for its flexibility.

+1 for Lua. I think it is simply the fastest scripting language out there, even more with LuaJIT.
Have a comparison of Lua vs. Python (and it isn't even LuaJIT, which is even faster!).
Also, it is probably the most used one.

In any case, I'd advise against Python as it is neither the fastest nor the most used one in games
Not sure about scripting languages outside of gaming, though, tbh.
Maya uses Python.

Sorry for spamming links

[arishapiro]

From what I read around here, people tend to use LUA. I personally use AngelScript for its flexibility.

+1 for Lua. I think it is simply the fastest scripting language out there, even more with LuaJIT.
Have a comparison of Lua vs. Python (and it isn't even LuaJIT, which is even faster!).
Also, it is probably the most used one.

In any case, I'd advise against Python as it is neither the fastest nor the most used one in games
Not sure about scripting languages outside of gaming, though, tbh.
Maya uses Python.

Sorry for spamming links

It's might be possible to add a SWIG interface that would allow other scripting languages (Lua, whatever) in the future (I'm using BOOST-Python which requires a separate file to map script function calls to C++ function calls, essentially the same thing as SWIG) but I won't be able to get to that for a number of months, since documentation and such are more of a priority. The Python interface itself only requires the Python DLL, and for the most part, doesn't need any of the Python libraries, so it shouldn't be too intrusive to include that as well (we've ported Python to many other platforms without issue: iOS, Android, flash, etc.). The Python interface is mostly used for character setup (although certainly it can be used during runtime as well).

Ari

[yaxinhoo]

please provide the iphonesimulator lib;

[jacmoe]

I haven't read the entire topic, but what is the price for a non-LGPL licensed version of SmartBody?

<edit>
I know that your are intentionally being vague about this, so consider this a a request for a more transparent licensing option for those people who wants to use the library on mobile devices.
</edit>

[Mind Calamity]

It's might be possible to add a SWIG interface that would allow other scripting languages (Lua, whatever) in the future (I'm using BOOST-Python which requires a separate file to map script function calls to C++ function calls, essentially the same thing as SWIG) but I won't be able to get to that for a number of months, since documentation and such are more of a priority. The Python interface itself only requires the Python DLL, and for the most part, doesn't need any of the Python libraries, so it shouldn't be too intrusive to include that as well (we've ported Python to many other platforms without issue: iOS, Android, flash, etc.). The Python interface is mostly used for character setup (although certainly it can be used during runtime as well).

Ari

SWIG sounds good (given that OGRE's userbase is pretty scattered when it comes to scripting languages), although I haven't been able to get it to work on Windows for my projects, but it would be the most optimal solution, since manually maintaining wrappers for different scripting languages is just not worth it.