[2008 - Accepted] Motion Synthesis based Character Animation

Threads related to Google Summer of Code
westine
Google Summer of Code Student
Google Summer of Code Student
Posts: 62
Joined: Sun Apr 01, 2007 12:12 pm

[2008 - Accepted] Motion Synthesis based Character Animation

Post by westine »

I am back this year. In the last year I have been working on my idea of soc 2007 in my leisure time out of usual research.

Here is the abstract of my soc 2008 idea which is still ANIMATION:
This proposal is motivated by my recent research on character motion synthesis. As the capture systems are widely used in game and feature movie industry, more captured motion data are available to ordinary users. In order to benefit from this abundant resource,in the last decade increasing researchers and engineers have been working on reusing motion capture data. Motion synthesis is one of the most universally adopted methods, which involves motion blending, interpolation, extrapolation, transition and so on. Meanwhile OGRE has its exporters set, which extract many skeleton&mesh models and animation from 3part tools, this help the ones who are familiar with these model authoring tools, but other users who are more likely to be novice to OGRE or 3D graphics engine may be don't use any other 3part model tools. All they need to do is create their animation in OGRE whether they are game makers or demo makers. Long in short, this application's work mainly includes two components: first, because as far as I know the largest on-line motion capture database is http://mocap.cs.cmu.edu/, and the data format of it is Asf/Amc for skeleton and character animation respectively, I will create a new exporter to convert these format to .skeleton and .mesh format which OGRE eats. Second, which is the most important and interesting part is the motion synthesis.
Last edited by westine on Tue Apr 22, 2008 5:00 am, edited 1 time in total.

westine
Google Summer of Code Student
Google Summer of Code Student
Posts: 62
Joined: Sun Apr 01, 2007 12:12 pm

detailed overview

Post by westine »

As the abstract mentions, this code of summer consist of two steps where the latter depends on the former.
The former: asf.amc to .mesh .skeleton exporter.
This exporter itself also comprise some parts.
Because the asf and amc formats do not include any mesh data which is so called skin, and most practical applications need skinning to enhance their visual reality. Thus we first need to bind skin to skeleton. Skins usually come from model authoring tool such as 3DsMax or Maya. Fortunately, most common modeling tools have ways feasible for users to skinning when skeleton are ready. All we need to do is put asf/amc format skeleton and motion animation into these tools and then export the resulted new format animation and model out into .skeleton .mesh using OGRE existed exporters.
It seems that now we could go ahead to the next part of our work ------- motion synthesis based character animation let some problem hidden behind the first step of data format conversion.
Until now, as I am a master student focus on 3D graphics, specially character animation, I have already developed a prototype version of motion synthesis based animation system.
My current work adopts a lot of prevailing methods used by other former researchers in character animation area.
They are :
Motion Graph referenced from
LEE, J., CHAI, J., REITSMA, P. S. A., HODGINS, J. K., AND
POLLARD, N. S. 2002. Interactive control of avatars animated
with human motion data. ACM Trans. on Graphics 21, 3, 491–
500.
ARIKAN, O., AND FORSYTH, D. A. 2002. Interactive motion
generation from examples. ACM Trans. on Graphics 21, 3, 483–
490.
KOVAR, L., GLEICHER, M., AND PIGHIN, F. 2002. Motion
graphs. ACM Trans. on Graphics 21, 3, 473–482.
Motion graph is a data structure for store motion clips to be synthesize and its associated algorithms for generating synthesis motion from motion graph.
And I have been extending their work with automatically any place transition detection and creation.
This is based on a new similar posture search algorithm which use dimension reduction technique from GPLVM( gaussian process latent variable model) with dynamic characteristics.
You can further reference to:
GROCHOW, K., MARTIN, S. L., HERTZMANN, A., AND
POPOVI´C , Z. 2004. Style-based inverse kinematics. In ACM
Transactions on Graphics. 23(3):522–531.
Wang, J. M., Fleet, D. J., & Hertzmann, A. (2006). Gaussian
process dynamical models. Adv. Neural Information
Processing Systems 18 (pp. 1441–1448). Proc. NIPS ’05.

I have done some basic features which are essential for the university transition generation algorithm. They include multiple characters loading with different motion assigned and play-back any selecting character motion. It is possible to select any bone of any character's skeleton, leaving the select any face of character's mesh to be continued. Translation and rotation of character are also implemented.
Motion graph's construction have been implemented with two version already, the Splice and the Agglomerate version.
The Splice version construction method calculates the transition matrices and use the most similar entry in transition matrices to connect motion graph. This method is the most used one in the reference papers.
I have partly implement an Agglomerate version for motion construction, which use annotation motions and scripted motion to set the high level and semantic motion graph, which is intuitive for model adjustment. The lower level of this version is based on the new GPVLM motion search algorithm which take around all semantic similar motion such as walking and running into the same GPLVM model, and looking for the posture with the least variance in statistical metric in prior learning model. For the Locality property of GPLVM, we draw a path in the latent space according the constraint set by users in the observation space( here refer to the real motion space).
Though all the above listens good and maybe exciting,
as soc 2008 is not a research work, I will keep this in mind all the time when I am working on soc 2008 for OGRE, that the most contribution I can make to OGRE is to make OGRE more usable for users.
Thus some more academic method would be simplified.
Now we should come back to the first .skeleton .mesh format conversion step, for what I have done on the second step of motion synthesis is based on the asf./amc. data format, there are two alternatives to chose to address this problem, 1ã€

westine
Google Summer of Code Student
Google Summer of Code Student
Posts: 62
Joined: Sun Apr 01, 2007 12:12 pm

schedule not yet very specific

Post by westine »

1ã€

User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 2653
Joined: Wed Sep 24, 2003 8:07 am
Location: Haute Garonne, France
x 4
Contact:

Post by tuan kuranes »

Part 1

you propose import/export of motion capture data into Ogre to those who don't have 3d tools or knowledge to use them ?

Do you propose some kind of Automatic Rigging as well ?
or does it needs an existing skeleton ? with constraints on joints and so ? or your algo tries to maps as it can ?


Part2

Those papers you cited are quite old for a very active researched domain, latest SIGGRAPH and i3d were full of motion graph related papers ?

Motion synthesis based character animation ? are those really applicable to real-time 3d or more to pre-bake animation to be played at run-time ? Do you plan to make a sort of constraint editor using Ogre ?

Isn't 'controllers' approach would be more 'real-time oriented' ?
Like Animation Controllers library&editor or active-learning-motion-controllers ?


Subsidiary question : As you might have to override how ogre control entity fom skeleton, can you be more precise on how do you plan to control ogre animation ? (animation blending/bone masking/blendtree,etc..)

westine
Google Summer of Code Student
Google Summer of Code Student
Posts: 62
Joined: Sun Apr 01, 2007 12:12 pm

thank you for attention

Post by westine »

Some things to be clear:
For current OGRE’s animation flow uses complete ready individual animation directly generated from other 3th-part tools, users whether they are familiar with some 3d tools or not, can not use motion capture data directly from motion database such as http://mocap.cs.cmu.edu. To explain what I want to do in part 1, the goal of part 2 should be clear first.
As you mentioned in Part 2, you doubted whether controller would be more suitable for this work. “Many studios use motion graphs these days, which are a simple way to get good looking animations into a game… but here are a few papers to help you take those ideas a bit further!â€

User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 2653
Joined: Wed Sep 24, 2003 8:07 am
Location: Haute Garonne, France
x 4
Contact:

Post by tuan kuranes »

Part 1:
I do need to import into or directly use motion capture data in OGRE.
Real-time ? don't you need some pre-processing step to compile it inside a more efficient binary form depending in motion capture set used ?

Part 2:
useful to understand why motion graph
Ok, didn't realize that paper listing was about introducing us to motion graph concept.
For my precise plan on how to override OGRE’s animation, could you give me some help document on .skeleton .mesh data format or anything you think is helpful. I am reading deeply into its code, and will give you my plan as soon as possible.
Override maybe too strong here, you might have to find a way to co-exist and add motion graph as a plugin that just control mesh skeleton joints. (letting interactive control being possible)

Code/Comments is the only way here, no manual (forums has many topic on the subject). You can ask specific question here.

Data format are binary (only created by Ogre) or xml human readable (export using OgreXMLconverter).

For now procedural control is not really plugged in, a patch is waiting to be applied.


Once you have a more detailed planning, can you develop on the deliverable list (code, plugin samples, utilities, editor, doc, etc.) ?

westine
Google Summer of Code Student
Google Summer of Code Student
Posts: 62
Joined: Sun Apr 01, 2007 12:12 pm

Post by westine »

tuan kuranes wrote:Part 1:
I do need to import into or directly use motion capture data in OGRE.
Real-time ? don't you need some pre-processing step to compile it inside a more efficient binary form depending in motion capture set used ?

I have read all animation related code in OGRE, discovering that the basic class diagram and control flowchart are substantially the same.
Please see my formatted document, rough video deom, shot and other material coming soon on: http://westine.googlepages.com
Part 2:
useful to understand why motion graph
Ok, didn't realize that paper listing was about introducing us to motion graph concept.
For my precise plan on how to override OGRE’s animation, could you give me some help document on .skeleton .mesh data format or anything you think is helpful. I am reading deeply into its code, and will give you my plan as soon as possible.
Override maybe too strong here, you might have to find a way to co-exist and add motion graph as a plugin that just control mesh skeleton joints. (letting interactive control being possible)

plugin? Is there a plugin template for its real implementation?
As far as I know, to port motion graph and base on it add interactive control needs a Refactoring, it thus maybe need a brach of OGRE's SVN develop stream.

Code/Comments is the only way here, no manual (forums has many topic on the subject). You can ask specific question here.
I have realized that code in OGRE are readable and comments are detailed, forum topics are specific, so they are enough for me even not yet with help of you and other mentors.

Data format are binary (only created by Ogre) or xml human readable (export using OgreXMLconverter).
A data format converter will sure to be realized, and it may not cost too much effort.

For now procedural control is not really plugged in, a patch is waiting to be applied.


Once you have a more detailed planning, can you develop on the deliverable list (code, plugin samples, utilities, editor, doc, etc.) ?
My detailed planing and corresponding explaination are on
http://westine.googlepages.com
As for the deilverable, I would like to be checked regularly, and I would like to become a long time member of OGRE not only on animation.
Thanks , Tuan.

User avatar
Kencho
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 4011
Joined: Fri Sep 19, 2003 6:28 pm
Location: Burgos, Spain
x 2
Contact:

Post by Kencho »

Westine, some quick (and mostly irrelevant) notes:
- In your proposal, diagram 1, the relationship stereotyped as "uses" should actually be a dependency stereotyped as "use" (see The Unified Modeling Language User Guide, G. Booch, J. Rumbaugh, I. Jacobson; chapter 10, dependency section)
- In the second diagram, just to state that the bidirectional associations have default cardinalities of 1-to-1, so you don't need to put them (they mess the diagrams, IMO :P). It's okay if you leave them; just a personal preference ;)
- The timeline looks a bit tight to me. Obviously I don't know what's your expertise in this field, or what research/work you've done on this before, so I don't want to speak much beforehand, but I would definitely deprioritize the samples and manual controlling aspects. Consider you'll be asked for functionality, and then samples. If you don't have the first, you may not pass the final evaluation, which would be a pity IMO. I would try to focus on the actual functionality, and then, once successfully implemented, write samples of it (which can be done even after the GSoC program has finished!).
- As a personal preference, I think this should be an add-on rather than core. But as you already contemplate this in your planning, will be better to let you decide when to take this decision :)

Other than that, good luck on your project. I won't be mentoring this year, but will be around in case you need extra help or useless advice :P I'm really looking forward for this project, so keep up the morale!
Image

User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 2653
Joined: Wed Sep 24, 2003 8:07 am
Location: Haute Garonne, France
x 4
Contact:

Post by tuan kuranes »

Great application now, much clearer.

Now some remarks:

- Ogre Diagram miss 'animationstateset' (and its limitations, like the current indexing using Ogre::String that prevent blending of two same animation).
- 'OgreOde' first ragdoll implementation is due to 'monster'. updates and fixes only are from me.
- 1.3 => no insights of the motion graph implementation (node, edges, etc..) ?
- Some use-case of controller, how an Ogre User will be able to add its own 'lookat(Vector3 objectPos)' controller for instance ?
- 'Gaussian Process Dynamical Models' video needs a bit explanation to be interesting... at least a title to help.

- Still curious about how motion capture asm/asf will be converted into a real-time binary format.

- Deliverables list and planning is about making us understand what we'll be able to use and when. Ogre Gsoc focus is really to end with something that will be used by Ogre users.
1.2 section’s 1 2 3 have been basically implemented, 4 and 5 are being coded, help is needed, thanks for any advice you give to me in advance.
"Release early, release often" so that we can have more insights. Even if not using Ogre yet, having binaries/source of what you're proposing herewould help everyone understanding motion graphs, and therefore raise questions/contribution/'feature modelling.' .

Slightly OT, but as I saw that you used opencv, and know about animation, isn't motion capture using opencv, cheap webcam and motion graph becoming possible ?

westine
Google Summer of Code Student
Google Summer of Code Student
Posts: 62
Joined: Sun Apr 01, 2007 12:12 pm

Post by westine »

tuan kuranes wrote:Great application now, much clearer.

Now some remarks:

- Ogre Diagram miss 'animationstateset' (and its limitations, like the current indexing using Ogre::String that prevent blending of two same animation).
Updated.
- 'OgreOde' first ragdoll implementation is due to 'monster'. updates and fixes only are from me.
- 1.3 => no insights of the motion graph implementation (node, edges, etc..) ?
Update soon.
- Some use-case of controller, how an Ogre User will be able to add its own 'lookat(Vector3 objectPos)' controller for instance ?
Lookat ? What does this mean? Change a character's lookat pose while other actions are executed. It is a really cool feature.
- 'Gaussian Process Dynamical Models' video needs a bit explanation to be interesting... at least a title to help.
OK. GPDM's use for motion synthesis has not been researched seriously, there are many difficulties, such as how could I sample a new motion in latent space, traditional methods use Monte Carlo Sampling, which is slow, very slow for realtime applications.

- Still curious about how motion capture asm/asf will be converted into a real-time binary format.
Real-time? binary? They are read into memory on demand in data structures such as motion and skeleton.

- Deliverables list and planning is about making us understand what we'll be able to use and when. Ogre Gsoc focus is really to end with something that will be used by Ogre users.
Update soon.
1.2 section’s 1 2 3 have been basically implemented, 4 and 5 are being coded, help is needed, thanks for any advice you give to me in advance.
"Release early, release often" so that we can have more insights. Even if not using Ogre yet, having binaries/source of what you're proposing herewould help everyone understanding motion graphs, and therefore raise questions/contribution/'feature modelling.' .
Source code release soon when smooth transitions are added.

Slightly OT, but as I saw that you used opencv, and know about animation, isn't motion capture using opencv, cheap webcam and motion graph becoming possible ?
It is really a novel application(other people have tried it, but no practical application has been developed as far as I know), and is sure to be welcome for a wide market. Maybe what I need to go further is more suggestion and ideas provided by a community as OGRE or other opensource group.

westine
Google Summer of Code Student
Google Summer of Code Student
Posts: 62
Joined: Sun Apr 01, 2007 12:12 pm

Some other ideas

Post by westine »

As I have read many siggraph and sca ( I3Dã€

User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 2653
Joined: Wed Sep 24, 2003 8:07 am
Location: Haute Garonne, France
x 4
Contact:

Post by tuan kuranes »

Ogre Diagram miss 'animationstateset' (and its limitations, like the current indexing using Ogre::String that prevent blending of two same animation).
Updated.
Diagram in PDF still miss the animationstateset. Please udpate link with a version number so that we know when it's new.
Lookat ? What does this mean?
Look this video that comes from this thread.
lookat is moving the head so that actor look at a point (that could be moving) in space, even if playing an animation.
They are read into memory on demand in data structures such as motion and skeleton.
flat text file ?!? too slow, too big. Real-time has to be current animation Ogre format. (even binary, that will cost much time/memory if animationtrack not optimized/cleaned.)

Anyway, Thanks for the source release, not sure what I can do with it, since most preprocessing won't work if I don't have the './locomotion/motionlist' file... If you have a manual or video tutorial showing how it works, that would help.
Anyway I've seen code you're working on.

Anyway, now, you have to focus on explaining what final 'Interface' you'll provide to Ogre. (at C++ level.)

Keep in mind that gsoc focus is to deliver working code, that will really be useful to end user.

westine
Google Summer of Code Student
Google Summer of Code Student
Posts: 62
Joined: Sun Apr 01, 2007 12:12 pm

Post by westine »

PDF has been updated, sorry for last time not renew version number.
Class diagram updated,
"lootat" feature has been added, just use forward kinematic, as noche said, there is really no IK involved in his code.
motion file serializer has been designed for motion graph as shown in the new PDF.

Useful function offered to end user of OGRE has been added to new PDF.

c++ code is being developed.

Refer to http://westine.googlepages.com please,
thanks to all, any advice and help is welcome.
Good luck.

westine
Google Summer of Code Student
Google Summer of Code Student
Posts: 62
Joined: Sun Apr 01, 2007 12:12 pm

I have been accepted

Post by westine »

Luck to be accepted as this summer gsoc student, now really cool work begins.

I am building a novel animation synthesis system these days, so forgive me not give new updates so quickly, after April 28, they will be uploaded.

And I have noticed that OgreCollada has some exporters with skeleton ã€

kex
Google Summer of Code Mentor
Google Summer of Code Mentor
Posts: 49
Joined: Thu May 24, 2007 8:57 am

Re: I have been accepted

Post by kex »

[quote="westine"]
And I have noticed that OgreCollada has some exporters with skeleton ã€

BLaize
Gnoblar
Posts: 15
Joined: Thu Feb 15, 2007 1:31 pm

Post by BLaize »

wouldnt something like this be of interest to you ?
or are you purely going for animations fone with motion capture data ?

Developing Controllers for Biped Humanoid Locomotion: (also includes source-code)
http://www.ipab.inf.ed.ac.uk/slmc/SLMCs ... achar.html

also, this site seems to have quite some info/papers on in this area:
http://www.cs.ubc.ca/~van/papers/
Check this out, and compare it with "NaturalMotion: Euphoria"
http://www.cs.ubc.ca/~van/papers/2007-s ... mbicon.mov

Moving Beyond Ragdolls:
http://www.mmandel.com/blog/?page_id=33
Last edited by BLaize on Mon May 05, 2008 8:32 pm, edited 2 times in total.

User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

I definitely enjoyed the simbicon video.

Tomat1979
Gnoblar
Posts: 2
Joined: Wed May 07, 2008 2:10 pm

i"m interested in result of your described work

Post by Tomat1979 »

i implemented this part - Motion graphs KOVAR[2002] - in NDA engine \ game. But it had been an content autoring tool, wich had been using to select the best transation frames for nodes \ animations in animation \ blend tree.

Now i think to continue reaserch in that direction (aka motion syntesis), with OrgeCollada help.

there is one more intesting link http://graphics.cs.williams.edu/papers/DynamoVGS06/ "Dynamo: Dynamic, Data-driven Character Control with Adjustable Balance"

robtherich
Halfling
Posts: 87
Joined: Sun Mar 05, 2006 10:56 pm
Location: Brooklyn, NY

Post by robtherich »

this project looks really interesting. i will be following closely.

perhaps someone has already posted a link to this site, but i didn't see it. seems related and possibly useful:
http://www.magix.ucla.edu/dance/

westine
Google Summer of Code Student
Google Summer of Code Student
Posts: 62
Joined: Sun Apr 01, 2007 12:12 pm

very sorry for delay reply

Post by westine »

I have been busy with a conference in Tokyo, Japan and some postmortem work.

Now I will focus on Ogre, all the work I have done is the conversion between amc\asf file format and XML, and then XML between Ogre inbuilt skeleton mesh format.

westine
Google Summer of Code Student
Google Summer of Code Student
Posts: 62
Joined: Sun Apr 01, 2007 12:12 pm

Post by westine »

BLaize wrote:wouldnt something like this be of interest to you ?
or are you purely going for animations fone with motion capture data ?

Developing Controllers for Biped Humanoid Locomotion: (also includes source-code)
http://www.ipab.inf.ed.ac.uk/slmc/SLMCs ... achar.html

also, this site seems to have quite some info/papers on in this area:
http://www.cs.ubc.ca/~van/papers/
Check this out, and compare it with "NaturalMotion: Euphoria"
http://www.cs.ubc.ca/~van/papers/2007-s ... mbicon.mov

Moving Beyond Ragdolls:
http://www.mmandel.com/blog/?page_id=33
Physical simulation surely is a necessary extension of this character animation, while this is another large amount of work.
I plan to use OgreODE to add these after motion capture data can be well embedded in Ogre and flexible transition in Motion Graph is implemented.

westine
Google Summer of Code Student
Google Summer of Code Student
Posts: 62
Joined: Sun Apr 01, 2007 12:12 pm

It seems physical controller is intensively desired

Post by westine »

So I tend to reschedule my summer code plan as

May 26 ----- June 25 :

Port motion caputer data into Ogre, which is serialized first into XML, and then can be converted with OgreXMLConverter into binary format.
The mesh and bone assignment \ weight is consistent with jaiqua.mesh,

all we need to deal with is the bone id matching , bonehierarchy revision, and mainly animation keyframeing in .skeleton file.

Then you can see individual mocap in Ogre.

June 26 --- August 16:
Interactive controling of character, motion synthesis.
Mainly based on motion graph.

After August 17:
Try physical controller using OgreODE.

http://www.cs.berkeley.edu/b-cam/Papers ... -2005-PPA/
http://www.mmandel.com/blog/?page_id=33

westine
Google Summer of Code Student
Google Summer of Code Student
Posts: 62
Joined: Sun Apr 01, 2007 12:12 pm

Re: i"m interested in result of your described work

Post by westine »

Tomat1979 wrote:i implemented this part - Motion graphs KOVAR[2002] - in NDA engine \ game. But it had been an content autoring tool, wich had been using to select the best transation frames for nodes \ animations in animation \ blend tree.

Now i think to continue reaserch in that direction (aka motion syntesis), with OrgeCollada help.

there is one more intesting link http://graphics.cs.williams.edu/papers/DynamoVGS06/ "Dynamo: Dynamic, Data-driven Character Control with Adjustable Balance"
Glad to meet you here, motion synthesis based on motion graph is a basis, although many papers recently use this, they don't mention it explicitly anymore. That is to say, although it maybe regarded as a normal structure for researchers in this area, it hasn't been well builtin in Ogre and other general 3D Engines.

You said you use OrgeCollada to further your motion synthesis research,
are you involved in OrgeCollada?
Any suggestion is welcome. Thanks for your attention.

westine
Google Summer of Code Student
Google Summer of Code Student
Posts: 62
Joined: Sun Apr 01, 2007 12:12 pm

Import Motion Capture Data

Post by westine »

Image

Something to decide: Because motion capture data only has skeleton and animation, but no mesh or other material is bound within, and so far as I know, all 3D content authoring tools that has OgreExporters need mesh to work. I propose two alternative ways to import mocap:

1ã€
Last edited by westine on Fri May 23, 2008 7:55 am, edited 1 time in total.

westine
Google Summer of Code Student
Google Summer of Code Student
Posts: 62
Joined: Sun Apr 01, 2007 12:12 pm

sample rate of animation

Post by westine »

the original motion capture animation has an average FPS of 120 fps, downsample to 30 fps is OK.

As I know OgreAnimation is based on keyframing. To import motion capture data, I must first keyframing the original animation.

Post Reply