Flexible Animation Blending and IK Support

[xavier]

Copied from the Wiki:

Animation Enhancements

* High-level stateful animation blending helper system, similar to AnimationBlender in the wiki, but with much more power. Such as just being able to add a new animation to the mix and have the transition and which animations it replaces or blends with automatically determined based on an initial configuration.
* As described in Adaptive Dynamics of Articulated Bodies that can be used in data driven simulation, data+user control, user control. Could implement some physical controller samples as in animating virtual characters using motion data and simulation in a context dependant way in order to show simplification results in each configuration (data, simulation, data+simulation)
* Procedural Motion Warping - introducing external factors into the animation display such as placing feet on terrain and a character reaching for an object. Likely to use an IK solution of some sort (e.g. CCD-IK) in combination with pre-scripted animation.
* Animation compression

I'd like to begin the discussion on bringing this to fruition, and the first step in that is a formal assessment of the current state of animation support in Ogre. Then, a restatement of the descriptions above in more formal terms. From there, we can identify what parts of Ogre would be affected, and what needs to be done to each. Following that, a task list and plan of attack can be generated. Backward-compatibility with all existing animations shall be preserved.

Background: Current state of animation support in Ogre

As of Ogre 1.5, Ogre supports two main types of animation:

• Vertex Animation. Pose (as well as the older 'morph' animation) are types of animations on the actual mesh data itself -- vertex positions, normals, etc., actually move with time. Each 'pose' and/or 'morph target' are kept in keyframed snapshots of the mesh configuration and vertex data is interpolated from keyframe to keyframe over time.
• Skeletal Animation. A transform hierarchy of 'bones' is maintained, each of which can affect one or more vertices in an associated mesh/submesh in a process known as 'skinning'. More accurately, 'matrix palette skinning', where the matrix palette is the set of transform matrices that correspond to the bones' states over time, and the relationship between the bones and the vertices they affect is kept in a set of 'bone assignments' that map possibly many-to-many bones to vertices in the mesh (a vertex can also be affected by more than one bone -- in Ogre, the limit on bone influences for a vertex is 4, and any influences above that number are ignored).

This article focuses primarily on the second type of animation -- skeletal animation. Blending of vertex animation with skeletal animation is already supported more than adequately in Ogre, and vertex animation automatically benefits from any enhancements to the skeletal animation support in Ogre. Enhancements to vertex animation are less important, since access to the vertex data for procedural purposes such as 'soft-body deformation' is likewise already supported more than adequately. However, for the purposes of creating a general animation blend network, consideration should be given to vertex animation types in terms of supporting those within the blending framework. [q1]

Ogre is Forward-Kinematic

Currently, skeletal animation in Ogre is strictly 'forward kinematic'; bone positions are interpolated between keyframes over time and the transform hierarchy is calculated 'outward' from the root bone(s) to the end effectors (to borrow a term from IK). The end effector has no say in where it ends up in the world -- it is entirely determined by the bones in its parenting hierarchy. This is by far the most efficient form of animating characters, and the process most often used for pre-scripted ('authored') animations in games and other interactive media.

The asset pipeline

Skeleton and animation data in Ogre can be created offline and exported from a tool such as SoftImage|XSI, 3DS Max, Maya and Blender; created fully in code using the Ogre API; or can be created using any combination of the two. Most of the time skeletons and their animations are created offline and exported ultimately to .skeleton files which are loaded by Ogre and parsed into instances of Skeleton and Animation (and their associated classes). Less often, applications may create their own skeletons and animations and apply those arbitrarily to objects in their application (applications that use a custom skeleton/animation format, for example, might do this in a ManualResourceLoader).

Skeletons and animations typically are exported in their entirety when exported to the intermediate XML format. Direct-to-binary exporters that use the Ogre SkeletonSerializer class, as well as exporters that support invocation of the XMLConvertor command-line tool, have the option of Ogre support in eliminating redundant bones in an exported skeleton. However, all keyframes/samples are exported unchanged.

Manual bone control, or animation control, but not both

Currently, a user can control a bone either by animation track, or by manually controlling it, but not by both at the same time.

Hardware acceleration of both major animation types

Ogre supports acceleration of both vertex and skeletal animation in GPU programs. This support is simplified by integration into the Ogre material pipeline, and often an animation can be hardware-accelerated simply by adding a few lines to a material script (plus of course, the creation of the GPu program and its associated declarations).


The case for enhancements

The above features and limitations of the Ogre animation system have served rather well, countless projects in a wide variety of venues, for a long time now (Ogre is 7 years old). Several commerical games and applications have shipped using this system, with great success. However, there is no doubt that it can be better.

• Bone control. The inability to mix authored animations with calculations based on Inverse Kinematics (IK), articulated-body methods (ABM) or other procedural methods, is actually a fairly significant limitation, especially given the current trend towards more articulated dynamic character simulations in games and 3D applications. With the current state of skeletal animation control in Ogre, you can either have fully authored animations, fully articulated simulation results, but not both -- there is no way to blend multiple control paths on the same bone.
• Animation Blending. Congruent with the above, Ogre supports only the low-level blending between 'animation states' (a high-level handle to the set of data that defines an individual authored animation). While it is true that an animation technically could be created on the fly (or altered on the fly), the fact is that the bone influences being blended still are under the control of animation tracks. This prevents true procedural animations not only from being blended with tracked animations, but also with each other (at least within the framework of the Ogre animation system -- it is certainly possible to create one's own procedural blender and just control the bones manually using the final blend output).
• Animation Compression. All keyframes or samples are exported during animation/skeleton export, regardless of whether the keyframe or sample effects a change relative to the one before it in a track. This can lead to larger-than-necessary file sizes, which in turn lead to larger-than-necessary memory footprints and longer-than-necessary load times. While the user can perform their own data compression on the exported data, Ogre should also provide this option within its toolchain, with compression parameters under the control of the user.
• Inverse Kinematics. Inverse kinematics is a way to define an animation based not on the cumulative effects of transforms in a kinematic chain (bones and their parents, in other words), but instead on the desired transform of an "end effector" in that kinematic chain -- put simply, instead of the arm bones telling the hand where to go, the hand says "I want to go here" and the arm bones automatically adjust to accommodate that desire. 3D Animators work regularly with IK targets when they are creating animations in their animation packages -- but the results of those animations are exported to keyframed animation tracks. Pursuant to the above point, IK export can serve as a form of data compression in that you could possibly express the animations of a skeleton of 50 bones by the positions of 7 or 8 effectors -- and export the keyframe tracks for only those effectors instead of a track per bone, resulting in a huge savings in exported animation size. Plus, IK is a good halfway point between full ABM and fully authored animations.

Impacted Ogre Systems

As I laid out the above, it became clear to me that the actual impact on existing Ogre systems really is minimal. Both manual and animation-track control over bone nodes can be decoupled from the controlled node, and those two types of control can simply be inputs to a blend network. The final output of the blend network would then be the only control over bone nodes in a skeleton. All existing Ogre systems, such as the animation interpolators, can be leveraged virtually unchanged, and additional systems for IK solving can be added; these additional systems would be transparent, masquerading as manual bone control systems, which also are just inputs to a blend network. Likewise for ABM simluators, which again serve simply as manual bone node controllers. Both CPU skinning and GPU acceleration of skeletal animation is unchanged as well, since they take as input simply the transforms of the bones, regardless of how the bones arrived at their configuration at that point in time. And since existing methods of bone control (manual control as well as fully authored animation) simply become inputs to a single-node blend network, nothing special has to be done to support those either.

Regarding the toolchain, exporters would need to be able to export the keyframed animations of the IK controllers, in addition to the current methods of keyframed export. New tags or attributes would be needed in the skeleton XML DTD for IK animations, but the actual animation tracks can remain unchanged -- they contain IK controller keyframe data instead of bone keyframe data, but the layout is the same.

As for animation compression, either/both of the XMLConverter or the SkeletonSerializer can be tasked with performing keyframe compression -- it could also be added as an option to the optimiseAllAnimations() method of Ogre::Skeleton.


So, essentially there is one major new system, the animation blend network controller, and enhancements to three existing systems:

• IK would be added to the AnimationTrack system of classes (potientially as a new class/classes)
• Animation compression would be added either to Skeleton or SkeletonSerializer
• SkeletonSerializer would also be enhanced to support serialization of IK data

Exporters that support IK would need to implement the additional tags/attributes needed, but that is primarily for XML exporters -- those that export directly to binary would simply need to populate the new data items in the skeletons they create. IK targets would exist in Ogre as nodes just like Bones, and their positions interpolated just how animated nodes are currently.


Tasks

So there are four major high-level tasks involved:

• Design/implement an animation blend network controller, integrate with node animation system
• Design/implement an IK solving subsystem, integrate with blend network system
• Design/implement an animation optimization strategy, integrate with SkeletonSerializer or create new class
• Design/implement serialization support for IK targets, integrate with SkeletonSerializer and IK solver system

[xavier]

One thing I want to say on IK, is that I'd like to offer a "pluggable" solver -- the default (sample) implementation that would be provided, would likely be the basic CCD (because it's easy to understand and implement) as well as one of the Jacobian methods (because they are the best tradeoff between accuracy, speed and pleasant results, this last which the CCD doens't always provide).

The idea of the pluggable solver (at least parts of it) is that users can use/try their own algorithms on the state space without having to alter the Ogre code, and without having to write the entire solver/controller themselves. One of the design principles then for the IK task would be to design it in such a way that this could be done -- and the implementation portion would provide two general-purpose ready-to-use sample implementations.

K, so that's my analysis, whipped up in a couple of hours. Comments/debate/suggestions/volunteers welcome of course.

[smernesto]

Why the animation compression feature in the skeleton/SkeletonSerializer ? what about scenenode animation? this also uses keyframes and can benefit from animation compression.

[Praetor]

Are not decent joint-based physics engines suitable for IK? I imagine their joint implementations will mostly be a decent mechanism to implement it. ODE is already part of the the Ogre distribution, so perhaps it can be used as an animation plugin. It should be able to be statically linked (BSD license I believe) to hide its dependency. New joint types could be made in a pluggable fashion so people can experiment that way. Or they provide a whole new plugin not based on ODE.

The biggest benefit I see is slightly augmenting authored shaders. Splinter cell's walk cycles are a good example. The walk animation is totally authored, but at runtime IK is used to properly place the feet on the walking surface. Using physics in concert with an authored animation the swing of a sword could react realistically to striking a shield or wall. But it seems to me these applications rely mostly on external control of the bones. Ogre would only need to provide a) a decent mechanism for getting at the bone (done already), b) the ability to share control of a bone between multiple animations, or animations and external control, so some of its motion comes from keyframes and some does not.

[xavier]

Why the animation compression feature in the skeleton/SkeletonSerializer ? what about scenenode animation? this also uses keyframes and can benefit from animation compression.

What applications export scene node animation?

[xavier]

Are not decent joint-based physics engines suitable for IK?

The point is to have it within Ogre, without relying on an external library. Like I said, it's a useful halfway point between that and fully authored animation.

[sinbad]

On the compression point, your comments aren't completely up to date - see Animation::optimise which removes unnecessary keyframes and tracks. Some exporters use this (e.g. XSI) to do a sample-then-optimise pass before export, meaning the simplest kind of compression has already been done. More advanced compression could certainly be done but would be more intrusive, such as enforcing more restrictions on the skeletal structure so that simpler data can be used (e.g. optimising for certain keyframe data and lossy compressing others).

The rest sounds ok, I would try to implement this through a layering as much as possible (which I think is your intent) so that systems like IK and procedural controls / constraints are combined relatively autonomously, meaning there's not too much intrusive change needed.

[xavier]

On the compression point, your comments aren't completely up to date - see Animation::optimise which removes unnecessary keyframes and tracks. Some exporters use this (e.g. XSI) to do a sample-then-optimise pass before export, meaning the simplest kind of compression has already been done. More advanced compression could certainly be done but would be more intrusive, such as enforcing more restrictions on the skeletal structure so that simpler data can be used (e.g. optimising for certain keyframe data and lossy compressing others).

It's the lossy compression I am talking about -- I did mention that Ogre does optimization on animations. The optimization I am talking about would require user interaction in the process to see if they can stand the level of lossy compression used.

The rest sounds ok, I would try to implement this through a layering as much as possible (which I think is your intent) so that systems like IK and procedural controls / constraints are combined relatively autonomously, meaning there's not too much intrusive change needed.

That's the goal. I realized as I was writing all that that not only is it largely non-intrusive (not much has to be changed) but the three main efforts in it are orthogonal to each other for the most part.

[smernesto]

What applications export scene node animation?

I know that Ogremax and ofusion can export scene nodes animation to NodeAnimationTracks.

The idea of lossy animation compression benefit me , because if we create many keyframes for many objects and long times, the applications will consume much memory.

[xavier]

Given that the compression method is on Animation, anything that uses that class would benefit.

Lossy compression is awesome when you are sampling unknown controllers every frame in the tool.

[prinz]

Hi there.
I read this thread about one month ago and hoped there would be an IK support in the new OGRE release 1.6. Just today I mentioned the release of 1.6 and read the changelog, but I was'nt able to find information there about this topic - only the short hint about 'skeletal blend masks'. Furthermore there is no hint in the manual.
Is IK support planned for a later release?