Kinect Yoga

[Moohasha]

For my final project this semester I had to make a program that would track a user's motions as he performed various yoga poses and tell him how well he did. I used an Xbox 360 Kinect sensor to capture user motions and animate a 3D model. The same code was also used to record our reference data, which is what the user's input is compared against to determine if he is doing the pose properly. Essentially, I used the Kinect sensor to record motion capture data that we could then playback.

I only had about 10 weeks to work on it, so it's not very fancy. I'm not a 3D artist, so I just used the nice Sinbad model and some of the code from the SampleAppSinbad to animate him using input from the Kinect.

This demo shows me performing two of the recorded motions. For the first one I do it properly and score pretty high. For the second one I intentionally goof off to show that your accuracy is based on how long your hands and feet are properly performing the selection motion.

The color video in the bottom corner is simply provided to show you what the sensor sees. This helps verify that you are in the sensor's field of view in case you are having problem getting tracked. The video looked fine on my PC, but for some reason the color channels got flipped when I ran it on my laptop.

And yes, I really used SdkTrays for my GUI. =P I was in a hurry and didn't have time to make a "real" GUI.

[youtube]kZwSsYIAZjY[/youtube]

[spacegaier]

Nice idea, to let sinbad make some Yoga! Nicely done!

BTW: Did you do some changes to SDKtrays or is that the version out-of-the-box?

[Moohasha]

Thanks.

Yeah, I made some changes. I added callback methods for buttonEnter() and buttonExit() to know when I moused over buttons, and I created a new custom widget called Thumbnail that contained an image and a label and acted like a button. All of the other "widgets" you see are either standard SdkTrays widgets or just normal overlay elements with the SdkTray skins.

[spacegaier]

Any plans on releasing the changed SDKtray source? I think there are quite some people out there who would like a slightly extended SDKtrays version (including me ).

[Jabberwocky]

That's really cool. It looks pretty slick, especially given the time and art assets you had to work on it. I've never seen the Ogre looking so zen.

[nec]

Nice, I like the idea, this deserves kudos

[Moohasha]

Thanks for the feedback guys. Yeah, this isn't anything that's production quality, just a school project. I didn't have time to add more bells and whistles (was going to have more verbal instruction, additional visual effects when you aren't doing the pose correctly, etc.), but I was happy with how it turned out.

@spacegaier, my changes to SdkTrays were quick and hacky, so I wouldn't release them as is. I made the changes specifically to meet the needs of this project, so they aren't very flexible. However, I can provide the few lines of code I added to get mouse enter/exit events on a button.

In SdkTrayListener, add the following virtual methods:

Code: Select all

virtual void buttonEnter(Button* button) {}
virtual void buttonExit(Button* button) {}

Then update Button::setState() to look as follows:

Code: Select all

void setState(const ButtonState& bs)
{
  if (bs == BS_OVER)
  {
    if (mListener && (mState != BS_OVER && mState != BS_DOWN))
      mListener->buttonEnter(this);

    mBP->setBorderMaterialName("SdkTrays/Button/Over");
    mBP->setMaterialName("SdkTrays/Button/Over");
  }
  else if (bs == BS_UP)
  {
    if (mListener && (mState == BS_OVER || mState == BS_DOWN))
      mListener->buttonExit(this);

    mBP->setBorderMaterialName("SdkTrays/Button/Up");
    mBP->setMaterialName("SdkTrays/Button/Up");
  }
  else
  {
    mBP->setBorderMaterialName("SdkTrays/Button/Down");
    mBP->setMaterialName("SdkTrays/Button/Down");
  }

  mState = bs;
}

I needed the mouse enter/exit events so that I could play sound effects when you moused over different buttons. But like I said, the thumbnail class I added was really more of a custom overlay and not very integrated with SdkTrays.

[jacmoe]

Really splendid!
One criticism is that Sinbad is badly skinned.. But it takes an expert modeler to fix it. It's a classic shoulders issue.

[shadowfeign]

Really splendid!
One criticism is that Sinbad is badly skinned.. But it takes an expert modeler to fix it. It's a classic shoulders issue.

actually the while the shoulder issue is classic, its the lack of animation on the clavicle that's the problem. The arm can't actually go above a t-pose without engaging the clavicle.

for some great anatomy ref, watch these videos by Jeff Hesser that can be found on riggingdojo.com
[vimeo]16075544[/vimeo]
[vimeo]16078456[/vimeo]

[jacmoe]

Splendid reference.
Sinbad needs to be fixed.

[duststorm]

This is a really cool technical demo! What libraries and code did you use to interface with the kinect? I suppose you used the OpenNI drivers, is that correct?
What did you use for the pose recognition? Did you use some existing middleware or libraries, I know there already are some tech demos available (such as Nicholas Burrus RGBDemo: http://nicolas.burrus.name/index.php/Re ... tRgbDemoV6), or did you do it all yourself by interpreting the RGBD (RGB+Depth) image?

This works by tracking a skeleton in the kinect image and linking that skeleton to the skeleton of a 3D model in your application, right?

I wonder, given supporting libraries, how compact a simple kinect sample in Ogre3d could be made.

PS: I hadn't noticed that animation issue with Sinbad yet. Thanks for pointing it out. Another piece of wisdom I can add to my (very limited) knowledge of modeling and rigging.

[Moohasha]

Really splendid!
One criticism is that Sinbad is badly skinned.. But it takes an expert modeler to fix it. It's a classic shoulders issue.

Yeah, it doesn't show up too much with the standard Sinbad animations, but when manually controlling the bones it becomes a problem. This is because I'm using the orientation data that I get from the sensor, which isn't always correct (or natural).

This is a really cool technical demo! What libraries and code did you use to interface with the kinect? I suppose you used the OpenNI drivers, is that correct?

Thanks. Yes, I used OpenNI because it gives you both position and orientation for each bone (though not both position AND orientation for ALL bones), which made it easier to rig the mesh. The Microsoft Kinect SDK was a lot easier to use, but only gives you position for each bone. In addition to OpenNI, I had to use the PrimeSense Nite middleware module (also available from http://www.openni.org) and the SensorKinect hardware module.

What did you use for the pose recognition? Did you use some existing middleware or libraries, I know there already are some tech demos available (such as Nicholas Burrus RGBDemo: http://nicolas.burrus.name/index.php/Re ... tRgbDemoV6), or did you do it all yourself by interpreting the RGBD (RGB+Depth) image?

OpenNI gives you the skeleton data automatically. While it can do pose detection (ie, let you know when you raise your hand), that's not what I was trying to do. I was doing pose tracking. I was trying to determine how accurately you performed a full 4D (position and time) pose, so I did it all myself. Essentially, for each frame, I compared the (x, y, z) position of the hands and feet to the reference data and measured deviation. This had to be adjusted to compensate for differences in size between the user and reference data.

This works by tracking a skeleton in the kinect image and linking that skeleton to the skeleton of a 3D model in your application, right?

I wonder, given supporting libraries, how compact a simple kinect sample in Ogre3d could be made.

Correct. As stated above, OpenNI gives you the skeleton data automatically, so there's no image processing necessary. The actual rigging isn't very straightforward, so I'm indebted to OpenNI (I think they're the ones who provided it) for providing the sample application that already did this (see my original post).

[duststorm]

Thanks for the explanation.
I wasn't aware that the OpenNI package already included that much useful tools for skeleton tracking.

And you're right about the difference between skeleton (or pose) tracking and pose detection.

[Moohasha]

Thanks for the explanation.
I wasn't aware that the OpenNI package already included that much useful tools for skeleton tracking.

Yeah, it gives you quite a bit. It also gives you hand positions which I wanted to use to control a cursor so you don't have to use the mouse, I just ran out of time and it was a low priority. But unless you want to do some more advanced image processing or do it all manually, you never really have to mess with the raw RGB or depth buffers.