Page 1 of 1

proposal - Comp. vision toolset for game development

Posted: Sat Mar 24, 2007 3:25 pm
by jarek108
this is first sketch. please comment.

My goal is to bring together results of state of the art academic research in computer vision algorithms and using my knowledge about them develop set of tools to form a platform or a for various tasks is game development.

Ultimate long term aim is to enable full body 3d posture analysis from one/multiple web cam quality video sources to be used (among others) for:
-full body game control,
-hand interface control(finger pointing vs mouse),
-animation creation support(video -> simple moving 3d model)
-applications for non game fields would be a second unlimited dimension.

Despite of the fact that partial algorithms(or even entire commercial systems) are known to be implemented such system is a goal for years not months. I do NOT intend to reach for ultimate aim during gSoC, but i DO believe I have abilities and knowledge to build during that time a modular toolset with some specific algorithms forming well documented platform preforming real tasks including:
extraction and recognition of known(kept in library) shapes from image/video
converting videos of limited subset of human motion -> simple 3d posture sequences (link to closest existing system: ... ck/#dancer )
tools for expanding library of shapes and human poses
tools for introducing new probability distribution/schema of human motion sequences WITHOUT a need to go into probabilistic/Markov chain details
easy interface for adding/replacing modules on any level
set of low level operation needed in system any way(like state of the art image noise elimination (link), tool for cutting shapes out of background(like smart pen in hotoshop, but much smarter: (link) and others)
if time allows some really small toy example body game controller

Computer vision has achieved the level of being able to support the game industry long time ago, but it is just now that commercial solutions using that technology begin to appear, and open source is even further behind.
One of the reasons for that is lack of contact between those producing technology and those who could find it useful. I believe that using ogre to establish a platform on witch academic research and game programing can meet is going to allow game programmers to use tools that they may not build because of lack of time/ theoretical knowledge, playing with them and making them a part of their projects. For comp. vision. scientists it will be a chance to compare their algorithms with others in one environment on “real lifeâ€

Posted: Sat Mar 24, 2007 5:43 pm
by sinbad
Whilst this is a very interesting project, I don't think it has much to do with Ogre. Remember that projects in the SoC should be aligned with the goals of the mentoring organisation, and computer vision isn't something we'd ever be adding to Ogre. We are not a game engine, remember.

Posted: Sat Mar 24, 2007 6:05 pm
by Lioric
Yes this project/lib should interface with the application framework, not with the graphics rendering susbsystem

Posted: Sat Mar 24, 2007 6:53 pm
by jarek108
ofcourse i`m aware that this toolset would never be a part of ogre, rather it would use ogre and be used to create animations for ogre / provide user control for games made in ogre.
i though that support tool like that would fit here, i do not know any better place in gSoC. should i understand that this project has really small chance to get accepted sinbad?
perhaps you could make a suggestion what i could do to make it fit better as i invested some time in it and really want to be a part of gSoC.

Posted: Sat Mar 24, 2007 7:21 pm
by sinbad
Whilst we do like to see new tools, we're really talking about tools that directly support the use of OGRE - this tool is really 'higher up the stack' in that it's helping you produce the final application, whether that's a game or otherwise. As such its connection with OGRE is actually very tenuous and thus no, it doesn't have a high chance of making it into our list.

Like I said, it's an extremely interesting project but it isn't aligned with what OGRE is at all - one of those things I'd gladly see in the Showcase forum but not something that should be a part of the Summer of Code (for us, anyway) because it's too tangental to what our project is doing. And I don't think it really can be aligned with OGRE, it's addressing a completely different problem. SoC isn't about using the mentoring project to do something cool, it's about enhancing the project itself (and some tools can do that, but this one is a little too far out in the fringes).

Sorry you've already spent time on this but it's only fair for us to be totally honest with you about the suitability of this project for an OGRE SoC project.

Posted: Sat Mar 24, 2007 8:37 pm
by jarek108
damn. it`s a shame.

but i have to say i see your point and propably you are right. i`ll be back with revenge with IK proposal that xavier posted ;).

i`ll return to this project some other time. that`s a promise.

Posted: Mon Mar 26, 2007 12:35 am
by Fredz
You project is quite interesting, but I don't understand why you do need prior knowledge about human motion to extract animations. One of your papers even says that 3D motion is underdetermined when using a single camera.

There has been a lot of research papers published showing methods about how to recover 3D information from 2D images without prior knowledge of the scenes, why wouldn't it be possible to extend this to 3D animation extraction ?

Posted: Thu Mar 29, 2007 5:47 pm
by jarek108
while im sure there are some general methods for retriving 3d from 2d without previous assumptions about model of body/motion it is not realistic to expect any good results from them in human motion rec. because like you say the 2d -> 3d transition is not well determined meaning usually there are limitles possibilities of interpreting 2d pose into 3d. Normaly all methods have some assumptions about what is possible for human motion and what is not and use it as a distribution of propob. and try to extract the most propoble solution. Assumptions like that are really nececery, they are even present in our brains (you have to often imagine how the person looks like from the side you are notable to observe in order to understand its pose, you can read about how full of assumptions is our perception process in prof Pinker's book i think it was "how mind works" or sth.)

Posted: Thu Mar 29, 2007 6:14 pm
by Fredz
I didn't say that the 2d -> 3d transition was not well determined, I just said that's what you wrote in your paper. There are many experimental results that show the contrary, and it's a subject well covered in the computer vision research field (epipolar geometry, features detection, fundamental matrix, etc.).

I just wanted to know if you knew about those results and if you tried to use them.