SampleFramework as a UnitTesting tool

[Shadow007]

Hi,

I've begun to try and use the new Samples framework to create a "visual UnitTesting" tool for Ogre :
The goal would be to provide for each sample a set of reference {"state" and "screenshot"} pairs,
and by running the SampleBrowser with each reference state, compare the screenshot with the reference one.

So far, I've added on my version a few capabilities to the SdkSample framework :
- a key that takes a screenshot after having disabled the overlays
- a key to save the state to an XML file (named after the Sample)
- a key to restore the state from the XML file.

By now, in addition to the existing "camera position and orientation" (for CS_Freelook), I added the possibility to save/restore the camera's position in CS_Orbit mode if the target is the RootSceneNode (which is the case of most samples)

I also added a generic save/restore to the SDKTrays widgets value.
(with a few snags for the FacialAnimation).

I next tried to tackle the problem of time dependant animations, with the CubeMapping sample. I save/restore the fishes animation quite well, but have a problem with the skydome, which seems to be a "hardware managed" animated texture.
Do any of you know of a way to retreive/restore the texture's state from the driver/card ?
Alternatively, I guess I could create an alternative technique/material without the animation...

After I have found a technical way to bypass all these problems (which could then be applied to a great many of the samples), I plan to go forward and code the screenshot comparison, difference presentation etc.

[tuan kuranes]

Great Idea and work !

I next tried to tackle the problem of time dependant animations, with the CubeMapping sample. I save/restore the fishes animation quite well, but have a problem with the skydome, which seems to be a "hardware managed" animated texture.

Not sure to get the problem you have.

A sort of global 'settime' that set all Ogre dependent object time that work for Node animation should work for texture scroll animation ?

(btw, nice round up on graphic testing techniques here: http://www.gamerendering.com/category/s ... gineering/ )

[Shadow007]

Thanks, I had seen these posts... and will try and take them into account

As for the "texture scroll animation", with a few more searches, I just found out about the "ControlerManager::getElapsedTime() / setElapsedTime" functions, which seems to be what I had been looking for...

Problem is my working machine is a netbook with absolutely abysmall performances (compiling / grepping are LOOOOOOOOONNNNNNNGGGG operations ! ).

[Assaf Raman]

Nice!
We added a similer idea to the Help Requested page a few months back.

A unit testing framework
An infrastructure that will enable us to define tests for OGRE and run them one after the other so we will know that status of the version, because OGRE is mainly graphic - a part of the infrastructure will be to enable running the test in a "record" mode to record a positive state - and take a screen shot, this screen shot will be compared to the result of the same test running in "test" mode.

I really think this can help the project.

[Shadow007]

Yep, I hope when it's finished you'll be able to use it...

I'll try and make a first preliminary version in a little month (yes, only working 1 hour here and there, on vastly underpowered HW), then (if I'm able to have a better dev environment) clean the code and submit a patch ...
At the moment, I'm just trying to clean the big hurdles out of the way before doing the nitty-gritty details...

[sinbad]

Nice. I asked Bill Hoffman from CMake/CTest/VTK hwo they do this, because they run visual unit tests and I wondered how they handled variations between hardware (because there are always subtle ones). He said they have a database of iamges from different hardware setups where essentially a user can submit a new 'correct' one where there's a variation that doesn't matter, and the unit test just has to match one of these images to pass.

As regards controlling time, all you have to do is suppress frame events, or call ControllerManager::setTimeFactor to globally alter the controllers perception of the passage of time (these control texture animation).

[CABAListic]

I wonder, though, if the Samples really are the best place to add such a capability? The new SampleBrowser is designed and specialised to render interactive scenes which serve as a demonstration. The point of a unit testing tool is very different from that.

[Shadow007]

I wonder, though, if the Samples really are the best place to add such a capability? The new SampleBrowser is designed and specialised to render interactive scenes which serve as a demonstration. The point of a unit testing tool is very different from that.

Yep, I'm well aware of that, and agree that the SampleBrowser may not be the best way to adress the specific needs of a UnitTesting Tool, but the Samples framework allow a few things :
1- it at least partly adresses the need of creating the unit tests : Most of the Samples can already be used
2- Since it is interactive, it can help the process of creating the reference testcases with for each sample a list of separate states ( regarding positions, widget values etc)...
3- It already presents the facility to independently load/start the distinct samples.
4- It is quite modular and easlily extendable to add new samples/test cases.
5- By adding new Samples, one can easily add new UnitTests (and Vice-Versa) (though interest can be varying)
...

Globally, it adds a non-constraining option to the samples workflow, without adding too much constraints, and avoids some duplication through the need of a dedicated Unit-Tests plugins production.

[Shadow007]

Nice. I asked Bill Hoffman from CMake/CTest/VTK hwo they do this, because they run visual unit tests and I wondered how they handled variations between hardware (because there are always subtle ones). He said they have a database of iamges from different hardware setups where essentially a user can submit a new 'correct' one where there's a variation that doesn't matter, and the unit test just has to match one of these images to pass.

As regards controlling time, all you have to do is suppress frame events, or call ControllerManager::setTimeFactor to globally alter the controllers perception of the passage of time (these control texture animation).

Regarding different platform (hardware+driver) setups, do you prefer this approach of storing a list of valid images for each sample state (which is simple) , or should go a little bit further and keep for each platform a list of its corresponding screenshots ?

Regarding screenshots, in what format should I store them ? is png OK ?

Thanks, regarding controlling time, I'll keep that in mind.