[GSoC 2011 - Accepted] Unit Testing Framework

Threads related to Google Summer of Code
Post Reply
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA

[GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian » Tue Mar 08, 2011 8:33 am

Project Proposal (rough draft)

I would like to add a visual unit testing framework to OGRE.

Traditional unit testing doesn't help much in a context like OGRE, given the sheer size and heavy dependence on graphics API's. The best way of doing tests without needing significant modifications or mock rendersystems or something to that effect, appears to be a visual solution, wherein test screenshots are compared between builds.

I intend to use the existing sample framework as a a basis, since tests aren't too far off from samples and it just seems like a logical choice. There would be two primary components to the project:
  • The TestContext
    • Extended from SampleContext, it is to tests what the SampleBrowser is to samples.
    • Initially just runs through the tests as quickly as possible taking screenshots and then exits.
    • However, in weeks 6 and 7 I'll add the ability to manually inspect individual tests, select various options, launch the image comparison tool, etc.
  • The Image Comparison Tool
    • Built as a Sample for use with the existing SampleBrowser (and later, the TestBrowser itself).
    • Initially just allows you to manually flip through sets of images and compare them.
    • Later in the schedule some more advanced comparison tools (various methods of overlaying and diffing, some automated comparison that can give a quick summary of a whole image set, etc).
(If I get time, I may throw together a couple mockups of what these would look like.)

The primary focus is on making a useable, well-documented framework, so I won't necessarily be making a lot of actual tests (aside from a few here and there to test the workflow).

The creation of actual tests will be very similar to creating a regular Sample with the existing Sample Framework, with the addition of specifying when the test screenshot(s) should be taken, and using a fixed timestep to ensure deterministic behavior.

Due to hardware/driver differences this is not guaranteed to work across different machines, for the time being the goal is just to make it consistent with images generated on the same machine.

Unity successfully uses a similar system for their testing, see this blog post for some details about their implementation. I've attached an image of Unity's tests.

How will this project benefit OGRE users?

While it doesn't add any flashy features that would be visible in an end product, it would certainly help in the development and testing of OGRE which is definitely good for users. It would also make testing a lot easier for a user who needs to modify OGRE. Additionally, the image comparison tool would allow users to compare test shots from their own projects (for instance, it could be very helpful in shader debugging or something to that effect).

Is this project within the core scope of OGRE?

Definitely, as a testing framework it's obviously meant to be used directly with OGRE.

Is it realistic to achieve over a summer?

I believe so.

The main area of concern is probably the image comparison tool, since tools inevitably entail a lot of polish and documentation; but given the excellent sample framework and SDK trays stuff already in place, I think it should be feasible. Even if some of the nicer features can't be implemented in time, I can always fall back on a simpler version (I've purposely planned to make a very basic, but functional and deliverable version early on, and then add additional features towards the end).

Schedule

I will have school and exams until June 9th, so I'm aiming to go a little slower for the first weeks and catch up in the following weeks. Other than that, however, I have no vacation plans or other obligations.
  • April 25th – May 23rd -
    • Get to know mentor.
    • Get more familiar with the sample framework, Mercurial, CMake, etc.
    Week 1 (May 23rd) –
    • Start coding.
    • Create “VisualTest” class that will form the basis for tests. Derived from the the existing SampleFramework's Sample class, with the addition of options for screenshots, and taking various measures to ensure animations/particles/etc are deterministic (I've already been able to do this by modifying the existing playpen sample base class).
    • Convert the existing playpen samples to use this (the 10 or so that are already set up for the sample browser, and should be trivial to convert).
    Week 2 (May 30th) –
    • Write a simple TestContext that automatically runs through a set of tests (no extra menu/options stuff yet, just runs straight through the tests taking the test images, and exits).
    Week 3 (June 6th) –
    • My final exams are this week, so I will be mostly unavailable, I'll probably end up tinkering and testing a little bit anyways, but nothing definite.
    Week 4 (June 13th) –
    • Create a very basic image comparison tool. At this point it will just allow you to flip through images and manually compare them. Created as a sample plugin, so it can be used with the existing browser, and eventually be run straight from the TestContext.
    Week 5 (June 20th) –
    • Create a new test or two, use this to start working out some of the documentation, and work out any kinks in the workflow.
    Week 6 (June 27th) –
    • Add a menu to the TestContext (allows you to set various options before running the tests, manually launch individual tests, open the image comparison tool, etc).
    Week 7 (July 4th) –
    • Polish and finish up the TestContext's UI and related features.
    Week 8 (July 11th) –
    • Mid Term evaluation
    • Write some image comparison code that can be run automatically (outputs percent differences between images, various other stats).
    Week 9 (July 18th) –
    • Add additional functionality to the image comparison tool:
      • Various modes of overlaying and comparing image sets (see github's new image comparison tool for some potential functionality)
      • Use the automated comparison code from the previous week to give some overall stats about image sets and perhaps simple visualizations thereof.
    Week 10 (July 25th) –
    • Polish and document the comparison tool.
    Week 11 (August 1st)
    • Convert more old playpen tests over to the new system (certainly not all of them, but enough to adequately test the framework and work out any potential issues).
    Week 12 (August 7th) –
    • Overflow for anything that needs more work, more documentation, cleanup, etc.
    • If there's extra time, look into integrating it with OGRE's existing unit tests.
    Week 13 (August 14th) –
    • August 15th is the suggested pencils down date.
    • Final documentation and cleanup.
    End August 22nd
Future Extensions

Since the idea here is to create a solid framework, rather than a definite set of tests, the obvious next step would be to write a set of useful, general tests using it.

Some sort of continuous integration server and perhaps a repository of reference images could also be useful next steps.

Why You're The Person For This Project

I'm a 19-year-old student in the US, I'm studying Computer Science at the University of Washington in Seattle.

I've been using C++ for over 4 years now, and OGRE for almost as long. I've been pretty much constantly working on some project or another using OGRE over the years. You can see some of my past projects here (all the projects listed there use OGRE).

I have some code up on my github, much of it is really gross 48-hour Ludum Dare code, but “OryxEngine”, a game engine I've been working with for a number of months is pretty clean.

I'm not an expert by any means, my programming experience is mostly self-taught at this point, but I simply love programming and 3d development. I love a good challenge and I'm eager to learn.

I unfortunately haven't been very involved with the OGRE community lately, but I'll try and change that. In terms of other community involvements, I'm pretty active on the TigSource forums, and I actively participate in Ludum Dare (using OGRE, as an added bonus) every few months.

Why OGRE?

I've been using OGRE for years; it's easily been the single most influential thing on my coding style and learning process over the past years. I think it'd be incredible to be able to give back and try to return the favor.

Anything Else

Nothing else comes to mind, but I'll probably add more to this application before I officially submit it.
Attachments
unity_test-shots.png
Unity's visual tests.
Last edited by Praetorian on Mon Apr 04, 2011 2:07 am, edited 4 times in total.
0 x
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page

User avatar
so0os
Bugbear
Posts: 833
Joined: Thu Apr 15, 2010 7:42 am
Location: Poznan, Poland

Re: [GSoC 2011] Unit Testing Framework

Post by so0os » Tue Mar 08, 2011 8:53 am

Sorry for being prejudicial, but unit testing isn't suited for video game development imo.
0 x
Sos Sosowski :)
http://www.sos.gd

User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA

Re: [GSoC 2011] Unit Testing Framework

Post by Praetorian » Tue Mar 08, 2011 9:26 am

so0os wrote:Sorry for being prejudicial, but unit testing isn't suited for video game development imo.
Well, it was listed as wanted on the wiki, so someone wanted it :) And it may not be well suited to game development, but I think for something like Ogre that's pretty stable feature-wise (and those features are more easily quantifiable than a game's (basic rendering stuff should always look the same)), it'd be helpful to be able to make sure the basic features all still work the same after revisions.

The discussion here seems relevant: http://www.ogre3d.org/forums/viewtopic. ... 3&p=369112
0 x
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page

CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
Contact:

Re: [GSoC 2011] Unit Testing Framework

Post by CABAListic » Tue Mar 08, 2011 9:28 am

Conventional unit testing isn't, but rendering a predefined scene and comparing to previous results (screenshots) would be a very powerful tool to ensure that changes to the codebase don't break anything. This is one of the more important projects, imho.

However, I would forget about the Boost unit testing, we will need something custom for this purpose. Best try to find examples of visual tests and use those as a model.
0 x

User avatar
so0os
Bugbear
Posts: 833
Joined: Thu Apr 15, 2010 7:42 am
Location: Poznan, Poland

Re: [GSoC 2011] Unit Testing Framework

Post by so0os » Tue Mar 08, 2011 9:32 am

Aah visual unit tests, that sounds pretty cool. I was thinking of the "write everything twice" unit test thingie. Thereby, I approve ;)
0 x
Sos Sosowski :)
http://www.sos.gd

User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA

Re: [GSoC 2011] Unit Testing Framework

Post by Praetorian » Tue Mar 08, 2011 9:41 am

CABAListic wrote:Conventional unit testing isn't, but rendering a predefined scene and comparing to previous results (screenshots) would be a very powerful tool to ensure that changes to the codebase don't break anything. This is one of the more important projects, imho.

However, I would forget about the Boost unit testing, we will need something custom for this purpose. Best try to find examples of visual tests and use those as a model.
Alright, thanks for the feedback. I was thinking a conventional unit testing framework like boost's might make a good basis to build the rendering tests on top of, but I guess the conventional tests really don't serve enough of a purpose to merit the extra dependency. I've bookmarked a few sources on visual testsing already, so I'll start digging into those.
0 x
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page

User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4291
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 2
Contact:

Re: [GSoC 2011] Unit Testing Framework

Post by spacegaier » Tue Mar 08, 2011 10:15 am

Just as a sidenote: Assaf once found this: http://supercell.osuosl.org/

Might in the long run be an option for where to generate the reference screens.
0 x
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...

User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA

Re: [GSoC 2011] Unit Testing Framework

Post by Praetorian » Tue Mar 08, 2011 10:46 am

spacegaier wrote:Just as a sidenote: Assaf once found this: http://supercell.osuosl.org/

Might in the long run be an option for where to generate the reference screens.
Oooh, that's interesting... (haha, I have a few friends going to OSU, they get all the cool open source stuff... :? )

From some cursory research it looks as though one set of overall references might not be enough due to driver/hardware/etc discrepancies. Adding some tolerance to the screen comparisons might work, but how do you decide what the generic, middle-ground reference set should be? Judging by how Unity did it (basically buying a computer with every chip they wanted to target and generating references for each) there might be more variance between gpus/drivers than I'd originally suspected... perhaps building up a few general sets (say, generic ps2.0-level nvidia card on dX9 etc etc) and then leaving some tolerance in the comparisons would work? Seems like we'd still wind up with an awful lot of images to keep track of... and a need for a lot of testing hardware.

I dunno, I should really be sleeping and/or doing literature homework right now, but when I have time I might run some preliminary tests on a few machines and get a feel for how much the screenshots can vary based on hardware and drivers and such...
0 x
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page

User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4291
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 2
Contact:

Re: [GSoC 2011] Unit Testing Framework

Post by spacegaier » Tue Mar 08, 2011 11:13 am

Well, I guess we could design it more community driven as well. So a simple executable that everyone (also without SDK or any coding setup) can run that will then locally create the screens and transfer them to our server (or whereever we will handle of these things then). Then some sort of algorithm would compare the image to others we already obtained with the same / similar configuration and and increase our statistic that this is the reference outcome if it matches the previously as refence declared screenshots (e.g. by the team or expert users or whomever) or flag as possible "bad screen".

So that in the end we would basically have two lines: "Real" confirmed reference images provided by Ogre/team and what ever hardware we have and a second line of screens submitted by regular users/anyone out there that get compared with all other shots from similar hardware as some sort of not official refrence, but "crowd reference" with some statistics (likeliness of being "perfect", number of submitted shots this decision is based on, ...).

However, this would require some sort of server solution (application, space, traffic) as opposed to that we ship the reference screens with the SDK and then locallcy compare them to the shots locallcy created by our test utility.
0 x
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...

User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 2653
Joined: Wed Sep 24, 2003 8:07 am
Location: Haute Garonne, France
Contact:

Re: [GSoC 2011] Unit Testing Framework

Post by tuan kuranes » Tue Mar 08, 2011 12:00 pm

That framework could help ogre developpement (comparing screenshot would be helping a lot those making new renderer or scenemanager plugins. ) and ogre users (updating you app/game to a new version of Ogre with a set of visual tests would give very fast feedback on any Ogre bug.)

- A first simple step is using the sample framework and the playpen project to generate a set of image for each playpen test (playpen test), and providing a tool to display each against an history of screenshots. Visual diff would be human based at this step

- Second step would be adding semi-automated diff that could be done in the sample browser providing some helper tools (diffing colors, depth, edge, thresold, interleaving, histogram, etc.)

- As a final optional step, if enough time, full automated diff integrated into unit testing. It's only relevant on local, but is still pretty useful. Ogre does use cpptest and imho can still use that. Ogre does use cmake, and could very easily use CTest.

Imho, server side is interesting, but really would be too large a scope for a gsoc.
0 x

User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4291
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 2
Contact:

Re: [GSoC 2011] Unit Testing Framework

Post by spacegaier » Tue Mar 08, 2011 12:18 pm

tuan kuranes wrote:Imho, server side is interesting, but really would be too large a scope for a gsoc.
Yes, think so as well. That would need a second student who would cover that in order to build this fully integrated server solution.
0 x
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...

User avatar
Zonder
Ogre Magi
Posts: 1127
Joined: Mon Aug 04, 2008 7:51 pm
Location: Manchester - England
x 19

Re: [GSoC 2011] Unit Testing Framework

Post by Zonder » Wed Mar 09, 2011 9:40 am

I would think the screen comparison would be used between major releases.

The screens are generated on a stable version then compared to the same screens from the dev branch. So that way it can be done on the same hardware also an end user could also test between branches with their own projects as well.
0 x
There are 10 types of people in the world: Those who understand binary, and those who don't...

User avatar
Wolfmanfx
OGRE Team Member
OGRE Team Member
Posts: 1525
Joined: Fri Feb 03, 2006 10:37 pm
Location: Austria - Leoben
x 1
Contact:

Re: [GSoC 2011] Unit Testing Framework

Post by Wolfmanfx » Wed Mar 09, 2011 10:55 am

Maybe part of the project could also to setup continues integration (build server) in combination with automated tests.
0 x

User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4291
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 2
Contact:

Re: [GSoC 2011] Unit Testing Framework

Post by spacegaier » Wed Mar 09, 2011 11:35 am

Wolfmanfx wrote:Maybe part of the project could also to setup continues integration (build server) in combination with automated tests.
I think that this should be another project to not make this idea here too big. But those two ideas are partly heading in same directions (the server part) so the two students covering those ideas will have to work toghether in some parts.
0 x
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...

User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA

Re: [GSoC 2011] Unit Testing Framework

Post by Praetorian » Sat Mar 12, 2011 6:06 am

Thanks for the feedback so far! I'm stuck cramming for exams finals until next thursday or so, so I'm kinda distracted :? , but I've been digging around a bit more on this... (after exams I'll have break and be able to make a more solid proposal).
Zonder wrote:I would think the screen comparison would be used between major releases.
The screens are generated on a stable version then compared to the same screens from the dev branch. So that way it can be done on the same hardware also an end user could also test between branches with their own projects as well.
Yeah, I think that's gonna be the ideal situation for now, since a server thing is out of this project's scope, I'd definitely like to make things flexible enough that you could set it up to work with such a system without much trouble though.
tuan kuranes wrote:- A first simple step is using the sample framework and the playpen project to generate a set of image for each playpen test (playpen test), and providing a tool to display each against an history of screenshots. Visual diff would be human based at this step
Yeah, I'd definitely want take advantage of the existing sample framework and playpen stuff. I'd probably create a simple SampleContext that runs through the tests automatically taking screenshots at the allotted times (I don't think there's a way to render w/o a window(?) but it could just run through the tests quickly, with the option to pause and manually inspect something). I'd probably need to take some measures to make sure the timing is consistent and everything's deterministic.

For the tool for comparing images, I'd want to make it flexible enough to work for any sets of images, so there'll need to be some way of defining a 'set' of images and the history thereof.
tuan kuranes wrote:- Second step would be adding semi-automated diff that could be done in the sample browser providing some helper tools (diffing colors, depth, edge, thresold, interleaving, histogram, etc.)
Integrating the diffing tools into the sample browser is a great idea; it could give overall stats about the set of images, provide various color comparisons and such for individual images like you mentioned, etc. I know tools projects aren't generally very good for GSoC's but I'd definitely like to try and make something fairly robust for this part.
tuan kuranes wrote:- As a final optional step, if enough time, full automated diff integrated into unit testing. It's only relevant on local, but is still pretty useful. Ogre does use cpptest and imho can still use that. Ogre does use cmake, and could very easily use CTest.
Ahh, wow, not sure how I missed that Ogre is already using cppunit. But yeah it'd be a nice bonus to integrate the diffing stuff into that if there's extra time.
spacegaier wrote:
Wolfmanfx wrote:Maybe part of the project could also to setup continues integration (build server) in combination with automated tests.
I think that this should be another project to not make this idea here too big. But those two ideas are partly heading in same directions (the server part) so the two students covering those ideas will have to work toghether in some parts.
I'd definitely be willing to collaborate if another student wants to do a project focusing on that sort of thing.
0 x
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page

User avatar
syd
Gnome
Posts: 362
Joined: Thu May 01, 2008 1:55 am
Location: Paris, France

Re: [GSoC 2011] Unit Testing Framework

Post by syd » Sat Mar 12, 2011 10:02 am

My two cents about c++ unitTest frameworks, there's an interesting blog entry comparing them all. cxxTest was the winner, but later the author published his own unit test framework which is UnitTest++, which I tried, it appears to be lite and straightforward.

For Ogre, I think it would be great to use the plugin system to attach tests, all launched by a common GUI application, each test would share the same viewport.
It would be useful also to have a Visual Studio OgreTest project, that would attach itself to the OgreTests Gui process when launched, this way the Gui window is always kept open, and remain usable at debug times, so break point remain triggered.

I've been exploring about a unit test framework for ogre, and I believe something like this would drasticaly improve productivity on developpements of any Ogre ptoject.

I'm all for it
0 x

CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
Contact:

Re: [GSoC 2011] Unit Testing Framework

Post by CABAListic » Sat Mar 12, 2011 3:12 pm

Just a random thought: Instead of a dedicated server architecture, it might also be possible to use another (Mercurial) repository to store the reference screenshots. Anyone who'd like to run the tests could download the repository and let the test application compare his screenshots to the reference material.
If there is no suitabel reference material for his hardware, but by manual comparison he thinks that the tests ran correctly, he could then add his screenshots to the local repository, upload to a fork on BitBucket and issue a merge request with our reference repository.
0 x

User avatar
syd
Gnome
Posts: 362
Joined: Thu May 01, 2008 1:55 am
Location: Paris, France

Re: [GSoC 2011] Unit Testing Framework

Post by syd » Sat Mar 12, 2011 10:55 pm

CABAListic wrote:Just a random thought: Instead of a dedicated server architecture, it might also be possible to use another (Mercurial) repository to store the reference screenshots. Anyone who'd like to run the tests could download the repository and let the test application compare his screenshots to the reference material.
If there is no suitabel reference material for his hardware, but by manual comparison he thinks that the tests ran correctly, he could then add his screenshots to the local repository, upload to a fork on BitBucket and issue a merge request with our reference repository.
Right! (was think about that too) :)
This would also provide benchmark statistics for gpu/cpu
0 x

denisw
Gnoblar
Posts: 1
Joined: Sun Mar 20, 2011 1:26 pm

Re: [GSoC 2011] Unit Testing Framework

Post by denisw » Sun Mar 20, 2011 1:57 pm

Praetorian wrote: [*]The wiki mentioned screenshot comparison for some of the testing, which I think would be a good chunk of the work. I suspect there would be issues with minor driver/hardware variations causing slight differences in the images, but perhaps it could do automatic checking with some set level of tolerance and then prompt a manual review if there's a serious discrepancy? (or maybe maintain a database of 'correct' images from various hardware setups and the like? I seem to recall reading that Unity does something like this).
How about avoiding the problem of different GPU/driver configuration by using software rasterization of the underlying graphics API commands? For instance, there is the Gallium3D LLVMpipe software rasterizer (http://www.phoronix.com/scan.php?page=n ... &px=ODA1OA) which, due to the arcitecture of Gallium3D, potentially can be made to work with any graphics API (there is OpenGL support already). This would ensure that the renderings are always exactly the same regardless for the computer they are run on (the only left variable would be the software renderer's version) and myake simple bit-by-bit screenshot comparison 100% accurate.

Naturally, this approach works depends on the quality of the software rasterizer - especially the supported OpenGL/DirectX/etc. operations and shaders - and doesn't help to detect driver-specific problems; but on the other hand, it allows to clearly determine if a specific rendering bug is the fault of Ogre3D (if it fails with the software renderer) or the driver (if it works with the software rasterizer, but not with the driver).
0 x

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel

Re: [GSoC 2011] Unit Testing Framework

Post by Assaf Raman » Mon Mar 21, 2011 4:44 pm

denisw wrote:
Praetorian wrote: [*]The wiki mentioned screenshot comparison for some of the testing, which I think would be a good chunk of the work. I suspect there would be issues with minor driver/hardware variations causing slight differences in the images, but perhaps it could do automatic checking with some set level of tolerance and then prompt a manual review if there's a serious discrepancy? (or maybe maintain a database of 'correct' images from various hardware setups and the like? I seem to recall reading that Unity does something like this).
How about avoiding the problem of different GPU/driver configuration by using software rasterization of the underlying graphics API commands? For instance, there is the Gallium3D LLVMpipe software rasterizer (http://www.phoronix.com/scan.php?page=n ... &px=ODA1OA) which, due to the arcitecture of Gallium3D, potentially can be made to work with any graphics API (there is OpenGL support already). This would ensure that the renderings are always exactly the same regardless for the computer they are run on (the only left variable would be the software renderer's version) and myake simple bit-by-bit screenshot comparison 100% accurate.

Naturally, this approach works depends on the quality of the software rasterizer - especially the supported OpenGL/DirectX/etc. operations and shaders - and doesn't help to detect driver-specific problems; but on the other hand, it allows to clearly determine if a specific rendering bug is the fault of Ogre3D (if it fails with the software renderer) or the driver (if it works with the software rasterizer, but not with the driver).
I don't think we have to deal with the different GPU issue, we can assume that all the screenshots were produced on the same computer. Producing screenshots from "old" versions of the code is simple (compile an old version or run an old copy of an executable that saves the images).
I don't think we need a "store of old screenshots" - but perhaps a "store of old exes that creates screenshots".
0 x
Watch out for my OGRE related tweets here.

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel

Re: [GSoC 2011] Unit Testing Framework

Post by Assaf Raman » Mon Mar 21, 2011 4:48 pm

BTW: Software rendering support shouldn't be that hard if we need it, if we use mesa.
Software rendering can be useful in case we can have a free server that can run our tests - but doesn't have a GPU.
0 x
Watch out for my OGRE related tweets here.

User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA

Re: [GSoC 2011] Unit Testing Framework

Post by Praetorian » Tue Mar 22, 2011 1:29 am

Finally on break now, and I've been looking into this some more.

I modified the existing playpen base a little bit (hid the framerate indicator, set the camera to be fixed, changed it to use a fixed timestep, and added the ability to set times for screenshots) and so far I'm able to reliably take perfectly identical shots of the tests, even with animations and such. At the moment it's running in the existing sample browser, but the idea would be to make a separate SampleContext that can prompt for a few options and then run through the tests quickly and automatically.

I'll compile in Windows in a little while and test on a few different machines to see if everything stays deterministic.

It looks like a part of the work could be converting more of the old commented out playpen tests to be useable with the sample framework? Only a handful are included in the current playpen plugin.
Assaf Raman wrote:BTW: Software rendering support shouldn't be that hard if we need it, if we use mesa.
Software rendering can be useful in case we can have a free server that can run our tests - but doesn't have a GPU.
I can't say I have much experience with software renderers, but I'd definitely be willing to look into it if it seems like it'd be a helpful addition.
0 x
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page

User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 2653
Joined: Wed Sep 24, 2003 8:07 am
Location: Haute Garonne, France
Contact:

Re: [GSoC 2011] Unit Testing Framework

Post by tuan kuranes » Tue Mar 22, 2011 8:11 am

It looks like a part of the work could be converting more of the old commented out playpen tests to be useable with the sample framework? Only a handful are included in the current playpen plugin.
Not old commented code, those are also tests... that you revive by de-commenting them and commenting current ones to test another feature you need to check. That's the "manually" part that has to go away the faster. It slow all the tests development and also test check a lot... That's clearly why we need a TestBrowser...

About server part, you may want to discuss with tdev, as its proposal and experience could be invaluable : http://www.ogre3d.org/forums/viewtopic. ... 90#p422790

btw github added a nice image diff worth a check : https://github.com/blog/817-behold-image-view-modes
0 x

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel

Re: [GSoC 2011] Unit Testing Framework

Post by Assaf Raman » Tue Mar 22, 2011 9:23 am

As I see it - the important part of this project is creating a maintainable and extendable framework - with some tests that demonstrate the different features of the framework. I don't expect the student to cover "all" or even "many" cases. I want the focus to be on the quality of the code, its documentation and having a "working and useable output" for the project.
0 x
Watch out for my OGRE related tweets here.

User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA

Re: [GSoC 2011] Unit Testing Framework - Proposal draft is u

Post by Praetorian » Mon Apr 04, 2011 1:57 am

Alright, I'm still alive, sorry for going quiet for a little while there, I had some registration issues at school (I applied and got into my school's CS department on accelerated admission and then had to completely rearrange my schedule just before the quarter started...).

A proposal draft is up on the first post, I'd love any feedback y'all can offer (especially schedule-wise) :)
0 x
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page

Post Reply