[GSoC 2011 - Accepted] Unit Testing Framework

Threads related to Google Summer of Code
Post Reply
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

I think I'll begin working towards more meaningful comparison, starting by comparing pixel by pixel, and tracking some statistics about the avg/stddev/etc in color discrepancies. I'm not exactly sure what the options are as far as more advanced comparison, (I may look into something like PerceptualDiff, if time permits), but I think this simple approach should at least be a good start.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Why do we need anything more advanced then comparing pixel by pixel?
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Assaf Raman wrote:Why do we need anything more advanced then comparing pixel by pixel?
If two images are identical, but one is translated by one pixel then this would report a significant difference, even though the images are nearly identical. I suppose the main idea is checking if the image is identical though, so any extra info about how different the image is for a failed test, is just a bonus and doesn't necessarily need to be perfect.

I think this will work fine for this purpose, but a more advanced comparison might be a good later addition to allow for some more flexibility (perhaps there's a driver configuration that doesn't always produce perfectly identical images...).
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

The same driver configuration always produce perfectly identical images - as far I know.
Watch out for my OGRE related tweets here.
User avatar
Wolfmanfx
OGRE Team Member
OGRE Team Member
Posts: 1525
Joined: Fri Feb 03, 2006 10:37 pm
Location: Austria - Leoben
x 99
Contact:

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Wolfmanfx »

Maybe just prepare the test system so it can be extented in future. For example you could create an interface IImageValditator which gets two image memory buffer and can return information if this image are the same or not.
Just my 2 cents
CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
x 58
Contact:

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by CABAListic »

Assaf Raman wrote:The same driver configuration always produce perfectly identical images - as far I know.
Perhaps, but it could allow to create a more general set of reference pictures so that not everyone has to create their own set of valid references manually. The less hassle the testing involves, the more testing might be done :)
But it's a secondary issue for now.
User avatar
Thoran
Halfling
Posts: 94
Joined: Mon Dec 01, 2008 2:04 pm
Location: Germany
x 1
Contact:

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Thoran »

Assaf Raman wrote:The same driver configuration always produce perfectly identical images - as far I know.
What about rounding erros?I don't think you can exclude them completely, even with the same settings. So maybe a comparision of pixels within a delta range is possible, if Praetorian is not already doing it.

Thoran
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

CABAListic wrote:
Assaf Raman wrote:The same driver configuration always produce perfectly identical images - as far I know.
Perhaps, but it could allow to create a more general set of reference pictures so that not everyone has to create their own set of valid references manually. The less hassle the testing involves, the more testing might be done :)
But it's a secondary issue for now.
That was a thought as well; but yeah, for now it isn't a huge concern, I'm just going to leave it open to be extended later, as Wolfmanfx suggested.
Thoran wrote:
Assaf Raman wrote:The same driver configuration always produce perfectly identical images - as far I know.
What about rounding erros?I don't think you can exclude them completely, even with the same settings. So maybe a comparision of pixels within a delta range is possible, if Praetorian is not already doing it.

Thoran
In theory (and in practice, from what I've done thus far), if all the input is kept consistent, all the rounding and such should always be the same (there should be no way that (on a single machine) the same floating point operation on the same operands would produce a different result). I can't think of any exceptions to this at this point (perhaps a multi-GPU setup with slightly different GPUs?), though it does seem like it'd be easy to have an uninitialized value or something somewhere that would throw everything off (though I suspect you'd have larger problems in such a case). On separate machines with different processors/architectures/etc though, it's definitely unlikely that things would stay consistent.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Noman
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 714
Joined: Mon Jan 31, 2005 7:21 pm
Location: Israel
x 2
Contact:

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Noman »

I don't know if it has been mentioned in this thread, but there is a nice post by the graphic lead in unity about testing graphics code (and what they ended up doing) :

http://altdevblogaday.com/2011/06/17/te ... ars-later/

Good read IMO
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Noman wrote:I don't know if it has been mentioned in this thread, but there is a nice post by the graphic lead in unity about testing graphics code (and what they ended up doing) :

http://altdevblogaday.com/2011/06/17/te ... ars-later/

Good read IMO
Thanks for linking it, I'd read about Unity's testing earlier, but this post is new to me.

One thing that stuck out to me:
On many platforms you have to allow a couple of wrong pixels because many consumer GPUs are not fully deterministic it seems.
I'm guessing it's probably more on the integrated/mobile/etc side of things, but good to know that there may be a need for some sort of tolerances in some cases... perhaps at some point I'll test on some lower end machines/GPUs and see if I can reproduce anything like this.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
andrei_radu
Google Summer of Code Student
Google Summer of Code Student
Posts: 55
Joined: Fri Mar 18, 2011 8:37 pm
x 8

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by andrei_radu »

One possible reason for this is that most video memory(basically on every consumer card) lacks ECC protection. That means bits in the V RAM can get corrupted by cosmic radiation and such(I'm not kidding, it is possible), not to mention they have smaller refresh times than normal DDR. Hence one source of non-deterministic input.
My Google summer of code 2011 topic: Modern Illumination Techniques
My Google summer of code thread
My Google summer of code wiki page
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

andrei_radu wrote:One possible reason for this is that most video memory(basically on every consumer card) lacks ECC protection. That means bits in the V RAM can get corrupted by cosmic radiation and such(I'm not kidding, it is possible), not to mention they have smaller refresh times than normal DDR. Hence one source of non-deterministic input.
Wow, that's interesting; makes sense though.

I've added several common image comparison metrics that get displayed when tests fail (and I'll be adding a means by which to set thresholds for making tests pass/fail based on these). At the moment it gives you the absolute number of incorrect pixels, the mean squared error (MSE), peak signal-to-noise ratio (PSNR), and the structural similarity index (SSIM).

Here's another sample output: http://rileyadams.net/gsoc/July5/out.html
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

What is the plan for this weekend?
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

I think I'll be working on updating the documentation (it's getting to about the halfway point). I've also gotten some occasional, slight differences in a couple of the images over these past couple days (generally a pixel or two, but as many as 32 once), so I think I'll be trying to figure out if there's anything that can be done to help with this (test on a couple more machines perhaps, see if I figure out a good tolerance for a 'passing' test).
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Alright, I've updated the wiki page a bit, I'll still need to add thumbnails for the list of tests, and add to the command line option documentation as I add functionality, but most of it is set for the time being.

I've also been trying to reproduce the non-identical images issue. I've tested on a couple more machines and haven't been able to reproduce it (or even in Windows on my primary machine), so I think it's probably iffy Linux drivers that are to blame in this case. Even in Linux on this machine it's not reliably reproduceable, it seems to only happen if I generate a reference image set, and then reboot a few times, and wait a while.

I think it'll probably work fine to just set a threshold of a few pixels, and have an option for that to raised in the case of more sensitive hardware/drivers. From the sounds of that article on Unity's framework, these kinds of differences shouldn't ever be more than a few pixels.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

You passed the mid-term, congratulation. :)
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Whoohoo! :)

Alright, so for the next day or two, I think I'm going to be doing some general cleanup/reorganizing in a few places. After that I'm not entirely sure where to focus, perhaps I'll add/convert some more tests.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Yes, that is what I was going to suggest, convert as many as you can.
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Alright, I finished up the cleanup I was doing, so next up will be more playpen conversions.

There's a lot of playpen tests that'll work really well here. However, I'm noticing that some seem to use media assets that are no longer a part of the SDK, so I may need to dig into some older versions, or recreate/make substitutions for some assets.

Also, just to note, I probably won't be available most of tomorrow (going away party for a friend who's moving across the country), so I probably won't be able to start too much work on these conversions until after then.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4304
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 135
Contact:

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by spacegaier »

Praetorian wrote:However, I'm noticing that some seem to use media assets that are no longer a part of the SDK, so I may need to dig into some older versions, or recreate/make substitutions for some assets.
You mean some media assets have been removed in 1.8?
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Well, can you list the missing assets here in a thread - by sample?
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

I looked a bit closer and there aren't as many instances of this as I'd originally thought (by some coincidence the first two tests I'd picked to convert both happened to be missing something...).

Here's what I found missing in a skim of the meshes/textures (I didn't check for the materials used, and since the file of playpen tests is ~9000 lines I probably missed something...):

testTransparencyMipMaps:
-ogretext.png
testCthNewBlending:
-ogrehead_2.mesh
testAnimation:
-skeletonandpose.mesh
testAnimationBlend:
-sticky.mesh
testSimpleMesh:
-xsicylinder.mesh
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Here is what I found:
ogretext.png - change the sample to use sdk_logo.png (it is in Samples\Media\packs\SdkTrays.zip)

Leave the ones with issues out for now, leave them for the end.

On the topic of tests, here are two tests that I think that should be included also in the end result:
1. resources without a unique name - use this sample code as base, and use these models.

2. Stencil Glow Demo - based on this code - http://assaframan.googlepages.com/StencilGlow_src.zip
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Productive day :). I added the stencil glow test, and converted 47 playpen tests (so there's now a total of 64 tests). At this point I'm just trying to get as many working as possible, so I'll have to go back to a lot of them to tidy up, tweak camera angles, etc.

Here's an output (just a warning, it's 13+mb, even with very small renders...): http://rileyadams.net/gsoc/July18/out.html
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Very very nice.
Watch out for my OGRE related tweets here.
Post Reply