[GSoC 2011 - Accepted] Unit Testing Framework

Threads related to Google Summer of Code
Post Reply
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian » Sat Jul 02, 2011 8:33 pm

I think I'll begin working towards more meaningful comparison, starting by comparing pixel by pixel, and tracking some statistics about the avg/stddev/etc in color discrepancies. I'm not exactly sure what the options are as far as more advanced comparison, (I may look into something like PerceptualDiff, if time permits), but I think this simple approach should at least be a good start.
0 x
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman » Sat Jul 02, 2011 9:30 pm

Why do we need anything more advanced then comparing pixel by pixel?
0 x
Watch out for my OGRE related tweets here.

User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian » Sun Jul 03, 2011 6:17 am

Assaf Raman wrote:Why do we need anything more advanced then comparing pixel by pixel?
If two images are identical, but one is translated by one pixel then this would report a significant difference, even though the images are nearly identical. I suppose the main idea is checking if the image is identical though, so any extra info about how different the image is for a failed test, is just a bonus and doesn't necessarily need to be perfect.

I think this will work fine for this purpose, but a more advanced comparison might be a good later addition to allow for some more flexibility (perhaps there's a driver configuration that doesn't always produce perfectly identical images...).
0 x
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman » Sun Jul 03, 2011 7:41 am

The same driver configuration always produce perfectly identical images - as far I know.
0 x
Watch out for my OGRE related tweets here.

User avatar
Wolfmanfx
OGRE Team Member
OGRE Team Member
Posts: 1525
Joined: Fri Feb 03, 2006 10:37 pm
Location: Austria - Leoben
x 1
Contact:

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Wolfmanfx » Sun Jul 03, 2011 8:13 am

Maybe just prepare the test system so it can be extented in future. For example you could create an interface IImageValditator which gets two image memory buffer and can return information if this image are the same or not.
Just my 2 cents
0 x

CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
Contact:

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by CABAListic » Sun Jul 03, 2011 8:18 am

Assaf Raman wrote:The same driver configuration always produce perfectly identical images - as far I know.
Perhaps, but it could allow to create a more general set of reference pictures so that not everyone has to create their own set of valid references manually. The less hassle the testing involves, the more testing might be done :)
But it's a secondary issue for now.
0 x

User avatar
Thoran
Halfling
Posts: 94
Joined: Mon Dec 01, 2008 2:04 pm
Location: Germany
Contact:

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Thoran » Sun Jul 03, 2011 10:56 am

Assaf Raman wrote:The same driver configuration always produce perfectly identical images - as far I know.
What about rounding erros?I don't think you can exclude them completely, even with the same settings. So maybe a comparision of pixels within a delta range is possible, if Praetorian is not already doing it.

Thoran
0 x

User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian » Sun Jul 03, 2011 5:37 pm

CABAListic wrote:
Assaf Raman wrote:The same driver configuration always produce perfectly identical images - as far I know.
Perhaps, but it could allow to create a more general set of reference pictures so that not everyone has to create their own set of valid references manually. The less hassle the testing involves, the more testing might be done :)
But it's a secondary issue for now.
That was a thought as well; but yeah, for now it isn't a huge concern, I'm just going to leave it open to be extended later, as Wolfmanfx suggested.
Thoran wrote:
Assaf Raman wrote:The same driver configuration always produce perfectly identical images - as far I know.
What about rounding erros?I don't think you can exclude them completely, even with the same settings. So maybe a comparision of pixels within a delta range is possible, if Praetorian is not already doing it.

Thoran
In theory (and in practice, from what I've done thus far), if all the input is kept consistent, all the rounding and such should always be the same (there should be no way that (on a single machine) the same floating point operation on the same operands would produce a different result). I can't think of any exceptions to this at this point (perhaps a multi-GPU setup with slightly different GPUs?), though it does seem like it'd be easy to have an uninitialized value or something somewhere that would throw everything off (though I suspect you'd have larger problems in such a case). On separate machines with different processors/architectures/etc though, it's definitely unlikely that things would stay consistent.
0 x
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page

User avatar
Noman
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 714
Joined: Mon Jan 31, 2005 7:21 pm
Location: Israel
Contact:

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Noman » Sun Jul 03, 2011 6:18 pm

I don't know if it has been mentioned in this thread, but there is a nice post by the graphic lead in unity about testing graphics code (and what they ended up doing) :

http://altdevblogaday.com/2011/06/17/te ... ars-later/

Good read IMO
0 x

User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian » Sun Jul 03, 2011 7:14 pm

Noman wrote:I don't know if it has been mentioned in this thread, but there is a nice post by the graphic lead in unity about testing graphics code (and what they ended up doing) :

http://altdevblogaday.com/2011/06/17/te ... ars-later/

Good read IMO
Thanks for linking it, I'd read about Unity's testing earlier, but this post is new to me.

One thing that stuck out to me:
On many platforms you have to allow a couple of wrong pixels because many consumer GPUs are not fully deterministic it seems.
I'm guessing it's probably more on the integrated/mobile/etc side of things, but good to know that there may be a need for some sort of tolerances in some cases... perhaps at some point I'll test on some lower end machines/GPUs and see if I can reproduce anything like this.
0 x
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page

andrei_radu
Google Summer of Code Student
Google Summer of Code Student
Posts: 55
Joined: Fri Mar 18, 2011 8:37 pm

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by andrei_radu » Mon Jul 04, 2011 4:22 pm

One possible reason for this is that most video memory(basically on every consumer card) lacks ECC protection. That means bits in the V RAM can get corrupted by cosmic radiation and such(I'm not kidding, it is possible), not to mention they have smaller refresh times than normal DDR. Hence one source of non-deterministic input.
0 x
My Google summer of code 2011 topic: Modern Illumination Techniques
My Google summer of code thread
My Google summer of code wiki page

User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian » Wed Jul 06, 2011 8:21 am

andrei_radu wrote:One possible reason for this is that most video memory(basically on every consumer card) lacks ECC protection. That means bits in the V RAM can get corrupted by cosmic radiation and such(I'm not kidding, it is possible), not to mention they have smaller refresh times than normal DDR. Hence one source of non-deterministic input.
Wow, that's interesting; makes sense though.

I've added several common image comparison metrics that get displayed when tests fail (and I'll be adding a means by which to set thresholds for making tests pass/fail based on these). At the moment it gives you the absolute number of incorrect pixels, the mean squared error (MSE), peak signal-to-noise ratio (PSNR), and the structural similarity index (SSIM).

Here's another sample output: http://rileyadams.net/gsoc/July5/out.html
0 x
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman » Sat Jul 09, 2011 12:16 pm

What is the plan for this weekend?
0 x
Watch out for my OGRE related tweets here.

User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian » Sat Jul 09, 2011 6:29 pm

I think I'll be working on updating the documentation (it's getting to about the halfway point). I've also gotten some occasional, slight differences in a couple of the images over these past couple days (generally a pixel or two, but as many as 32 once), so I think I'll be trying to figure out if there's anything that can be done to help with this (test on a couple more machines perhaps, see if I figure out a good tolerance for a 'passing' test).
0 x
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page

User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian » Tue Jul 12, 2011 8:46 am

Alright, I've updated the wiki page a bit, I'll still need to add thumbnails for the list of tests, and add to the command line option documentation as I add functionality, but most of it is set for the time being.

I've also been trying to reproduce the non-identical images issue. I've tested on a couple more machines and haven't been able to reproduce it (or even in Windows on my primary machine), so I think it's probably iffy Linux drivers that are to blame in this case. Even in Linux on this machine it's not reliably reproduceable, it seems to only happen if I generate a reference image set, and then reboot a few times, and wait a while.

I think it'll probably work fine to just set a threshold of a few pixels, and have an option for that to raised in the case of more sensitive hardware/drivers. From the sounds of that article on Unity's framework, these kinds of differences shouldn't ever be more than a few pixels.
0 x
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman » Wed Jul 13, 2011 2:11 pm

You passed the mid-term, congratulation. :)
0 x
Watch out for my OGRE related tweets here.

User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian » Thu Jul 14, 2011 12:18 am

Whoohoo! :)

Alright, so for the next day or two, I think I'm going to be doing some general cleanup/reorganizing in a few places. After that I'm not entirely sure where to focus, perhaps I'll add/convert some more tests.
0 x
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman » Thu Jul 14, 2011 12:39 am

Yes, that is what I was going to suggest, convert as many as you can.
0 x
Watch out for my OGRE related tweets here.

User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian » Sat Jul 16, 2011 6:25 am

Alright, I finished up the cleanup I was doing, so next up will be more playpen conversions.

There's a lot of playpen tests that'll work really well here. However, I'm noticing that some seem to use media assets that are no longer a part of the SDK, so I may need to dig into some older versions, or recreate/make substitutions for some assets.

Also, just to note, I probably won't be available most of tomorrow (going away party for a friend who's moving across the country), so I probably won't be able to start too much work on these conversions until after then.
0 x
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page

User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4293
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 2
Contact:

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by spacegaier » Sat Jul 16, 2011 1:27 pm

Praetorian wrote:However, I'm noticing that some seem to use media assets that are no longer a part of the SDK, so I may need to dig into some older versions, or recreate/make substitutions for some assets.
You mean some media assets have been removed in 1.8?
0 x
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman » Sat Jul 16, 2011 1:34 pm

Well, can you list the missing assets here in a thread - by sample?
0 x
Watch out for my OGRE related tweets here.

User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian » Sat Jul 16, 2011 2:42 pm

I looked a bit closer and there aren't as many instances of this as I'd originally thought (by some coincidence the first two tests I'd picked to convert both happened to be missing something...).

Here's what I found missing in a skim of the meshes/textures (I didn't check for the materials used, and since the file of playpen tests is ~9000 lines I probably missed something...):

testTransparencyMipMaps:
-ogretext.png
testCthNewBlending:
-ogrehead_2.mesh
testAnimation:
-skeletonandpose.mesh
testAnimationBlend:
-sticky.mesh
testSimpleMesh:
-xsicylinder.mesh
0 x
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman » Sat Jul 16, 2011 4:02 pm

Here is what I found:
ogretext.png - change the sample to use sdk_logo.png (it is in Samples\Media\packs\SdkTrays.zip)

Leave the ones with issues out for now, leave them for the end.

On the topic of tests, here are two tests that I think that should be included also in the end result:
1. resources without a unique name - use this sample code as base, and use these models.

2. Stencil Glow Demo - based on this code - http://assaframan.googlepages.com/StencilGlow_src.zip
0 x
Watch out for my OGRE related tweets here.

User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian » Tue Jul 19, 2011 6:08 am

Productive day :). I added the stencil glow test, and converted 47 playpen tests (so there's now a total of 64 tests). At this point I'm just trying to get as many working as possible, so I'll have to go back to a lot of them to tidy up, tweak camera angles, etc.

Here's an output (just a warning, it's 13+mb, even with very small renders...): http://rileyadams.net/gsoc/July18/out.html
0 x
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman » Tue Jul 19, 2011 8:40 am

Very very nice.
0 x
Watch out for my OGRE related tweets here.

Post Reply