[GSoC 2011 - Accepted] Unit Testing Framework
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
I think I'll begin working towards more meaningful comparison, starting by comparing pixel by pixel, and tracking some statistics about the avg/stddev/etc in color discrepancies. I'm not exactly sure what the options are as far as more advanced comparison, (I may look into something like PerceptualDiff, if time permits), but I think this simple approach should at least be a good start.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Why do we need anything more advanced then comparing pixel by pixel?
Watch out for my OGRE related tweets here.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
If two images are identical, but one is translated by one pixel then this would report a significant difference, even though the images are nearly identical. I suppose the main idea is checking if the image is identical though, so any extra info about how different the image is for a failed test, is just a bonus and doesn't necessarily need to be perfect.Assaf Raman wrote:Why do we need anything more advanced then comparing pixel by pixel?
I think this will work fine for this purpose, but a more advanced comparison might be a good later addition to allow for some more flexibility (perhaps there's a driver configuration that doesn't always produce perfectly identical images...).
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
The same driver configuration always produce perfectly identical images - as far I know.
Watch out for my OGRE related tweets here.
- Wolfmanfx
- OGRE Team Member
- Posts: 1525
- Joined: Fri Feb 03, 2006 10:37 pm
- Location: Austria - Leoben
- x 99
- Contact:
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Maybe just prepare the test system so it can be extented in future. For example you could create an interface IImageValditator which gets two image memory buffer and can return information if this image are the same or not.
Just my 2 cents
Just my 2 cents
-
- OGRE Retired Team Member
- Posts: 2903
- Joined: Thu Jan 18, 2007 2:48 pm
- x 58
- Contact:
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Perhaps, but it could allow to create a more general set of reference pictures so that not everyone has to create their own set of valid references manually. The less hassle the testing involves, the more testing might be doneAssaf Raman wrote:The same driver configuration always produce perfectly identical images - as far I know.
But it's a secondary issue for now.
- Thoran
- Halfling
- Posts: 94
- Joined: Mon Dec 01, 2008 2:04 pm
- Location: Germany
- x 1
- Contact:
Re: [GSoC 2011 - Accepted] Unit Testing Framework
What about rounding erros?I don't think you can exclude them completely, even with the same settings. So maybe a comparision of pixels within a delta range is possible, if Praetorian is not already doing it.Assaf Raman wrote:The same driver configuration always produce perfectly identical images - as far I know.
Thoran
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
That was a thought as well; but yeah, for now it isn't a huge concern, I'm just going to leave it open to be extended later, as Wolfmanfx suggested.CABAListic wrote:Perhaps, but it could allow to create a more general set of reference pictures so that not everyone has to create their own set of valid references manually. The less hassle the testing involves, the more testing might be doneAssaf Raman wrote:The same driver configuration always produce perfectly identical images - as far I know.
But it's a secondary issue for now.
In theory (and in practice, from what I've done thus far), if all the input is kept consistent, all the rounding and such should always be the same (there should be no way that (on a single machine) the same floating point operation on the same operands would produce a different result). I can't think of any exceptions to this at this point (perhaps a multi-GPU setup with slightly different GPUs?), though it does seem like it'd be easy to have an uninitialized value or something somewhere that would throw everything off (though I suspect you'd have larger problems in such a case). On separate machines with different processors/architectures/etc though, it's definitely unlikely that things would stay consistent.Thoran wrote:What about rounding erros?I don't think you can exclude them completely, even with the same settings. So maybe a comparision of pixels within a delta range is possible, if Praetorian is not already doing it.Assaf Raman wrote:The same driver configuration always produce perfectly identical images - as far I know.
Thoran
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Noman
- OGRE Retired Team Member
- Posts: 714
- Joined: Mon Jan 31, 2005 7:21 pm
- Location: Israel
- x 2
- Contact:
Re: [GSoC 2011 - Accepted] Unit Testing Framework
I don't know if it has been mentioned in this thread, but there is a nice post by the graphic lead in unity about testing graphics code (and what they ended up doing) :
http://altdevblogaday.com/2011/06/17/te ... ars-later/
Good read IMO
http://altdevblogaday.com/2011/06/17/te ... ars-later/
Good read IMO
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Thanks for linking it, I'd read about Unity's testing earlier, but this post is new to me.Noman wrote:I don't know if it has been mentioned in this thread, but there is a nice post by the graphic lead in unity about testing graphics code (and what they ended up doing) :
http://altdevblogaday.com/2011/06/17/te ... ars-later/
Good read IMO
One thing that stuck out to me:
I'm guessing it's probably more on the integrated/mobile/etc side of things, but good to know that there may be a need for some sort of tolerances in some cases... perhaps at some point I'll test on some lower end machines/GPUs and see if I can reproduce anything like this.On many platforms you have to allow a couple of wrong pixels because many consumer GPUs are not fully deterministic it seems.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
-
- Google Summer of Code Student
- Posts: 55
- Joined: Fri Mar 18, 2011 8:37 pm
- x 8
Re: [GSoC 2011 - Accepted] Unit Testing Framework
One possible reason for this is that most video memory(basically on every consumer card) lacks ECC protection. That means bits in the V RAM can get corrupted by cosmic radiation and such(I'm not kidding, it is possible), not to mention they have smaller refresh times than normal DDR. Hence one source of non-deterministic input.
My Google summer of code 2011 topic: Modern Illumination Techniques
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Wow, that's interesting; makes sense though.andrei_radu wrote:One possible reason for this is that most video memory(basically on every consumer card) lacks ECC protection. That means bits in the V RAM can get corrupted by cosmic radiation and such(I'm not kidding, it is possible), not to mention they have smaller refresh times than normal DDR. Hence one source of non-deterministic input.
I've added several common image comparison metrics that get displayed when tests fail (and I'll be adding a means by which to set thresholds for making tests pass/fail based on these). At the moment it gives you the absolute number of incorrect pixels, the mean squared error (MSE), peak signal-to-noise ratio (PSNR), and the structural similarity index (SSIM).
Here's another sample output: http://rileyadams.net/gsoc/July5/out.html
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
What is the plan for this weekend?
Watch out for my OGRE related tweets here.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
I think I'll be working on updating the documentation (it's getting to about the halfway point). I've also gotten some occasional, slight differences in a couple of the images over these past couple days (generally a pixel or two, but as many as 32 once), so I think I'll be trying to figure out if there's anything that can be done to help with this (test on a couple more machines perhaps, see if I figure out a good tolerance for a 'passing' test).
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Alright, I've updated the wiki page a bit, I'll still need to add thumbnails for the list of tests, and add to the command line option documentation as I add functionality, but most of it is set for the time being.
I've also been trying to reproduce the non-identical images issue. I've tested on a couple more machines and haven't been able to reproduce it (or even in Windows on my primary machine), so I think it's probably iffy Linux drivers that are to blame in this case. Even in Linux on this machine it's not reliably reproduceable, it seems to only happen if I generate a reference image set, and then reboot a few times, and wait a while.
I think it'll probably work fine to just set a threshold of a few pixels, and have an option for that to raised in the case of more sensitive hardware/drivers. From the sounds of that article on Unity's framework, these kinds of differences shouldn't ever be more than a few pixels.
I've also been trying to reproduce the non-identical images issue. I've tested on a couple more machines and haven't been able to reproduce it (or even in Windows on my primary machine), so I think it's probably iffy Linux drivers that are to blame in this case. Even in Linux on this machine it's not reliably reproduceable, it seems to only happen if I generate a reference image set, and then reboot a few times, and wait a while.
I think it'll probably work fine to just set a threshold of a few pixels, and have an option for that to raised in the case of more sensitive hardware/drivers. From the sounds of that article on Unity's framework, these kinds of differences shouldn't ever be more than a few pixels.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
You passed the mid-term, congratulation.
Watch out for my OGRE related tweets here.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Whoohoo!
Alright, so for the next day or two, I think I'm going to be doing some general cleanup/reorganizing in a few places. After that I'm not entirely sure where to focus, perhaps I'll add/convert some more tests.
Alright, so for the next day or two, I think I'm going to be doing some general cleanup/reorganizing in a few places. After that I'm not entirely sure where to focus, perhaps I'll add/convert some more tests.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Yes, that is what I was going to suggest, convert as many as you can.
Watch out for my OGRE related tweets here.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Alright, I finished up the cleanup I was doing, so next up will be more playpen conversions.
There's a lot of playpen tests that'll work really well here. However, I'm noticing that some seem to use media assets that are no longer a part of the SDK, so I may need to dig into some older versions, or recreate/make substitutions for some assets.
Also, just to note, I probably won't be available most of tomorrow (going away party for a friend who's moving across the country), so I probably won't be able to start too much work on these conversions until after then.
There's a lot of playpen tests that'll work really well here. However, I'm noticing that some seem to use media assets that are no longer a part of the SDK, so I may need to dig into some older versions, or recreate/make substitutions for some assets.
Also, just to note, I probably won't be available most of tomorrow (going away party for a friend who's moving across the country), so I probably won't be able to start too much work on these conversions until after then.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- spacegaier
- OGRE Team Member
- Posts: 4304
- Joined: Mon Feb 04, 2008 2:02 pm
- Location: Germany
- x 135
- Contact:
Re: [GSoC 2011 - Accepted] Unit Testing Framework
You mean some media assets have been removed in 1.8?Praetorian wrote:However, I'm noticing that some seem to use media assets that are no longer a part of the SDK, so I may need to dig into some older versions, or recreate/make substitutions for some assets.
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Well, can you list the missing assets here in a thread - by sample?
Watch out for my OGRE related tweets here.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
I looked a bit closer and there aren't as many instances of this as I'd originally thought (by some coincidence the first two tests I'd picked to convert both happened to be missing something...).
Here's what I found missing in a skim of the meshes/textures (I didn't check for the materials used, and since the file of playpen tests is ~9000 lines I probably missed something...):
testTransparencyMipMaps:
-ogretext.png
testCthNewBlending:
-ogrehead_2.mesh
testAnimation:
-skeletonandpose.mesh
testAnimationBlend:
-sticky.mesh
testSimpleMesh:
-xsicylinder.mesh
Here's what I found missing in a skim of the meshes/textures (I didn't check for the materials used, and since the file of playpen tests is ~9000 lines I probably missed something...):
testTransparencyMipMaps:
-ogretext.png
testCthNewBlending:
-ogrehead_2.mesh
testAnimation:
-skeletonandpose.mesh
testAnimationBlend:
-sticky.mesh
testSimpleMesh:
-xsicylinder.mesh
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Here is what I found:
ogretext.png - change the sample to use sdk_logo.png (it is in Samples\Media\packs\SdkTrays.zip)
Leave the ones with issues out for now, leave them for the end.
On the topic of tests, here are two tests that I think that should be included also in the end result:
1. resources without a unique name - use this sample code as base, and use these models.
2. Stencil Glow Demo - based on this code - http://assaframan.googlepages.com/StencilGlow_src.zip
ogretext.png - change the sample to use sdk_logo.png (it is in Samples\Media\packs\SdkTrays.zip)
Leave the ones with issues out for now, leave them for the end.
On the topic of tests, here are two tests that I think that should be included also in the end result:
1. resources without a unique name - use this sample code as base, and use these models.
2. Stencil Glow Demo - based on this code - http://assaframan.googlepages.com/StencilGlow_src.zip
Watch out for my OGRE related tweets here.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Productive day . I added the stencil glow test, and converted 47 playpen tests (so there's now a total of 64 tests). At this point I'm just trying to get as many working as possible, so I'll have to go back to a lot of them to tidy up, tweak camera angles, etc.
Here's an output (just a warning, it's 13+mb, even with very small renders...): http://rileyadams.net/gsoc/July18/out.html
Here's an output (just a warning, it's 13+mb, even with very small renders...): http://rileyadams.net/gsoc/July18/out.html
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Very very nice.
Watch out for my OGRE related tweets here.