[GSoC 2011 - Accepted] Unit Testing Framework
- Noman
- OGRE Retired Team Member
- Posts: 714
- Joined: Mon Jan 31, 2005 7:21 pm
- Location: Israel
- x 2
- Contact:
Re: [GSoC 2011] Unit Testing Framework - Proposal draft is u
Nice proposal!
About the image comparison tool - did you try to read on the internet if there are any tools available? You might be able to steal some code or even use an external app as part of your project.
I would like to see a roundup of image comparison tools (if such exist) because it sounds like a huge subject that people before us probably tried to tackle.
Well written overall.
About the image comparison tool - did you try to read on the internet if there are any tools available? You might be able to steal some code or even use an external app as part of your project.
I would like to see a roundup of image comparison tools (if such exist) because it sounds like a huge subject that people before us probably tried to tackle.
Well written overall.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011] Unit Testing Framework - Proposal draft is u
Thanks for the feedback!
Most of the basic functionality should be pretty straightforward (overlaying two images, displaying the difference, etc), it's the automatic comparison that's looking a bit tougher. I've been reading around, and there's some really interesting work with "perceptual image comparison" (here is a relevant presentation from Siggraph '04), which seems like it would be an excellent thing to apply here, since it could potentially do a lot towards preventing false positives. However, I'm really not confident enough in my skills in this area to actually implement it with the time available, given all the other work. The only open source implementation I found (PerceptualDiff) is GPL, which wouldn't fly with OGRE's licensing.
ImageMagick has a simple comparison tool which could be adapted (its license is fairly permissive), although I'll need to take a closer look at it to judge just how feasible that would be.
My initial tests have resulted in perfectly identical test shots on my machine, so I could see a simpler custom solution or the ImageMagick tool working out alright (and it could always be a future addition to implement a more advanced algorithm that could potentially handle discrepancies from driver/hardware differences).
I'll do some more reading around and then update the proposal itself.
Most of the basic functionality should be pretty straightforward (overlaying two images, displaying the difference, etc), it's the automatic comparison that's looking a bit tougher. I've been reading around, and there's some really interesting work with "perceptual image comparison" (here is a relevant presentation from Siggraph '04), which seems like it would be an excellent thing to apply here, since it could potentially do a lot towards preventing false positives. However, I'm really not confident enough in my skills in this area to actually implement it with the time available, given all the other work. The only open source implementation I found (PerceptualDiff) is GPL, which wouldn't fly with OGRE's licensing.
ImageMagick has a simple comparison tool which could be adapted (its license is fairly permissive), although I'll need to take a closer look at it to judge just how feasible that would be.
My initial tests have resulted in perfectly identical test shots on my machine, so I could see a simpler custom solution or the ImageMagick tool working out alright (and it could always be a future addition to implement a more advanced algorithm that could potentially handle discrepancies from driver/hardware differences).
I'll do some more reading around and then update the proposal itself.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
-
- OGRE Retired Team Member
- Posts: 2903
- Joined: Thu Jan 18, 2007 2:48 pm
- x 58
- Contact:
Re: [GSoC 2011] Unit Testing Framework - Proposal draft is u
If it really benefitted the project, a GPL license might be tolerable. After all, the testing framework is a separate component on which the rest of Ogre does not depend, so it could be licensed under GPL without ill effects. However, unless absolutely necessary, we probably still don't want to do that
- jacmoe
- OGRE Retired Moderator
- Posts: 20570
- Joined: Thu Jan 22, 2004 10:13 am
- Location: Denmark
- x 179
- Contact:
Re: [GSoC 2011] Unit Testing Framework - Proposal draft is u
Would OpenCV work?
/* Less noise. More signal. */
Ogitor Scenebuilder - powered by Ogre, presented by Qt, fueled by Passion.
OgreAddons - the Ogre code suppository.
Ogitor Scenebuilder - powered by Ogre, presented by Qt, fueled by Passion.
OgreAddons - the Ogre code suppository.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011] Unit Testing Framework - Proposal draft is u
I looked into it a little, but it looks like OpenCV's focused mainly on finding/recognizing objects and such in images, I couldn't turn up much on directly comparing images. I'll read into it a little more and see if some piece of it could be applicable though.jacmoe wrote:Would OpenCV work?
I dug into ImageMagick a little more and it looks like its comparison tool just checks pixel by pixel and gets standard deviation/etc from there, which I don't see any problem implementing myself. Given that the produced images have been perfectly identical in my testing thus far, I think this will be sufficient (and worst comes to worst, PerceptualDiff looks like it could be substituted in pretty readily, with the less ideal licensing situation).
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Noman
- OGRE Retired Team Member
- Posts: 714
- Joined: Mon Jan 31, 2005 7:21 pm
- Location: Israel
- x 2
- Contact:
Re: [GSoC 2011] Unit Testing Framework - Proposal draft is u
Just a reminder - the application submission deadline is on friday, and I'm not sure exactly when (and in which time zone!). I saw that you didn't upload your proposal to google-melange.com yet. Make sure to do it ASAP! If you have updates you can always modify it later...
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011] Unit Testing Framework - Proposal draft is u
Yes, go ahead and submit your proposal in the Google dedicated site: http://www.google-melange.com
Time is short - don't wait.
Time is short - don't wait.
Watch out for my OGRE related tweets here.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011] Unit Testing Framework - Proposal draft is u
Thanks for the heads-up! I wasn't aware that you could edit it after submitting, so I'd been holding off on that. I'll submit it as soon as I can.
EDIT: Alright, it's submitted, I'll probably update it a few times as I get time.
EDIT: Alright, it's submitted, I'll probably update it a few times as I get time.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Noman
- OGRE Retired Team Member
- Posts: 714
- Joined: Mon Jan 31, 2005 7:21 pm
- Location: Israel
- x 2
- Contact:
Re: [GSoC 2011] Unit Testing Framework - Proposal draft is u
Congratulations! Your project has been accepted to Google Summer of Code 2011, and Assaf Raman will be your mentor.
We'll be setting up a working environment in the coming weeks. In the meantime, could you please update the wiki page of your project with the latest version of your proposal?
http://www.ogre3d.org/tikiwiki/SummerOf ... evelopment
We'll be setting up a working environment in the coming weeks. In the meantime, could you please update the wiki page of your project with the latest version of your proposal?
http://www.ogre3d.org/tikiwiki/SummerOf ... evelopment
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Awesome! I'm looking forward to getting started.
I'll update the wiki in a moment.
EDIT: alright, final proposal is up on the wiki.
I'll update the wiki in a moment.
EDIT: alright, final proposal is up on the wiki.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
-
- Gnoblar
- Posts: 24
- Joined: Mon Sep 26, 2005 6:39 am
- x 1
Re: [GSoC 2011 - Accepted] Unit Testing Framework
http://jeffkreeftmeijer.com/2011/compar ... age-diffs/ has a good start on implementing image differences. (It has source code) It'll be useful once you get an entire set of images to compare. I especially like the last method.
I'm not sure what works out well for the interface of an image comparison app. https://github.com/blog/817-behold-image-view-modes Github Image view modes inspired the first link. You could have something that flips from the original image to the compared image using a key. Or just difference. Github gives you a horizontal menu of different ways of viewing the comparison.
Also github uses a JavaScript library for Photoshop image modes. Maybe we can use the shader code for Photoshop image modes that's available.
I'm not sure what works out well for the interface of an image comparison app. https://github.com/blog/817-behold-image-view-modes Github Image view modes inspired the first link. You could have something that flips from the original image to the compared image using a key. Or just difference. Github gives you a horizontal menu of different ways of viewing the comparison.
Also github uses a JavaScript library for Photoshop image modes. Maybe we can use the shader code for Photoshop image modes that's available.
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Congratulation Praetorian, I think you selected a great topic that can befit the project in many ways.
I (Assaf) will be your mentor in the project, I selected your project from the four and am happy to have you.
I want to start off by offering a few base principles for this combined effort.
My goal as I see it is to help you reach our combine goal - working and maintainable code that will be added to our code base.
After saying that I want to define what that means, here are a few of the points I will focus on:
The reason I want you to post so many posts here on the thread is that OGRE is not a code project but a code community, the programers are not the "managers" of the project but its tools, meaning that the work you do needs to be up to the community standards. But don't worry - if you share - the community will help you back, most of the smarter people you will ever meet are part of this forum.
I think this is enough text for this post. I will write more - after thinking more.
The first thing I want to ask from you is to create a fork of the current OGRE code base (open an account in bitbucket and use the fork option there - they have a web interface for that), get your new fork code and see that it compiles for you. Post on this thread before (what you are going to do), during (only if you need to - like getting stuck), and after with a summary of what you did and the issue\conclusions you had. This will be a sort of a "test" for the communication type we are going to have.
I (Assaf) will be your mentor in the project, I selected your project from the four and am happy to have you.
I want to start off by offering a few base principles for this combined effort.
My goal as I see it is to help you reach our combine goal - working and maintainable code that will be added to our code base.
After saying that I want to define what that means, here are a few of the points I will focus on:
- Make sure you have good task list - clear and detailed.
- Follow your progress - making sure you finish your task and review your products (and helping you if you get stuck).
- Make sure that there is a good progress report of you work, meaning daily post on this thread - at the beginning of the day -
- what you plan, at the end of the day - what you finished, and along the day if you need to update\have questions, the next
- week plan and summary of the finished week.
- Make sure sure your products are testable and documented.
- Make sure you are in good spirit and enjoying the work.
The reason I want you to post so many posts here on the thread is that OGRE is not a code project but a code community, the programers are not the "managers" of the project but its tools, meaning that the work you do needs to be up to the community standards. But don't worry - if you share - the community will help you back, most of the smarter people you will ever meet are part of this forum.
I think this is enough text for this post. I will write more - after thinking more.
The first thing I want to ask from you is to create a fork of the current OGRE code base (open an account in bitbucket and use the fork option there - they have a web interface for that), get your new fork code and see that it compiles for you. Post on this thread before (what you are going to do), during (only if you need to - like getting stuck), and after with a summary of what you did and the issue\conclusions you had. This will be a sort of a "test" for the communication type we are going to have.
Watch out for my OGRE related tweets here.
- spacegaier
- OGRE Team Member
- Posts: 4304
- Joined: Mon Feb 04, 2008 2:02 pm
- Location: Germany
- x 135
- Contact:
Re: [GSoC 2011 - Accepted] Unit Testing Framework
And please add important points such as the link to the fork also to the wiki page, so that those information are stored at a central place. Thanks!
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Yes, having an updated wiki page is an important part of the documentation beside code comments and such.
Watch out for my OGRE related tweets here.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
That all definitely sounds reasonable, looking forward to working with you!Assaf Raman wrote:Congratulation Praetorian, I think you selected a great topic that can befit the project in many ways.
I (Assaf) will be your mentor in the project, I selected your project from the four and am happy to have you.
I want to start off by offering a few base principles for this combined effort.
My goal as I see it is to help you reach our combine goal - working and maintainable code that will be added to our code base.
After saying that I want to define what that means, here are a few of the points I will focus on:There are lots of other small points - but the major ones are communication and giving a good end result. The two are the biggest issues we had with other projects - students disappearing and unusable projects.
- Make sure you have good task list - clear and detailed.
- Follow your progress - making sure you finish your task and review your products (and helping you if you get stuck).
- Make sure that there is a good progress report of you work, meaning daily post on this thread - at the beginning of the day -
- what you plan, at the end of the day - what you finished, and along the day if you need to update\have questions, the next
- week plan and summary of the finished week.
- Make sure sure your products are testable and documented.
- Make sure you are in good spirit and enjoying the work.
The reason I want you to post so many posts here on the thread is that OGRE is not a code project but a code community, the programers are not the "managers" of the project but its tools, meaning that the work you do needs to be up to the community standards. But don't worry - if you share - the community will help you back, most of the smarter people you will ever meet are part of this forum.
I think this is enough text for this post. I will write more - after thinking more.
The first thing I want to ask from you is to create a fork of the current OGRE code base (open an account in bitbucket and use the fork option there - they have a web interface for that), get your new fork code and see that it compiles for you. Post on this thread before (what you are going to do), during (only if you need to - like getting stuck), and after with a summary of what you did and the issue\conclusions you had. This will be a sort of a "test" for the communication type we are going to have.
I just got back from an all-nighter (final stretch of a robotics build thing, many hours of drilling, sawing and quadcopter tuning...), so I'm not especially awake right now, but here's my "before"/plan for the test task:
- Set up Bitbucket account, ensure Mercurial is properly installed and configured.
- Create the fork.
- Check out the fork.
- Get CMake to generate appropriate makefiles.
- Compile.
- Ensure the SampleBrowser runs.
I have to get to class now, but I'll post the results ASAP.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Take your time. Get some sleep.
Watch out for my OGRE related tweets here.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Okay, so I finished this first test task. As I laid out in the previous post, I made a Bitbucket account, forked ogre, and set up and built a local repo.
The fork is here: https://bitbucket.org/RileyA/ogre-gsoc- ... k/overview I'll be updating the wiki in a moment.
I've done this a few times already (minus creating the fork), so it was all pretty straightforward.
A few conclusions/things of note:
- I haven't done a whole lot with Mercurial in the past, but it seems pretty similar to git, which I have been using a lot lately (and on that note, Bitbucket is very much like Github).
- The sample browser doesn't handle my dual monitor setup in Linux very well, the only resolution it finds is 3200x1200 (the combined resolution), but it only fullscreens onto my main (1920x1080) monitor. This is something I've encountered in the past, and is largely just because I have a strange setup; but given this project's reliance on the sample framework, it may be worth looking for some way to handle this.
The fork is here: https://bitbucket.org/RileyA/ogre-gsoc- ... k/overview I'll be updating the wiki in a moment.
I've done this a few times already (minus creating the fork), so it was all pretty straightforward.
A few conclusions/things of note:
- I haven't done a whole lot with Mercurial in the past, but it seems pretty similar to git, which I have been using a lot lately (and on that note, Bitbucket is very much like Github).
- The sample browser doesn't handle my dual monitor setup in Linux very well, the only resolution it finds is 3200x1200 (the combined resolution), but it only fullscreens onto my main (1920x1080) monitor. This is something I've encountered in the past, and is largely just because I have a strange setup; but given this project's reliance on the sample framework, it may be worth looking for some way to handle this.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Nice work.
Lets solve the two monitor issue, do you have any GL program that handle you monitors the "right" way?
Lets solve the two monitor issue, do you have any GL program that handle you monitors the "right" way?
Watch out for my OGRE related tweets here.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
I don't believe I actually have any fullscreen Linux GL apps lying around. I dug into the source a bit and it seems as though that's just what the driver is returning (the standard config dialogue does the same thing, it isn't just the browser). I suppose it's because I'm using TwinView with the Linux Nvidia driver, which treats the two screens as one large X screen. I've run into this issue in the past with some game projects using Ogre, but I've usually just decided I have a rare setup and hard-coded the resolutions for testing with my machine.Assaf Raman wrote:Nice work.
Lets solve the two monitor issue, do you have any GL program that handle you monitors the "right" way?
I'll play around with some alternate configurations, and maybe look for a program that handles it properly.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
https://wiki.archlinux.org/index.php/NVIDIA#Troubleshooting wrote:Gaming using Twinview...
...For OpenGL, add the appropiate Metamodes to your xorg.conf in section Device and restart X:Code: Select all
Option "Metamodes" "1680x1050,1680x1050; 1280x1024,1280x1024; 1680x1050,NULL; 1280x1024,NULL;"
Watch out for my OGRE related tweets here.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Awesome, that did it, thanks!
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
wiki page titled "Visual Unit Testing Framework"
Then next step should be creating a new wiki page for your result titled "Visual Unit Testing Framework".
That page will mention the Google summer of code, that page will be the wiki documentation of the feature you are going to be implementing.
Some titles for that page:
Sounds right? Feel free to write what you think, if you don't like some of what I wrote lets discuss it before you do anything you don't like.
That page will mention the Google summer of code, that page will be the wiki documentation of the feature you are going to be implementing.
Some titles for that page:
- Abstract
- Introduction - motivation\need(the current state of testing in the project), suggested solution(your feature in a short description).
- How to use.
- Description of the output (we need to decide on it but I sort of see it as a long html - that begins by showing errors with images side by side - then move on to good results images, perhaps the html doc should start with a thumbnail summary of all the test were good and bad test have frames with different color - but each test thumbnail will always be at the same place - a sort of a test overview map? you posted a smiler image.) Add a link to a sample output when you will have it later on.
- "List of test" summary table (for each a column - description, sample image, what parts of the system it tests and others as you wish)
- Known issues.
- Future development suggestions
Sounds right? Feel free to write what you think, if you don't like some of what I wrote lets discuss it before you do anything you don't like.
Watch out for my OGRE related tweets here.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: wiki page titled "Visual Unit Testing Framework"
Yeah, that all sounds about right. Should I keep adding to the existing wiki page for the project (which at the moment has the proposal and link to the fork), or start a whole new page for this?Assaf Raman wrote:Then next step should be creating a new wiki page for your result titled "Visual Unit Testing Framework".
That page will mention the Google summer of code, that page will be the wiki documentation of the feature you are going to be implementing.
Some titles for that page:Feel free to do it your way - I just wanted to give you the spirit of it - this wiki page should be more or less complete before your first commit or code writing. We can assume you can't complete the full test list before you start - but add lets at least four lines to the table to demonstrate it. Same goes to the sample output, create a something to demonstrate the idea for now, later replace with final product.
- Abstract
- Introduction - motivation\need(the current state of testing in the project), suggested solution(your feature in a short description).
- How to use.
- Description of the output (we need to decide on it but I sort of see it as a long html - that begins by showing errors with images side by side - then move on to good results images, perhaps the html doc should start with a thumbnail summary of all the test were good and bad test have frames with different color - but each test thumbnail will always be at the same place - a sort of a test overview map? you posted a smiler image.) Add a link to a sample output when you will have it later on.
- "List of test" summary table (for each a column - description, sample image, what parts of the system it tests and others as you wish)
- Known issues.
- Future development suggestions
Sounds right? Feel free to write what you think, if you don't like some of what I wrote lets discuss it before you do anything you don't like.
I've got a big discrete math exam in a couple days, so I'm a little preoccupied with studying, but I'll start outlining and putting this together as I get time, and I'll be able to give it my full attention after I'm done with the exam.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- spacegaier
- OGRE Team Member
- Posts: 4304
- Joined: Mon Feb 04, 2008 2:02 pm
- Location: Germany
- x 135
- Contact:
Re: [GSoC 2011 - Accepted] Unit Testing Framework
I think Assaf meant a separate page. The current one is for the GSoC stuff. The new one will be about what you are about to create and that will become one part of the final documentation (the how-to guide use and how-to extend for your testing framework).
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Yes, a new page.
Watch out for my OGRE related tweets here.