[GSoC 2011 - Accepted] Unit Testing Framework

Threads related to Google Summer of Code
Post Reply
User avatar
Noman
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 714
Joined: Mon Jan 31, 2005 7:21 pm
Location: Israel
x 2
Contact:

Re: [GSoC 2011] Unit Testing Framework - Proposal draft is u

Post by Noman »

Nice proposal!

About the image comparison tool - did you try to read on the internet if there are any tools available? You might be able to steal some code or even use an external app as part of your project.

I would like to see a roundup of image comparison tools (if such exist) because it sounds like a huge subject that people before us probably tried to tackle.

Well written overall.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011] Unit Testing Framework - Proposal draft is u

Post by Praetorian »

Thanks for the feedback!

Most of the basic functionality should be pretty straightforward (overlaying two images, displaying the difference, etc), it's the automatic comparison that's looking a bit tougher. I've been reading around, and there's some really interesting work with "perceptual image comparison" (here is a relevant presentation from Siggraph '04), which seems like it would be an excellent thing to apply here, since it could potentially do a lot towards preventing false positives. However, I'm really not confident enough in my skills in this area to actually implement it with the time available, given all the other work. The only open source implementation I found (PerceptualDiff) is GPL, which wouldn't fly with OGRE's licensing.

ImageMagick has a simple comparison tool which could be adapted (its license is fairly permissive), although I'll need to take a closer look at it to judge just how feasible that would be.

My initial tests have resulted in perfectly identical test shots on my machine, so I could see a simpler custom solution or the ImageMagick tool working out alright (and it could always be a future addition to implement a more advanced algorithm that could potentially handle discrepancies from driver/hardware differences).

I'll do some more reading around and then update the proposal itself.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
x 58
Contact:

Re: [GSoC 2011] Unit Testing Framework - Proposal draft is u

Post by CABAListic »

If it really benefitted the project, a GPL license might be tolerable. After all, the testing framework is a separate component on which the rest of Ogre does not depend, so it could be licensed under GPL without ill effects. However, unless absolutely necessary, we probably still don't want to do that :)
User avatar
jacmoe
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 20570
Joined: Thu Jan 22, 2004 10:13 am
Location: Denmark
x 179
Contact:

Re: [GSoC 2011] Unit Testing Framework - Proposal draft is u

Post by jacmoe »

Would OpenCV work?
/* Less noise. More signal. */
Ogitor Scenebuilder - powered by Ogre, presented by Qt, fueled by Passion.
OgreAddons - the Ogre code suppository.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011] Unit Testing Framework - Proposal draft is u

Post by Praetorian »

jacmoe wrote:Would OpenCV work?
I looked into it a little, but it looks like OpenCV's focused mainly on finding/recognizing objects and such in images, I couldn't turn up much on directly comparing images. I'll read into it a little more and see if some piece of it could be applicable though.

I dug into ImageMagick a little more and it looks like its comparison tool just checks pixel by pixel and gets standard deviation/etc from there, which I don't see any problem implementing myself. Given that the produced images have been perfectly identical in my testing thus far, I think this will be sufficient (and worst comes to worst, PerceptualDiff looks like it could be substituted in pretty readily, with the less ideal licensing situation).
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Noman
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 714
Joined: Mon Jan 31, 2005 7:21 pm
Location: Israel
x 2
Contact:

Re: [GSoC 2011] Unit Testing Framework - Proposal draft is u

Post by Noman »

Just a reminder - the application submission deadline is on friday, and I'm not sure exactly when (and in which time zone!). I saw that you didn't upload your proposal to google-melange.com yet. Make sure to do it ASAP! If you have updates you can always modify it later...
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011] Unit Testing Framework - Proposal draft is u

Post by Assaf Raman »

Yes, go ahead and submit your proposal in the Google dedicated site: http://www.google-melange.com
Time is short - don't wait.
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011] Unit Testing Framework - Proposal draft is u

Post by Praetorian »

Thanks for the heads-up! I wasn't aware that you could edit it after submitting, so I'd been holding off on that. I'll submit it as soon as I can.

EDIT: Alright, it's submitted, I'll probably update it a few times as I get time.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Noman
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 714
Joined: Mon Jan 31, 2005 7:21 pm
Location: Israel
x 2
Contact:

Re: [GSoC 2011] Unit Testing Framework - Proposal draft is u

Post by Noman »

Congratulations! Your project has been accepted to Google Summer of Code 2011, and Assaf Raman will be your mentor.

We'll be setting up a working environment in the coming weeks. In the meantime, could you please update the wiki page of your project with the latest version of your proposal?

http://www.ogre3d.org/tikiwiki/SummerOf ... evelopment
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Awesome! I'm looking forward to getting started. :D

I'll update the wiki in a moment.

EDIT: alright, final proposal is up on the wiki.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
iFire
Gnoblar
Posts: 24
Joined: Mon Sep 26, 2005 6:39 am
x 1

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by iFire »

http://jeffkreeftmeijer.com/2011/compar ... age-diffs/ has a good start on implementing image differences. (It has source code) It'll be useful once you get an entire set of images to compare. I especially like the last method.

I'm not sure what works out well for the interface of an image comparison app. https://github.com/blog/817-behold-image-view-modes Github Image view modes inspired the first link. You could have something that flips from the original image to the compared image using a key. Or just difference. Github gives you a horizontal menu of different ways of viewing the comparison.

Also github uses a JavaScript library for Photoshop image modes. Maybe we can use the shader code for Photoshop image modes that's available.
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Congratulation Praetorian, I think you selected a great topic that can befit the project in many ways.
I (Assaf) will be your mentor in the project, I selected your project from the four and am happy to have you.

I want to start off by offering a few base principles for this combined effort.
My goal as I see it is to help you reach our combine goal - working and maintainable code that will be added to our code base.
After saying that I want to define what that means, here are a few of the points I will focus on:
  • Make sure you have good task list - clear and detailed.
  • Follow your progress - making sure you finish your task and review your products (and helping you if you get stuck).
  • Make sure that there is a good progress report of you work, meaning daily post on this thread - at the beginning of the day -
  • what you plan, at the end of the day - what you finished, and along the day if you need to update\have questions, the next
  • week plan and summary of the finished week.
  • Make sure sure your products are testable and documented.
  • Make sure you are in good spirit and enjoying the work.
There are lots of other small points - but the major ones are communication and giving a good end result. The two are the biggest issues we had with other projects - students disappearing and unusable projects.

The reason I want you to post so many posts here on the thread is that OGRE is not a code project but a code community, the programers are not the "managers" of the project but its tools, meaning that the work you do needs to be up to the community standards. But don't worry - if you share - the community will help you back, most of the smarter people you will ever meet are part of this forum.

I think this is enough text for this post. I will write more - after thinking more.

The first thing I want to ask from you is to create a fork of the current OGRE code base (open an account in bitbucket and use the fork option there - they have a web interface for that), get your new fork code and see that it compiles for you. Post on this thread before (what you are going to do), during (only if you need to - like getting stuck), and after with a summary of what you did and the issue\conclusions you had. This will be a sort of a "test" for the communication type we are going to have.
Watch out for my OGRE related tweets here.
User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4304
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 135
Contact:

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by spacegaier »

And please add important points such as the link to the fork also to the wiki page, so that those information are stored at a central place. Thanks!
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Yes, having an updated wiki page is an important part of the documentation beside code comments and such.
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Assaf Raman wrote:Congratulation Praetorian, I think you selected a great topic that can befit the project in many ways.
I (Assaf) will be your mentor in the project, I selected your project from the four and am happy to have you.

I want to start off by offering a few base principles for this combined effort.
My goal as I see it is to help you reach our combine goal - working and maintainable code that will be added to our code base.
After saying that I want to define what that means, here are a few of the points I will focus on:
  • Make sure you have good task list - clear and detailed.
  • Follow your progress - making sure you finish your task and review your products (and helping you if you get stuck).
  • Make sure that there is a good progress report of you work, meaning daily post on this thread - at the beginning of the day -
  • what you plan, at the end of the day - what you finished, and along the day if you need to update\have questions, the next
  • week plan and summary of the finished week.
  • Make sure sure your products are testable and documented.
  • Make sure you are in good spirit and enjoying the work.
There are lots of other small points - but the major ones are communication and giving a good end result. The two are the biggest issues we had with other projects - students disappearing and unusable projects.

The reason I want you to post so many posts here on the thread is that OGRE is not a code project but a code community, the programers are not the "managers" of the project but its tools, meaning that the work you do needs to be up to the community standards. But don't worry - if you share - the community will help you back, most of the smarter people you will ever meet are part of this forum.

I think this is enough text for this post. I will write more - after thinking more.

The first thing I want to ask from you is to create a fork of the current OGRE code base (open an account in bitbucket and use the fork option there - they have a web interface for that), get your new fork code and see that it compiles for you. Post on this thread before (what you are going to do), during (only if you need to - like getting stuck), and after with a summary of what you did and the issue\conclusions you had. This will be a sort of a "test" for the communication type we are going to have.
That all definitely sounds reasonable, looking forward to working with you!

I just got back from an all-nighter (final stretch of a robotics build thing, many hours of drilling, sawing and quadcopter tuning...), so I'm not especially awake right now, but here's my "before"/plan for the test task:

- Set up Bitbucket account, ensure Mercurial is properly installed and configured.
- Create the fork.
- Check out the fork.
- Get CMake to generate appropriate makefiles.
- Compile.
- Ensure the SampleBrowser runs.

I have to get to class now, but I'll post the results ASAP.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Take your time. Get some sleep. :D
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Okay, so I finished this first test task. As I laid out in the previous post, I made a Bitbucket account, forked ogre, and set up and built a local repo.

The fork is here: https://bitbucket.org/RileyA/ogre-gsoc- ... k/overview I'll be updating the wiki in a moment.

I've done this a few times already (minus creating the fork), so it was all pretty straightforward.

A few conclusions/things of note:
- I haven't done a whole lot with Mercurial in the past, but it seems pretty similar to git, which I have been using a lot lately (and on that note, Bitbucket is very much like Github).
- The sample browser doesn't handle my dual monitor setup in Linux very well, the only resolution it finds is 3200x1200 (the combined resolution), but it only fullscreens onto my main (1920x1080) monitor. This is something I've encountered in the past, and is largely just because I have a strange setup; but given this project's reliance on the sample framework, it may be worth looking for some way to handle this.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Nice work.
Lets solve the two monitor issue, do you have any GL program that handle you monitors the "right" way?
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Assaf Raman wrote:Nice work.
Lets solve the two monitor issue, do you have any GL program that handle you monitors the "right" way?
I don't believe I actually have any fullscreen Linux GL apps lying around. I dug into the source a bit and it seems as though that's just what the driver is returning (the standard config dialogue does the same thing, it isn't just the browser). I suppose it's because I'm using TwinView with the Linux Nvidia driver, which treats the two screens as one large X screen. I've run into this issue in the past with some game projects using Ogre, but I've usually just decided I have a rare setup and hard-coded the resolutions for testing with my machine.

I'll play around with some alternate configurations, and maybe look for a program that handles it properly.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

https://wiki.archlinux.org/index.php/NVIDIA#Troubleshooting wrote:Gaming using Twinview...
...For OpenGL, add the appropiate Metamodes to your xorg.conf in section Device and restart X:

Code: Select all

Option "Metamodes" "1680x1050,1680x1050; 1280x1024,1280x1024; 1680x1050,NULL;  1280x1024,NULL;"
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Awesome, that did it, thanks!
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

wiki page titled "Visual Unit Testing Framework"

Post by Assaf Raman »

Then next step should be creating a new wiki page for your result titled "Visual Unit Testing Framework".
That page will mention the Google summer of code, that page will be the wiki documentation of the feature you are going to be implementing.
Some titles for that page:
  • Abstract
  • Introduction - motivation\need(the current state of testing in the project), suggested solution(your feature in a short description).
  • How to use.
  • Description of the output (we need to decide on it but I sort of see it as a long html - that begins by showing errors with images side by side - then move on to good results images, perhaps the html doc should start with a thumbnail summary of all the test were good and bad test have frames with different color - but each test thumbnail will always be at the same place - a sort of a test overview map? you posted a smiler image.) Add a link to a sample output when you will have it later on.
  • "List of test" summary table (for each a column - description, sample image, what parts of the system it tests and others as you wish)
  • Known issues.
  • Future development suggestions
Feel free to do it your way - I just wanted to give you the spirit of it - this wiki page should be more or less complete before your first commit or code writing. We can assume you can't complete the full test list before you start - but add lets at least four lines to the table to demonstrate it. Same goes to the sample output, create a something to demonstrate the idea for now, later replace with final product.

Sounds right? Feel free to write what you think, if you don't like some of what I wrote lets discuss it before you do anything you don't like. :D
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: wiki page titled "Visual Unit Testing Framework"

Post by Praetorian »

Assaf Raman wrote:Then next step should be creating a new wiki page for your result titled "Visual Unit Testing Framework".
That page will mention the Google summer of code, that page will be the wiki documentation of the feature you are going to be implementing.
Some titles for that page:
  • Abstract
  • Introduction - motivation\need(the current state of testing in the project), suggested solution(your feature in a short description).
  • How to use.
  • Description of the output (we need to decide on it but I sort of see it as a long html - that begins by showing errors with images side by side - then move on to good results images, perhaps the html doc should start with a thumbnail summary of all the test were good and bad test have frames with different color - but each test thumbnail will always be at the same place - a sort of a test overview map? you posted a smiler image.) Add a link to a sample output when you will have it later on.
  • "List of test" summary table (for each a column - description, sample image, what parts of the system it tests and others as you wish)
  • Known issues.
  • Future development suggestions
Feel free to do it your way - I just wanted to give you the spirit of it - this wiki page should be more or less complete before your first commit or code writing. We can assume you can't complete the full test list before you start - but add lets at least four lines to the table to demonstrate it. Same goes to the sample output, create a something to demonstrate the idea for now, later replace with final product.

Sounds right? Feel free to write what you think, if you don't like some of what I wrote lets discuss it before you do anything you don't like. :D
Yeah, that all sounds about right. Should I keep adding to the existing wiki page for the project (which at the moment has the proposal and link to the fork), or start a whole new page for this?

I've got a big discrete math exam in a couple days, so I'm a little preoccupied with studying, but I'll start outlining and putting this together as I get time, and I'll be able to give it my full attention after I'm done with the exam.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4304
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 135
Contact:

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by spacegaier »

I think Assaf meant a separate page. The current one is for the GSoC stuff. The new one will be about what you are about to create and that will become one part of the final documentation (the how-to guide use and how-to extend for your testing framework).
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Yes, a new page.
Watch out for my OGRE related tweets here.
Post Reply