[GSoC 2011 - Accepted] Unit Testing Framework

Threads related to Google Summer of Code
Post Reply
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Alright, so I didn't have a chance to do any work yesterday, but I did get a good deal of schoolwork out of the way, so I should be able to focus on working this evening.

My goals:
  • 1: Finish up VisualTest.
  • 2: Convert one of the playpen tests over.
  • 3: Get a sample plugin containing that test building and running in the Sample Browser.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Alright, so I finished all 3 goals, just pushed to the fork. I have a sample plugin with the first converted playpen test (manual LOD) derived from VisualTest, building and running in the Sample Browser.

I md5summed the shots generated by a few runs of the test and it looks like it's generating identical shots so far.

EDIT: As for tomorrow, I'll probably be tied up with schoolwork again, but I might end up converting a few more playpen tests over if I have some spare time. Friday and over the weekend I'll definitely be able to put a good deal of time in.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Good work.
See you Friday.
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

So I had more extra time than I expected yesterday, and between yesterday evening and this morning, I got all of the initial playpen tests converted and running in the sample browser.

I think my main goal for tonight will be to tidy things up, and make sure I'm still getting correct screenshots. Provided I don't hit any significant issues, I may begin working on the TestContext.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Excellent. Good progress. :D
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Alright, I forgot to post this morning, but I got a basic SampleContext up and running today, it goes through the tests automatically taking screenshots and quits immediately after it's done. At the moment the screenshots are just saved in the bin directory.

My final exams are coming up week after next, so I'll probably be less active as they get closer, but at this point I'm already ahead of my original schedule, so I think it'll be alright.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Well, I've done a little tidying up lately, but I'm sorta in full final exams studying mode now.

I'll be back after Thursday or Friday with school out and no other obligations.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

good luck with all the exams. :D
Watch out for my OGRE related tweets here.
User avatar
omniter
OGRE Contributor
OGRE Contributor
Posts: 424
Joined: Thu Mar 19, 2009 8:08 am
Location: Canada
x 44

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by omniter »

I think this is a really cool project, Praetorian. I'd be happy to answer any questions you have about the OgreBites framework. Good luck with exams! :D
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Hi, when do you think you are going to get back to work on the project?
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Right now :)

I had my last final yesterday and just finished getting all moved back home.

I'll be working on an initial version of the image comparison tool this week.

My goal for today will be to get the basics set up for the comparison tool's plugin, as well as working out a way of defining 'sets' of images for the tool to compare (most likely a simple config file with a list of the screenshots from a particular test run and some basic info about the set (resolution, perhaps a build or version number, etc)).
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Can you explain more about this "image comparison tool" and the work you plan to do on it?
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

The ultimate goal will be to make a simple, but flexible image diffing tool that lets you manually examine differences in the test screenshots.

For this week I plan on getting the basics in place: letting the user select a pair of test sets to compare, and then allowing them to flip through the images with the arrow keys; more advanced methods of overlaying and comparing the images will come in a later week (as well as some automated comparison to identify the images with the most differences).

It will be implemented in a sample plugin, so that it can run directly from the test context (so you can immediately inspect the results after running a batch of tests), or from the sample browser.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Why do we need such a tool? This feel to me really off topic.
We talked about an HTML output - why do we need a complicated tool? In what the tool better then the HTML output?

What need for OGRE is the option to run a sort of batch process, I was planning to take your work and have a process where every night it is run, and uploaded to a web site - giving the community a day by day status of the project.

The tool can be nice, but not much then that. The focus should be giving a summary output of the sate of the project. This is what we need.
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Assaf Raman wrote:Why do we need such a tool? This feel to me really off topic.
We talked about an HTML output - why do we need a complicated tool? In what the tool better then the HTML output?

What need for OGRE is the option to run a sort of batch process, I was planning to take your work and have a process where every night it is run, and uploaded to a web site - giving the community a day by day status of the project.

The tool can be nice, but not much then that. The focus should be giving a summary output of the sate of the project. This is what we need.
Hmmm, alright then. tuan kuranes had suggested this sort of thing early on and I'd assumed it would be helpful, but I suppose HTML output alone would be sufficient. I could see such a comparison tool being helpful by allowing for some advanced comparison modes (using shaders to highlight differences could be interesting), but I do suspect it would be a lot of work for not a whole lot of actual practical use.

I'd set aside a good deal of time in the schedule for this, so I'll need to rework a bit, but this should make things simpler and give me more time to focus on other aspects of the project.

I think I'll take the rest of tonight and tomorrow to work out the HTML output formatting then.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Sounds good, if we can have a good nightly unit testing status for many part of OGRE that is updated to the web (I will help with a computer to run the unit tests after the summer project is done) - I consider the project a win.
I may improve the project later to get each of the change sets from OGRE source control one by one - so I can get a notification to all the community when with the exact change that broke OGRE, but this is not a must part of your project (will happy if you will help me with that also - but your focus is creating a good code base for running relevant tests visual test of OGRE render results).
Anything else like the tool - nice to have (meaning if we have time for it - great - else - no worry).
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Alright, well here's an initial mockup of the HTML output (the final product will include a lot more statistics and details, but I think the basic idea is here): http://www.rileyadams.net/GSoC/Mockup/output.html

I'm wondering if it'd work better to have a separate overview page that links to individual pages for each test, to prevent having to load a ton of images at once?
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Looks good.
A page for each is not that of a good idea, but perhaps a page the ones that failed and a different page for passed.
Later you can also add a "diff" image or the sort to help see where the failure is (only to failed...)

You are doing great work! Nice touch with the colors of the HTML page.
Watch out for my OGRE related tweets here.
User avatar
omniter
OGRE Contributor
OGRE Contributor
Posts: 424
Joined: Thu Mar 19, 2009 8:08 am
Location: Canada
x 44

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by omniter »

Perhaps you could output linked HTML files? For example, a summary page that links (via relative paths) to other pages for individual tests.
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

I don't see the problem with multiple images on the same page for now.
Watch out for my OGRE related tweets here.
CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
x 58
Contact:

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by CABAListic »

Me neither. First and foremost I think that the results would be generated and viewed on your own computer, so there is no loading time involved even for many images. And I think a single page is simpler to use.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Perhaps later on there could be an option or two for output formatting (separate pages, include only failed tests, etc)?

For now I think this should be fine though, it wouldn't be hard to make some changes later on.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Yes, don't worry about it, an easy fix later on.
Watch out for my OGRE related tweets here.
User avatar
Wolfmanfx
OGRE Team Member
OGRE Team Member
Posts: 1525
Joined: Fri Feb 03, 2006 10:37 pm
Location: Austria - Leoben
x 99
Contact:

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Wolfmanfx »

Take into account an xml or file output so that the results can be evaluated from build server. If you stick with an standard xml output (like cppunit) most build server can parse these xml files.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

I think this week I'll generally be working towards actually getting this HTML output generating from the framework. Today I organized the screenshot output a bit more (a uniquely named directory is created to hold the screenshots and extra data each time a batch of tests is run, it also outputs a small config file with some info about the run, to be used in comparisons). I also cleaned/reorganized a few parts of the TestContext and VisualTest classes.
Wolfmanfx wrote:Take into account an xml or file output so that the results can be evaluated from build server. If you stick with an standard xml output (like cppunit) most build server can parse these xml files.
I'm not especially familiar with build server stuff, but given the specs I could definitely make it generate output that's usable there.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
Post Reply