Page 4 of 8

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Wed May 25, 2011 11:49 pm
by Praetorian
Alright, so I didn't have a chance to do any work yesterday, but I did get a good deal of schoolwork out of the way, so I should be able to focus on working this evening.

My goals:
  • 1: Finish up VisualTest.
  • 2: Convert one of the playpen tests over.
  • 3: Get a sample plugin containing that test building and running in the Sample Browser.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Thu May 26, 2011 9:26 am
by Praetorian
Alright, so I finished all 3 goals, just pushed to the fork. I have a sample plugin with the first converted playpen test (manual LOD) derived from VisualTest, building and running in the Sample Browser.

I md5summed the shots generated by a few runs of the test and it looks like it's generating identical shots so far.

EDIT: As for tomorrow, I'll probably be tied up with schoolwork again, but I might end up converting a few more playpen tests over if I have some spare time. Friday and over the weekend I'll definitely be able to put a good deal of time in.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Thu May 26, 2011 9:36 am
by Assaf Raman
Good work.
See you Friday.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Sat May 28, 2011 12:14 am
by Praetorian
So I had more extra time than I expected yesterday, and between yesterday evening and this morning, I got all of the initial playpen tests converted and running in the sample browser.

I think my main goal for tonight will be to tidy things up, and make sure I'm still getting correct screenshots. Provided I don't hit any significant issues, I may begin working on the TestContext.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Sat May 28, 2011 1:15 am
by Assaf Raman
Excellent. Good progress. :D

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Sun May 29, 2011 8:42 am
by Praetorian
Alright, I forgot to post this morning, but I got a basic SampleContext up and running today, it goes through the tests automatically taking screenshots and quits immediately after it's done. At the moment the screenshots are just saved in the bin directory.

My final exams are coming up week after next, so I'll probably be less active as they get closer, but at this point I'm already ahead of my original schedule, so I think it'll be alright.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Sat Jun 04, 2011 6:53 pm
by Praetorian
Well, I've done a little tidying up lately, but I'm sorta in full final exams studying mode now.

I'll be back after Thursday or Friday with school out and no other obligations.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Sat Jun 04, 2011 7:04 pm
by Assaf Raman
good luck with all the exams. :D

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Sun Jun 05, 2011 11:18 pm
by omniter
I think this is a really cool project, Praetorian. I'd be happy to answer any questions you have about the OgreBites framework. Good luck with exams! :D

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Sat Jun 11, 2011 10:00 pm
by Assaf Raman
Hi, when do you think you are going to get back to work on the project?

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Sat Jun 11, 2011 10:27 pm
by Praetorian
Right now :)

I had my last final yesterday and just finished getting all moved back home.

I'll be working on an initial version of the image comparison tool this week.

My goal for today will be to get the basics set up for the comparison tool's plugin, as well as working out a way of defining 'sets' of images for the tool to compare (most likely a simple config file with a list of the screenshots from a particular test run and some basic info about the set (resolution, perhaps a build or version number, etc)).

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Sat Jun 11, 2011 10:38 pm
by Assaf Raman
Can you explain more about this "image comparison tool" and the work you plan to do on it?

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Sun Jun 12, 2011 2:51 am
by Praetorian
The ultimate goal will be to make a simple, but flexible image diffing tool that lets you manually examine differences in the test screenshots.

For this week I plan on getting the basics in place: letting the user select a pair of test sets to compare, and then allowing them to flip through the images with the arrow keys; more advanced methods of overlaying and comparing the images will come in a later week (as well as some automated comparison to identify the images with the most differences).

It will be implemented in a sample plugin, so that it can run directly from the test context (so you can immediately inspect the results after running a batch of tests), or from the sample browser.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Sun Jun 12, 2011 7:01 am
by Assaf Raman
Why do we need such a tool? This feel to me really off topic.
We talked about an HTML output - why do we need a complicated tool? In what the tool better then the HTML output?

What need for OGRE is the option to run a sort of batch process, I was planning to take your work and have a process where every night it is run, and uploaded to a web site - giving the community a day by day status of the project.

The tool can be nice, but not much then that. The focus should be giving a summary output of the sate of the project. This is what we need.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Sun Jun 12, 2011 8:08 am
by Praetorian
Assaf Raman wrote:Why do we need such a tool? This feel to me really off topic.
We talked about an HTML output - why do we need a complicated tool? In what the tool better then the HTML output?

What need for OGRE is the option to run a sort of batch process, I was planning to take your work and have a process where every night it is run, and uploaded to a web site - giving the community a day by day status of the project.

The tool can be nice, but not much then that. The focus should be giving a summary output of the sate of the project. This is what we need.
Hmmm, alright then. tuan kuranes had suggested this sort of thing early on and I'd assumed it would be helpful, but I suppose HTML output alone would be sufficient. I could see such a comparison tool being helpful by allowing for some advanced comparison modes (using shaders to highlight differences could be interesting), but I do suspect it would be a lot of work for not a whole lot of actual practical use.

I'd set aside a good deal of time in the schedule for this, so I'll need to rework a bit, but this should make things simpler and give me more time to focus on other aspects of the project.

I think I'll take the rest of tonight and tomorrow to work out the HTML output formatting then.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Sun Jun 12, 2011 8:20 am
by Assaf Raman
Sounds good, if we can have a good nightly unit testing status for many part of OGRE that is updated to the web (I will help with a computer to run the unit tests after the summer project is done) - I consider the project a win.
I may improve the project later to get each of the change sets from OGRE source control one by one - so I can get a notification to all the community when with the exact change that broke OGRE, but this is not a must part of your project (will happy if you will help me with that also - but your focus is creating a good code base for running relevant tests visual test of OGRE render results).
Anything else like the tool - nice to have (meaning if we have time for it - great - else - no worry).

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Mon Jun 13, 2011 5:33 am
by Praetorian
Alright, well here's an initial mockup of the HTML output (the final product will include a lot more statistics and details, but I think the basic idea is here): http://www.rileyadams.net/GSoC/Mockup/output.html

I'm wondering if it'd work better to have a separate overview page that links to individual pages for each test, to prevent having to load a ton of images at once?

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Mon Jun 13, 2011 6:27 am
by Assaf Raman
Looks good.
A page for each is not that of a good idea, but perhaps a page the ones that failed and a different page for passed.
Later you can also add a "diff" image or the sort to help see where the failure is (only to failed...)

You are doing great work! Nice touch with the colors of the HTML page.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Mon Jun 13, 2011 10:20 am
by omniter
Perhaps you could output linked HTML files? For example, a summary page that links (via relative paths) to other pages for individual tests.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Mon Jun 13, 2011 11:07 am
by Assaf Raman
I don't see the problem with multiple images on the same page for now.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Mon Jun 13, 2011 11:14 am
by CABAListic
Me neither. First and foremost I think that the results would be generated and viewed on your own computer, so there is no loading time involved even for many images. And I think a single page is simpler to use.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Mon Jun 13, 2011 8:36 pm
by Praetorian
Perhaps later on there could be an option or two for output formatting (separate pages, include only failed tests, etc)?

For now I think this should be fine though, it wouldn't be hard to make some changes later on.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Mon Jun 13, 2011 9:24 pm
by Assaf Raman
Yes, don't worry about it, an easy fix later on.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Mon Jun 13, 2011 10:07 pm
by Wolfmanfx
Take into account an xml or file output so that the results can be evaluated from build server. If you stick with an standard xml output (like cppunit) most build server can parse these xml files.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Tue Jun 14, 2011 10:45 am
by Praetorian
I think this week I'll generally be working towards actually getting this HTML output generating from the framework. Today I organized the screenshot output a bit more (a uniquely named directory is created to hold the screenshots and extra data each time a batch of tests is run, it also outputs a small config file with some info about the run, to be used in comparisons). I also cleaned/reorganized a few parts of the TestContext and VisualTest classes.
Wolfmanfx wrote:Take into account an xml or file output so that the results can be evaluated from build server. If you stick with an standard xml output (like cppunit) most build server can parse these xml files.
I'm not especially familiar with build server stuff, but given the specs I could definitely make it generate output that's usable there.