Page 5 of 8

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Tue Jun 14, 2011 10:57 am
by Assaf Raman
Create a different folder for each render system - some samples look differently when using different render systems - and this is fine, in the end I will run the unit tests loop once per each render system.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Tue Jun 14, 2011 12:05 pm
by CABAListic
When creating output, please try to reuse the filesystem abstraction I implemented for the SampleBrowser, so that output is created in the user's home directory. Otherwise, on many platforms there could be problems with writing the output due to missing permissions.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Tue Jun 14, 2011 10:33 pm
by Praetorian
CABAListic wrote:When creating output, please try to reuse the filesystem abstraction I implemented for the SampleBrowser, so that output is created in the user's home directory. Otherwise, on many platforms there could be problems with writing the output due to missing permissions.
Yep, this is exactly what I've done, everything's saved in the home directory (located via FileSystemLayerImpl::getWritablePath) I did need to add a "createDirectory" function for easily adding folders for each run (unless there's already a way of doing this that I missed?), but everything else was reused.
Assaf Raman wrote:Create a different folder for each render system - some samples look differently when using different render systems - and this is fine, in the end I will run the unit tests loop once per each render system.
Sounds good, will do.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Thu Jun 16, 2011 12:36 am
by Praetorian
I started working on the output/comparison end of things yesterday (now when you run a batch of tests, it automatically looks for the most recent prior test run and does the comparison against that (though there isn't any actual image comparison being done just yet)).

I'm splitting this functionality into a separate header so that it can be reused elsewhere (I think a small command line tool for comparing test results might be a useful addition at some point (if you wanted to do some sort of batch comparison against a bunch of prior runs, or compare results from different rendersystems, etc)).

It also splits into different folders depending on rendersystem now.

I think I'll look into the specifics of generating the actual HTML today. I know Ogre uses TinyXML elsewhere, so I think I'll try playing with that.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Thu Jun 16, 2011 1:21 am
by spacegaier
Praetorian wrote:I know Ogre uses TinyXML elsewhere, so I think I'll try playing with that.
The only place I found was the XMLConverter tool. So there is no official dependency. Some forum users also advise to use RapidXML (see alternative DotSceneLoader here in the forums).

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Thu Jun 16, 2011 3:18 am
by Praetorian
spacegaier wrote:
Praetorian wrote:I know Ogre uses TinyXML elsewhere, so I think I'll try playing with that.
The only place I found was the XMLConverter tool. So there is no official dependency. Some forum users also advise to use RapidXML (see alternative DotSceneLoader here in the forums).
Hmm, alright, thanks for the heads-up. I'll give RapidXML a try as well (I like that it's header-only, and it looks a little lighter, tinyxml seemed like it required a lot of extra files...). When it comes down to it, all this will be doing is spitting out an HTML doc; there's no need to parse anything, so the actual choice probably won't make much difference...

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Thu Jun 16, 2011 7:10 am
by Wolfmanfx
To generate html you do not any lib. I will post an xml template today when i am at work. Cmd line version will be really useful -buildserver- ;) . Keep up the nice work!

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Thu Jun 16, 2011 7:43 am
by Praetorian
Wolfmanfx wrote:To generate html you do not any lib. I will post an xml template today when i am at work. Cmd line version will be really useful -buildserver- ;) . Keep up the nice work!
Cool, thanks! And yeah I'm realizing I was overthinking, and it'll be easier to do the HTML without an external lib. I thought it might be cleaner/more flexible to use some preexisting XML lib, but HTML is simple enough, and no parsing is needed. (I did give rapidxml a try, but it was rather verbose syntax-wise and it started giving me some strange memory issues after the document passed a certain length...).

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Thu Jun 16, 2011 7:47 am
by Assaf Raman
Doesn't really matter what you do as long:
1. The code is well documented and maintainable.
2. The code is cross platformed.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Thu Jun 16, 2011 1:12 pm
by Wolfmanfx

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Fri Jun 17, 2011 1:32 am
by Praetorian
Okay, so I wound up doing a really simple custom solution for the html, and it seems to work just fine. I haven't gotten everything all set up just yet but here's a snapshot of the output as generated now (whether or not tests "passed" is randomized at the moment, since no actual image comparison is set up yet): http://www.rileyadams.net/GSoC/output/out.html

I think today I'll continue on the output formatting (add in descriptions of tests and general info about the test runs, perhaps tidy up the html generation a bit as well).

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Mon Jun 20, 2011 6:50 pm
by Praetorian
So this weekend I didn't make too much in the way of functional progress, but I did a big rework and cleanup of a lot of the code for comparing test runs and generating the HTML. Also, the HTML output itself is pretty much set, I'll just need to add some extra stats and info when I get into the image comparison functionality.

I think today I'll get set up in Windows and make sure everything's working cross-platform. Depending on how that works out, I may also start adding some more options and functionality to the TestContext.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Fri Jun 24, 2011 8:13 pm
by Assaf Raman
Hi, what are the plans for this weekend?

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Fri Jun 24, 2011 9:55 pm
by Praetorian
I think I'll be making a couple tests in a new plugin to make sure it handles multiple plugins properly, and to test for some potential edge cases (I'll probably do one that tests particles, perhaps one with shadows, etc).

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Mon Jun 27, 2011 3:32 am
by Praetorian
Okay, I made 2 new tests (testing stencil shadows and particle effects), in a new test plugin, and everything seems to be working.

I also wrote some javascript for the HTML output that allows for generating diff images from within the browser using Canvas. Due to browser security settings (you can't access image data locally or cross-domain without browser-specific workarounds) and generally inconsistent browser support for HTML5/Canvas I'm not sure how practical/useful this actually is, but it was an interesting excuse to play with javascript for a bit. It seems to work fine in Chrome and FF on my machine, but I haven't tested anywhere else.

Here's the output at the moment for the new test plugin, with the added javascript. As before I'm not doing any meaningful image comparison yet (at the moment I just open both images with ifstreams and compare byte by byte...). I distorted the last image to make the diff actually show something.
http://rileyadams.net/GSoC/Out2/out.html

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Mon Jun 27, 2011 7:32 am
by Assaf Raman
Nice output.
Regarding the java script - nice, but a bit off topic.
What are the next goals?
I am not sure where we should focus, but I feel that getting more tests to our list is a good goal, it is a big part of the project, and I fear of having a good framework at the end - with a short list of tests that don't cover a lot of topics.

Regarding the comparison - the current image should be compared to a "success" image and not an image from the last run. If the last run had bad results - this run will show success when it shouldn't.

Just to be clear how this project is going to be used in the future after the summer is done:
In the end what the project need is a framework that runs on a host, constantly gets the code from the source control, then try to cmake, build and run each change-set. If the cmake, build or an image comparison in the run fails (and didn't in the previous change-set) - it sends an e-mail based on the e-mail of the source control user that committed the change-set. In any case it uploads the results to a web server so the community can get daily update of the status of the project.

Blame and status, that can be nice.

If you can create that full framework, that can be great, we defined the project at start to cover less - none of the source control\cmake\build\e-mail\upload issues - so you don't have do any that (it will be nice if you do - but you don't have to). If in the end of the summer I will have a framework to the batch test of many different aspects of OGRE (see playpen) with image comparison, that will be super, I will do the rest later - so finishing that - is the focus I see it. The rest is nice to have.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Mon Jun 27, 2011 9:35 am
by Praetorian
Thanks for the clarification, that helps a lot.

I'll set up a way of defining a "success" image set, and have comparisons done against this rather than the previous set.

I think I'll work on adding more tests over the next couple days (perhaps convert a few more of the existing playpen tests over).

Also, a question: since the main usage is going to be automated testing, is there any need for interaction within the test context (i.e. an optional menu screen with settings)? I'll definitely be making a good set of command line options for it, and the sample browser can run the tests on their own, so I don't see a huge need, but I did originally have it in my schedule.

I'd definitely like to help as much as I can with the extra build and automation stuff (after the other parts are set up and working completely, of course). Can we make any assumptions about the host that this will run on? I have very little experience with Windows batch scripts, but I think a lot of that would be pretty easily doable in Unix with a bash script scheduled with cron.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Mon Jun 27, 2011 9:44 am
by Assaf Raman
Regarding UI - yes, not really important.

Regarding batching - you can do batching with cmake - this way it will be cross platformed. Meaning - creating a new cmake project the does the batching. I didn't get to really learn all the options of cdash (an open source, web-based software testing server) - but I think that it works with cmake. There are all sort of open source tools from Kitware that we can try.

OGRE is cross platformed, it can be nice if we have something that is also cross platformed.

But for your question my target machine will be windows, the only os that can run both dx and gl.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Mon Jun 27, 2011 9:57 am
by Praetorian
Ahh, cool, I wasn't aware of cdash, I'll read up on its specifics when I get some time.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Mon Jun 27, 2011 10:27 am
by CABAListic
@Assaf: Are you sure an automated build test server is the primary goal for this project? Do we even have anything like that? One wouldn't be enough, we'd need different graphics cards, given the differences between Nvidia and Ati.

I was thinking that first and foremost we as developers would regularly run those tests to ensure that we didn't break any functionality. As such, we do need some kind of database of success screenshots, which may or may not need to differ per platform. (Small differences between Ati and Nvidia and possibly OpenGL / DirectX are to be expected.) We also need a way to enter a new 'success' screenshot into that database, either for a new test or for a new platform that wasn't covered before.

Of course, an automated test server would be great, but I consider that to be second phase.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Mon Jun 27, 2011 1:01 pm
by Assaf Raman
@CABAListic: Regarding automated build test server - I clearly defined it as a second phase (and not a must part of the project).

Regarding the "success screenshots", I was waiting for that question :D - the answer is that for each test a "good changeset" will be defined per render system (and perhaps platform) - there will be a process where the images are created on a "new" computer by getting change set by change set, compiling it, saving the relevant "success screenshots" and so on. This process is needed once per computer per sample, meaning the next time - only if a new sample was added or an existing updated - only the relevant "success screenshots" will be created for that sample, meaning it will normally take some time once per computer.

This is the easiest way I see to make sure we don't get a false positive because of different hardware.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Mon Jun 27, 2011 1:08 pm
by CABAListic
Ah, so basically you would require everyone to create their own "good" sample, possibly by comparing it manually against a public gallery? (Due to manual comparison, the differences in hardware wouldn't matter, since the human eye doesn't compare pixel by pixel.) And then in future the tests would automatically compare against your own set, whereby the comparison could be pretty much pixel by pixel, since the same hardware should be able to reproduce its own test results. Yeah, okay, that could work.

What happens if in your initial run you do spot a bug in manual comparison to the expected result?

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Mon Jun 27, 2011 1:15 pm
by Assaf Raman
That means that since the "good change set" - something broke - the image is different now.
If you accept the change - update the "good change set" to be the current one, else - you have an issue on the todo list.
I am not going to send an e-mails for the first run that compares the current images to the "good images", only for new ones after that.
And if there are issues - that is fine, that is what we want to know, right?
BTW: some samples don't run on some render system - so there will be no "good image" for that sample for that render system.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Sat Jul 02, 2011 3:57 am
by Praetorian
Okay, I had some computer troubles, so apologies for going quiet for a bit there.

I've added some command line options to the Test Context, including setting it up so you can create a "success" or reference set that everything will be compared against (this is done with the command line argument -r). You can also add a comment to a run, choose the name for the output image set, choose an alternate image set to compare against, etc. I'll document all the options in the wiki, in addition to adding a --help option with usage details.

I also added 4 tests (1 converted playpen test, and 3 new ones based on pieces of existing samples).

Here's the output with all the current tests (as before I've distorted a few images manually to make a few of the tests fail): http://rileyadams.net/gsoc/July1/out.html

I think I'll be working on the image comparison this weekend.

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Posted: Sat Jul 02, 2011 8:37 am
by Assaf Raman
Nice progress.
What do you want to work on regarding image comparison?