[GSoC 2011 - Accepted] Unit Testing Framework

Threads related to Google Summer of Code
Post Reply
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Hi,
Can you please post a list of the remaining tasks for this project?
Watch out for my OGRE related tweets here.
User avatar
boyamer
Orc
Posts: 459
Joined: Sat Jan 24, 2009 11:16 am
Location: Italy
x 6

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by boyamer »

Seams that PlayPen_ManualBlending is missing.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Sorry for the lack of updates, I'm still working through more playpen tests (75 total tests at the moment), it's been slowing down since I'm down to the harder to convert tests, and I've been needing to add fixes for some of the already converted ones.

I think I'm almost as far as I can get with the playpen (another day or two, and then I'll post a list of the tests that I wasn't able to convert for various reasons).

After that I think I'm almost done with most of the framework itself, I'll need to tidy up in a few places, and add a few command line options, but I can't think of anything big that's needed.

I think the next step might be to start looking into cDash (namely what sort of output format the framework will need to write for cDash to recognize which tests passed/failed, and any other changes/additions that may be needed for it to work).
boyamer wrote:Seams that PlayPen_ManualBlending is missing.
Hmmm, I may have left something out in a commit, I'll see what I can do.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Thanks for the update. Sounds good.
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Alright, just about set with Playpen stuff. Up to 101 tests.

Here's a list of the tests that I didn't/couldn't convert:

Code: Select all

Need media/materials
-testCthNewBlending
-testGpuPrograms
-testAnimation
-testAnimationBlend
-testAlpa
-testOverlayRelativeMode
-testSimpleMesh
-testOverlayZOrder
-testGLSLTangent
-testBug
Not Visual (console output, or otherwise not suitable for visual testing)
-testManualLoader
-testEdgeBuilderSingleIndexBufSingleVertexBuf
-testEdgeBuilderMultiIndexBufSingleVertexBuf
-testEdgeBuilderMultiIndexBufMultiVertexBuf
-testMaterialSerializer
-testMatrices
-test2Windows
-testRadixSort
-testStringTokenising
-testSplitPassesTooManyTexUnits
-testExportPrecompiledAssemblerProgram
-testBackgroundLoadResourceGroup
-testTimeCreateDestroyObject
-testSharedPtrBug
-testPrepareShadowVolume
Incomplete:
-testMaterial
Used terrain stuff that has changed:
-testNewTerrain
-testTwoNewTerrains
-testMultiSceneManager
Used Lod Functions that no longer exist:
-testLod
-testGenerateLod
My video card/setup didn't work with it, so I couldn't verify it works:
-testMRT
-testMRTCompositorScript
-testFloat16DDS
-testFloat32DDS
-testFloat64DDS
-testFloat128DDS
-test4bppPVR
-test4bppAlphaPVR
-test2bppPVR
-test2bppAlphaPVR
Next I'm going to start reading into cDash to see what sort of output and additions will be necessary.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Sound good. I will also read about it and try to understand better.
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

I'm still not totally sure on how to actually set up a build site to submit the build/test results (or if it's easily possible to do so without using cTest), but I did set up a testing dashboard on my personal webhosting: http://rileyadams.net/CDash/index.php?project=OGRE3D
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Do you want me to contact Kitware and get a free CDashPro account?
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

As far as I can tell Pro doesn't offer any additional features (seems like it's just hosting?), so I don't think it's necessary for now (later it may be useful, but for the moment I think using it on my web hosting will be a bit more flexible).

CDash's documentation is a little lacking, but I did find the XML schemas for submitting build/test results (http://www.vtk.org/Wiki/CDash:XML). So I'll just have to dig around a bit for the specifics of submitting the xml files.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

I hadn't realized how integrated CTest is with CMake (it's included in every CMake release); so I was actually able to set up test result submission without needing to set up any extra XML output for the test framework. CTest integrates nicely with the build system ("make test" with makefiles, and a "RUN TESTS" target in Visual Studio), and uploading results to CDash is easy as well (via makefile and VS targets).

I'm still working on making sure the setup runs properly in different build configurations and environments, and I'll need to make everything a bit more flexible, but so far so good (I'm able to upload results to the dashboard from both Linux and Windows (VS10)).

Here's a link to the dashboard (if no builds are showing up, hit 'previous' to see builds from the previous day): http://rileyadams.net/CDash/index.php?project=OGRE3D

On the web end, CDash doesn't seem to support Mercurial (it allows for some integration with repository browsers and such, but looks like SVN/CVS/git only), however for build clients, it should be possible to set up a script for continuous builds that grabs changes from the repo (there's an example that uses git for this on the Kitware wiki).
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Looks great, this is really good progress.
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Alight, I've got the testing setup fairly set, it runs the tests for each built rendersystem and everything integrates and uploads nicely. I've been doing some random cleanup over the past couple days.

Right now I looking into continuous/nightly builds a bit more. CTest has a scripting format that makes this pretty easily doable (I've managed to get cloning from the repo, building, testing and uploading working all from CTest with one short script). Since there's a lot of setup details (scheduling, build options, where the builds go, choice of project/makefile generator, etc) that aren't especially general, I think it'd probably work best to make a basic template for this sort of script and put it on the wiki with some setup instructions.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Wonderful.
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Okay, I've got nightly/continuous build scripts working fairly well in both Linux and Windows. In order to automate it in Windows I've had to use CMake's NMake generator (I don't know of any other way of building with MSVC++ from the commandline), which isn't very well supported (precompiled headers don't work, see this issue), but apart from one test producing different results (the PSSM playpen test in GL), which I'm guessing is due to some change caused by not using precompiled headers, it seems to be working well enough.

I'll be updating the wiki (including the nightly/continuous build scripts) tomorrow morning. Apart from some small cleanup and documentation, I think I'm about set for the deadline.

There's a Ludum Dare 48 hour game dev contest I'd like to take part in this weekend, so I think I'll be finishing up a little early; but I should still be available, and I've got a month or so free after that before my school starts up again, so I'll be around to make any additional fixes/improvements.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Okay, I've updated the wiki and gone through the code a couple times, I think I'm set for the deadline. Ludum Dare starts in a few hours, so I'll be working on that, but as I said I'll still be available and checking back here.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Sounds good. Thanks for the update.
Watch out for my OGRE related tweets here.
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Congratulation, you did excellent work in this project.
I have just given you a passing grade.
Make sure to commit all your changes and upload to your fork.
I am going to marge everything you worked on to OGRE's trunk.
I am also going to setup a build server on one of my old computers - so your work is going to be used.
I don't know how to thank you for this work - you did everything we wanted and more.
I hope you will stay involved with our small community even after the summer.
Also - if you have the time next summer - apply for a project next year.

Great work. Great student.
Watch out for my OGRE related tweets here.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Thanks! You've been a great mentor and this has been an awesome summer! :D

I'll definitely keep up with the community, and I'll keep this in mind for next year.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
User avatar
Jabberwocky
OGRE Moderator
OGRE Moderator
Posts: 2819
Joined: Mon Mar 05, 2007 11:17 pm
Location: Canada
x 218
Contact:

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Jabberwocky »

Thumbs up to both Praetorian and Assaf for this.

A Unit Testing Framework is sort of a behind-the-scenes feature, but anything that helps keep Ogre more stable and bug-free is full on awesome.
Image
User avatar
jacmoe
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 20570
Joined: Thu Jan 22, 2004 10:13 am
Location: Denmark
x 179
Contact:

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by jacmoe »

Yeah, it is. :)
And, who knows: maybe we can use it in our own Ogre applications.
A really useful project.
/* Less noise. More signal. */
Ogitor Scenebuilder - powered by Ogre, presented by Qt, fueled by Passion.
OgreAddons - the Ogre code suppository.
User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Assaf Raman »

Merged to the trunk: https://bitbucket.org/sinbad/ogre/chang ... 2092773fb/

I am going to add D3D11 render system support next.
Watch out for my OGRE related tweets here.
dermont
Bugbear
Posts: 812
Joined: Thu Dec 09, 2004 2:51 am
x 42

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by dermont »

Assaf Raman wrote:Merged to the trunk: https://bitbucket.org/sinbad/ogre/chang ... 2092773fb/

I am going to add D3D11 render system support next.
For me the test build fails on Linux - same error as the following link. Has this been updated? It doesn't appear possible build the Unit Testing Framework without building the tests.
http://www.ogre3d.org/forums/viewtopic.php?f=1&t=64537

Anyway nice job Praetorian, the OGRE Visual Test Output looks great.
User avatar
Praetorian
Google Summer of Code Student
Google Summer of Code Student
Posts: 171
Joined: Fri Aug 10, 2007 10:37 pm
Location: WA - USA
x 5

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by Praetorian »

Ah yes looks like that has been broken for some time, I'd never had cppUnit installed, so CMake wasn't finding it and that set of tests was never getting built, just reproduced it on my Linux machine (should fail in Windows too, if cppUnit is installed and found by CMake). I tried the patch in the thread you linked and that fixes it.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
dermont
Bugbear
Posts: 812
Joined: Thu Dec 09, 2004 2:51 am
x 42

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by dermont »

Praetorian wrote:Ah yes looks like that has been broken for some time, I'd never had cppUnit installed, so CMake wasn't finding it and that set of tests was never getting built, just reproduced it on my Linux machine (should fail in Windows too, if cppUnit is installed and found by CMake). I tried the patch in the thread you linked and that fixes it.
Ok I see, but this still hasn't been fixed.
CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
x 58
Contact:

Re: [GSoC 2011 - Accepted] Unit Testing Framework

Post by CABAListic »

Is there an entry in the patch or bug tracker concerning this? If not, please open one - we are going to tackle those lists soon in preparation for the release of 1.8, and posts in this forum easily get lost or simply forgotten.
Post Reply