[GSoC 2011 - Accepted] Unit Testing Framework
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Hi,
Can you please post a list of the remaining tasks for this project?
Can you please post a list of the remaining tasks for this project?
Watch out for my OGRE related tweets here.
- boyamer
- Orc
- Posts: 459
- Joined: Sat Jan 24, 2009 11:16 am
- Location: Italy
- x 6
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Seams that PlayPen_ManualBlending is missing.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Sorry for the lack of updates, I'm still working through more playpen tests (75 total tests at the moment), it's been slowing down since I'm down to the harder to convert tests, and I've been needing to add fixes for some of the already converted ones.
I think I'm almost as far as I can get with the playpen (another day or two, and then I'll post a list of the tests that I wasn't able to convert for various reasons).
After that I think I'm almost done with most of the framework itself, I'll need to tidy up in a few places, and add a few command line options, but I can't think of anything big that's needed.
I think the next step might be to start looking into cDash (namely what sort of output format the framework will need to write for cDash to recognize which tests passed/failed, and any other changes/additions that may be needed for it to work).
I think I'm almost as far as I can get with the playpen (another day or two, and then I'll post a list of the tests that I wasn't able to convert for various reasons).
After that I think I'm almost done with most of the framework itself, I'll need to tidy up in a few places, and add a few command line options, but I can't think of anything big that's needed.
I think the next step might be to start looking into cDash (namely what sort of output format the framework will need to write for cDash to recognize which tests passed/failed, and any other changes/additions that may be needed for it to work).
Hmmm, I may have left something out in a commit, I'll see what I can do.boyamer wrote:Seams that PlayPen_ManualBlending is missing.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Thanks for the update. Sounds good.
Watch out for my OGRE related tweets here.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Alright, just about set with Playpen stuff. Up to 101 tests.
Here's a list of the tests that I didn't/couldn't convert:
Next I'm going to start reading into cDash to see what sort of output and additions will be necessary.
Here's a list of the tests that I didn't/couldn't convert:
Code: Select all
Need media/materials
-testCthNewBlending
-testGpuPrograms
-testAnimation
-testAnimationBlend
-testAlpa
-testOverlayRelativeMode
-testSimpleMesh
-testOverlayZOrder
-testGLSLTangent
-testBug
Not Visual (console output, or otherwise not suitable for visual testing)
-testManualLoader
-testEdgeBuilderSingleIndexBufSingleVertexBuf
-testEdgeBuilderMultiIndexBufSingleVertexBuf
-testEdgeBuilderMultiIndexBufMultiVertexBuf
-testMaterialSerializer
-testMatrices
-test2Windows
-testRadixSort
-testStringTokenising
-testSplitPassesTooManyTexUnits
-testExportPrecompiledAssemblerProgram
-testBackgroundLoadResourceGroup
-testTimeCreateDestroyObject
-testSharedPtrBug
-testPrepareShadowVolume
Incomplete:
-testMaterial
Used terrain stuff that has changed:
-testNewTerrain
-testTwoNewTerrains
-testMultiSceneManager
Used Lod Functions that no longer exist:
-testLod
-testGenerateLod
My video card/setup didn't work with it, so I couldn't verify it works:
-testMRT
-testMRTCompositorScript
-testFloat16DDS
-testFloat32DDS
-testFloat64DDS
-testFloat128DDS
-test4bppPVR
-test4bppAlphaPVR
-test2bppPVR
-test2bppAlphaPVR
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Sound good. I will also read about it and try to understand better.
Watch out for my OGRE related tweets here.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
I'm still not totally sure on how to actually set up a build site to submit the build/test results (or if it's easily possible to do so without using cTest), but I did set up a testing dashboard on my personal webhosting: http://rileyadams.net/CDash/index.php?project=OGRE3D
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Do you want me to contact Kitware and get a free CDashPro account?
Watch out for my OGRE related tweets here.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
As far as I can tell Pro doesn't offer any additional features (seems like it's just hosting?), so I don't think it's necessary for now (later it may be useful, but for the moment I think using it on my web hosting will be a bit more flexible).
CDash's documentation is a little lacking, but I did find the XML schemas for submitting build/test results (http://www.vtk.org/Wiki/CDash:XML). So I'll just have to dig around a bit for the specifics of submitting the xml files.
CDash's documentation is a little lacking, but I did find the XML schemas for submitting build/test results (http://www.vtk.org/Wiki/CDash:XML). So I'll just have to dig around a bit for the specifics of submitting the xml files.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
I hadn't realized how integrated CTest is with CMake (it's included in every CMake release); so I was actually able to set up test result submission without needing to set up any extra XML output for the test framework. CTest integrates nicely with the build system ("make test" with makefiles, and a "RUN TESTS" target in Visual Studio), and uploading results to CDash is easy as well (via makefile and VS targets).
I'm still working on making sure the setup runs properly in different build configurations and environments, and I'll need to make everything a bit more flexible, but so far so good (I'm able to upload results to the dashboard from both Linux and Windows (VS10)).
Here's a link to the dashboard (if no builds are showing up, hit 'previous' to see builds from the previous day): http://rileyadams.net/CDash/index.php?project=OGRE3D
On the web end, CDash doesn't seem to support Mercurial (it allows for some integration with repository browsers and such, but looks like SVN/CVS/git only), however for build clients, it should be possible to set up a script for continuous builds that grabs changes from the repo (there's an example that uses git for this on the Kitware wiki).
I'm still working on making sure the setup runs properly in different build configurations and environments, and I'll need to make everything a bit more flexible, but so far so good (I'm able to upload results to the dashboard from both Linux and Windows (VS10)).
Here's a link to the dashboard (if no builds are showing up, hit 'previous' to see builds from the previous day): http://rileyadams.net/CDash/index.php?project=OGRE3D
On the web end, CDash doesn't seem to support Mercurial (it allows for some integration with repository browsers and such, but looks like SVN/CVS/git only), however for build clients, it should be possible to set up a script for continuous builds that grabs changes from the repo (there's an example that uses git for this on the Kitware wiki).
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Looks great, this is really good progress.
Watch out for my OGRE related tweets here.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Alight, I've got the testing setup fairly set, it runs the tests for each built rendersystem and everything integrates and uploads nicely. I've been doing some random cleanup over the past couple days.
Right now I looking into continuous/nightly builds a bit more. CTest has a scripting format that makes this pretty easily doable (I've managed to get cloning from the repo, building, testing and uploading working all from CTest with one short script). Since there's a lot of setup details (scheduling, build options, where the builds go, choice of project/makefile generator, etc) that aren't especially general, I think it'd probably work best to make a basic template for this sort of script and put it on the wiki with some setup instructions.
Right now I looking into continuous/nightly builds a bit more. CTest has a scripting format that makes this pretty easily doable (I've managed to get cloning from the repo, building, testing and uploading working all from CTest with one short script). Since there's a lot of setup details (scheduling, build options, where the builds go, choice of project/makefile generator, etc) that aren't especially general, I think it'd probably work best to make a basic template for this sort of script and put it on the wiki with some setup instructions.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Wonderful.
Watch out for my OGRE related tweets here.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Okay, I've got nightly/continuous build scripts working fairly well in both Linux and Windows. In order to automate it in Windows I've had to use CMake's NMake generator (I don't know of any other way of building with MSVC++ from the commandline), which isn't very well supported (precompiled headers don't work, see this issue), but apart from one test producing different results (the PSSM playpen test in GL), which I'm guessing is due to some change caused by not using precompiled headers, it seems to be working well enough.
I'll be updating the wiki (including the nightly/continuous build scripts) tomorrow morning. Apart from some small cleanup and documentation, I think I'm about set for the deadline.
There's a Ludum Dare 48 hour game dev contest I'd like to take part in this weekend, so I think I'll be finishing up a little early; but I should still be available, and I've got a month or so free after that before my school starts up again, so I'll be around to make any additional fixes/improvements.
I'll be updating the wiki (including the nightly/continuous build scripts) tomorrow morning. Apart from some small cleanup and documentation, I think I'm about set for the deadline.
There's a Ludum Dare 48 hour game dev contest I'd like to take part in this weekend, so I think I'll be finishing up a little early; but I should still be available, and I've got a month or so free after that before my school starts up again, so I'll be around to make any additional fixes/improvements.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Okay, I've updated the wiki and gone through the code a couple times, I think I'm set for the deadline. Ludum Dare starts in a few hours, so I'll be working on that, but as I said I'll still be available and checking back here.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Sounds good. Thanks for the update.
Watch out for my OGRE related tweets here.
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Congratulation, you did excellent work in this project.
I have just given you a passing grade.
Make sure to commit all your changes and upload to your fork.
I am going to marge everything you worked on to OGRE's trunk.
I am also going to setup a build server on one of my old computers - so your work is going to be used.
I don't know how to thank you for this work - you did everything we wanted and more.
I hope you will stay involved with our small community even after the summer.
Also - if you have the time next summer - apply for a project next year.
Great work. Great student.
I have just given you a passing grade.
Make sure to commit all your changes and upload to your fork.
I am going to marge everything you worked on to OGRE's trunk.
I am also going to setup a build server on one of my old computers - so your work is going to be used.
I don't know how to thank you for this work - you did everything we wanted and more.
I hope you will stay involved with our small community even after the summer.
Also - if you have the time next summer - apply for a project next year.
Great work. Great student.
Watch out for my OGRE related tweets here.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Thanks! You've been a great mentor and this has been an awesome summer!
I'll definitely keep up with the community, and I'll keep this in mind for next year.
I'll definitely keep up with the community, and I'll keep this in mind for next year.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
- Jabberwocky
- OGRE Moderator
- Posts: 2819
- Joined: Mon Mar 05, 2007 11:17 pm
- Location: Canada
- x 218
- Contact:
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Thumbs up to both Praetorian and Assaf for this.
A Unit Testing Framework is sort of a behind-the-scenes feature, but anything that helps keep Ogre more stable and bug-free is full on awesome.
A Unit Testing Framework is sort of a behind-the-scenes feature, but anything that helps keep Ogre more stable and bug-free is full on awesome.
- jacmoe
- OGRE Retired Moderator
- Posts: 20570
- Joined: Thu Jan 22, 2004 10:13 am
- Location: Denmark
- x 179
- Contact:
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Yeah, it is.
And, who knows: maybe we can use it in our own Ogre applications.
A really useful project.
And, who knows: maybe we can use it in our own Ogre applications.
A really useful project.
/* Less noise. More signal. */
Ogitor Scenebuilder - powered by Ogre, presented by Qt, fueled by Passion.
OgreAddons - the Ogre code suppository.
Ogitor Scenebuilder - powered by Ogre, presented by Qt, fueled by Passion.
OgreAddons - the Ogre code suppository.
- Assaf Raman
- OGRE Team Member
- Posts: 3092
- Joined: Tue Apr 11, 2006 3:58 pm
- Location: TLV, Israel
- x 76
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Merged to the trunk: https://bitbucket.org/sinbad/ogre/chang ... 2092773fb/
I am going to add D3D11 render system support next.
I am going to add D3D11 render system support next.
Watch out for my OGRE related tweets here.
-
- Bugbear
- Posts: 812
- Joined: Thu Dec 09, 2004 2:51 am
- x 42
Re: [GSoC 2011 - Accepted] Unit Testing Framework
For me the test build fails on Linux - same error as the following link. Has this been updated? It doesn't appear possible build the Unit Testing Framework without building the tests.Assaf Raman wrote:Merged to the trunk: https://bitbucket.org/sinbad/ogre/chang ... 2092773fb/
I am going to add D3D11 render system support next.
http://www.ogre3d.org/forums/viewtopic.php?f=1&t=64537
Anyway nice job Praetorian, the OGRE Visual Test Output looks great.
- Praetorian
- Google Summer of Code Student
- Posts: 171
- Joined: Fri Aug 10, 2007 10:37 pm
- Location: WA - USA
- x 5
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Ah yes looks like that has been broken for some time, I'd never had cppUnit installed, so CMake wasn't finding it and that set of tests was never getting built, just reproduced it on my Linux machine (should fail in Windows too, if cppUnit is installed and found by CMake). I tried the patch in the thread you linked and that fixes it.
My Google summer of code 2011 topic: Unit Testing Framework
My Google summer of code thread
My Google summer of code wiki page
My Google summer of code thread
My Google summer of code wiki page
-
- Bugbear
- Posts: 812
- Joined: Thu Dec 09, 2004 2:51 am
- x 42
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Ok I see, but this still hasn't been fixed.Praetorian wrote:Ah yes looks like that has been broken for some time, I'd never had cppUnit installed, so CMake wasn't finding it and that set of tests was never getting built, just reproduced it on my Linux machine (should fail in Windows too, if cppUnit is installed and found by CMake). I tried the patch in the thread you linked and that fixes it.
-
- OGRE Retired Team Member
- Posts: 2903
- Joined: Thu Jan 18, 2007 2:48 pm
- x 58
- Contact:
Re: [GSoC 2011 - Accepted] Unit Testing Framework
Is there an entry in the patch or bug tracker concerning this? If not, please open one - we are going to tackle those lists soon in preparation for the release of 1.8, and posts in this forum easily get lost or simply forgotten.