continuous integration

Discussion area about developing or extending OGRE, adding plugins for it or building applications on it. No newbie questions please, use the Help forum for that.
Post Reply
User avatar
tdev
Silver Sponsor
Silver Sponsor
Posts: 244
Joined: Thu Apr 12, 2007 9:21 pm
Location: Germany
x 14

continuous integration

Post by tdev »

I would like to discuss what tools we could use for ogre that would help the coders to do their work. So far i setup a buildbot instance (http://ogre3d.buildbotfarm.net/) and added the ogre svn to a fisheye instance i am running (http://188.40.228.51:8060/browse/ogre3d).

if we had automated tests we could run them on every commit. For example i added a cppcheck step in the current buildbot config that checks against c++ flaws.

Also, everyone is very invited to attach new buildslaves (windows, macos?) to the buildbot, so we can see the compatibility for every platform right after the commit. I think such a system could be helpful.
User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 2653
Joined: Wed Sep 24, 2003 8:07 am
Location: Haute Garonne, France
x 4
Contact:

Re: continuous integration

Post by tuan kuranes »

Nice !
if we had automated tests we could run them on every commit. For example i added a cppcheck step in the current buildbot config that checks against c++ flaws.
Some other tools to run :
-> clang static analysis (llvm)
-> gcc -Weffc++
-> splint
-> Duplo SourceMonitor
-> image results diff ?
Also, everyone is very invited to attach new buildslaves (windows, macos?) to the buildbot, so we can see the compatibility for every platform right after the commit. I think such a system could be helpful.
Any howtos ?
Any chance you can make an Ogre CI server config wiki howto entry ?
(could be usefaul for all projects: ogreaddons, private, where you run old ogre patched version, etc...)
User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
x 56
Contact:

Re: continuous integration

Post by Klaim »

It seems xavier is volunteer to setup a windows server to help here, see http://www.ogre3d.org/forums/viewtopic.php?f=4&t=50065

Automated tests running ogre demos and trying several inputs in each demo would be excellent.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Re: continuous integration

Post by sinbad »

Agreed, this will be a very useful thing to have. Please volunteeer machines & talk to tdev if you can!
User avatar
xavier
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 9481
Joined: Fri Feb 18, 2005 2:03 am
Location: Dublin, CA, US
x 22

Re: continuous integration

Post by xavier »

I'm not sure what "buildbotfarm" is -- is this a CI thing you rolled on your own, or is this a dedicated project? I ask because if you are rolling your own, I would suggest instead using CruiseControl/CruiseControl.NET?

http://cruisecontrol.sourceforge.net/
http://sourceforge.net/projects/ccnet/
Do you need help? What have you tried?

Image

Angels can fly because they take themselves lightly.
User avatar
tdev
Silver Sponsor
Silver Sponsor
Posts: 244
Joined: Thu Apr 12, 2007 9:21 pm
Location: Germany
x 14

Re: continuous integration

Post by tdev »

tuan kuranes wrote: Nice !
if we had automated tests we could run them on every commit. For example i added a cppcheck step in the current buildbot config that checks against c++ flaws.
Some other tools to run :
-> clang static analysis (llvm)
-> gcc -Weffc++
-> splint
-> Duplo SourceMonitor
-> image results diff ?
Also, everyone is very invited to attach new buildslaves (windows, macos?) to the buildbot, so we can see the compatibility for every platform right after the commit. I think such a system could be helpful.
Any howtos ?
Any chance you can make an Ogre CI server config wiki howto entry ?
(could be usefaul for all projects: ogreaddons, private, where you run old ogre patched version, etc...)
i looked at llvm, needs a much more complicated setup it seems. If someone has experience with it, i can create a virtual machine as build slave and give you a login.

added -Weffc++ which results in much more warnings obviously: http://ogre.buildbotfarm.net:10501/buil ... s/warnings
splint: isnt it c only? IIRC the c++ splint version was commercial? PCLint?
-> Duplo
this one? http://duplo.giants.ch
could be added

-> SourceMonitor
never heard of that
http://www.campwoodsw.com/sourcemonitor.html?

-> image results diff ?
we need test binaries for this that produce a comparable output.

if someone want to help me with the buildbot master, i am willing to give someone with high enough rank here a root account to the machine
User avatar
spookyboo
Silver Sponsor
Silver Sponsor
Posts: 1141
Joined: Tue Jul 06, 2004 5:57 am
x 149
Contact:

Re: continuous integration

Post by spookyboo »

I would suggest instead using CruiseControl/CruiseControl.NET?
We replaced CruiseControl in favour of Hudson (we use it for Java, but it can do C++ builds).
User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 2653
Joined: Wed Sep 24, 2003 8:07 am
Location: Haute Garonne, France
x 4
Contact:

Re: continuous integration

Post by tuan kuranes »

@xavier : http://buildbot.net is a python CI thingy, very powerful (open and flexible)

@tdev:
yes correct ones. (yes, splint is for C parts (ie Ogre command line tools?)
Clang checker can be used on makefiles http://clang-analyzer.llvm.org/scan-build.html
we need test binaries for this that produce a comparable output.
yes, we may need to build a test suite. With many buildslave (different gpu/driver/os) we could get warning/failure on visual glitch early.

if someone want to help me with the buildbot master
As said above, I can help adding botbuild slaves on a few computer here, but have to find how to do that ?
http://djmitche.github.com/buildbot/doc ... buildslave doc is not enough...
(which python version, buildbot version, parameters, scripts?)
User avatar
tdev
Silver Sponsor
Silver Sponsor
Posts: 244
Joined: Thu Apr 12, 2007 9:21 pm
Location: Germany
x 14

Re: continuous integration

Post by tdev »

@xavier, buildbotfarm is just a domain from me running a VM with some buildbots on it. The only custom thing is the config around buildbot so far.
tuan kuranes wrote: @tdev:
yes correct ones. (yes, splint is for C parts (ie Ogre command line tools?)
Clang checker can be used on makefiles http://clang-analyzer.llvm.org/scan-build.html
will check that later :)
tuan kuranes wrote:
we need test binaries for this that produce a comparable output.
yes, we may need to build a test suite. With many buildslave (different gpu/driver/os) we could get warning/failure on visual glitch early.
sounds like a good idea :)
tuan kuranes wrote:
if someone want to help me with the buildbot master
As said above, I can help adding botbuild slaves on a few computer here, but have to find how to do that ?
http://djmitche.github.com/buildbot/doc ... buildslave doc is not enough...
(which python version, buildbot version, parameters, scripts?)
sorry, i forgot to say that i am working on a tutorial for windows and linux:
http://www.ogre3d.org/wiki/index.php/Buildbot

i started with windows so far, as linux is really easy. Will continue with that later today.
User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 2653
Joined: Wed Sep 24, 2003 8:07 am
Location: Haute Garonne, France
x 4
Contact:

Re: continuous integration

Post by tuan kuranes »

Nice.
Started and everything installed fine, being stuck at :
setup cmake manually with cmake-gui and create the nmake build files
which cmake files ?
following wiki I just installed the dependencies ?
do I have to download ogre by myself ?
User avatar
xavier
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 9481
Joined: Fri Feb 18, 2005 2:03 am
Location: Dublin, CA, US
x 22

Re: continuous integration

Post by xavier »

tdev wrote:@xavier, buildbotfarm is just a domain from me running a VM with some buildbots on it. The only custom thing is the config around buildbot so far.
Right I understood that -- I was asking more about what buildbot itself is. I've since Googled it and tbh, not that impressed -- it seems to be extermely immature, basically just sort of tossed together...and configuration looks to be an absolute nightmare...as we're finding out, it seems. Furthermore, the sense I get from just the tone of the language on their site, is that the "product" is anything but user-oriented.

So, I'm seriously cooling on taking part if it's built around this software.
Do you need help? What have you tried?

Image

Angels can fly because they take themselves lightly.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Re: continuous integration

Post by sinbad »

I quite liked the waterfall view in buildbot, but if it's a pain to get set up we can try something else. I'd been thinking about CTest / CDash since we use CMake already, but CruiseControl always looks very nice.
User avatar
tdev
Silver Sponsor
Silver Sponsor
Posts: 244
Joined: Thu Apr 12, 2007 9:21 pm
Location: Germany
x 14

Re: continuous integration

Post by tdev »

xavier wrote:
tdev wrote:@xavier, buildbotfarm is just a domain from me running a VM with some buildbots on it. The only custom thing is the config around buildbot so far.
Right I understood that -- I was asking more about what buildbot itself is. I've since Googled it and tbh, not that impressed -- it seems to be extermely immature, basically just sort of tossed together...and configuration looks to be an absolute nightmare...as we're finding out, it seems. Furthermore, the sense I get from just the tone of the language on their site, is that the "product" is anything but user-oriented.

So, I'm seriously cooling on taking part if it's built around this software.
yes, that was my first impression as well, but once you start using, i saw that its a complete and handy utility. The configuration is in fact pure python code, since that allows you to couple things in a even more flexible way. There is currently no configuration interface for those tasks, but it wouldnt be used much anyhow: you configure it once and then let it run. The rest is done using the normal buildsystem. If you have any further doubts about buildbot, we could clear them up together if you like. About the professional aspect: Buildbot is just a set of scripts and tools, so you can create your own CI server. It is not comparable to the 'ready-shipped' CruiseControl.NET really as you might have seen ;)

sinbad wrote:I quite liked the waterfall view in buildbot, but if it's a pain to get set up we can try something else. I'd been thinking about CTest / CDash since we use CMake already, but CruiseControl always looks very nice.
i will improve buildbot and we can see how complex/simple/difficult to use it will be. I need to play around with the tool anyways.

progress slowly: Torchlight eats up my spare time currently xD
alexjc
Gnoblar
Posts: 21
Joined: Thu Jun 08, 2006 10:56 am

Re: continuous integration

Post by alexjc »

Tuan mentioned this thread, and since we have lots of experience with doing CC with automated tests for our AI Sandbox, I thought I'd share.
  • I had the same impression of buildbot, unprofessional hacked together, despite my positive Python biases it wasn't a good fit.
  • TeamCity is incredible, but well, it depends on your budget if you want to scale up.
  • CruiseControl could make sense, but we went for an integrated option rather than having a separate site.
  • Bitten (used with Trac) is what we chose in the end because it works seamlessly within the web UI we decided to use.
I had a bit of trouble setting up Bitten ~1 year ago when the SVN code was a bit out-of-date, but the project is actively maintained again and the Wiki instructions are enough to get it up and running. Also, I wrote some instructions for setting up Slaves and one of our contributors had no trouble setting up his own builder -- it caught me by surprise even seeing two build machines running the tests :-)

Bitten worked out nicely also because I can display custom reports after the build, and show nice graphs. Since I'm happy to code in Python, making such custom reports has been a useful process, and quite interesting.


I'm not sure I'd recommend Bitten if you don't use Trac, but then again I think it looks better than Cruise Control -- which is the only real other choice.

Alex
alexjc
Gnoblar
Posts: 21
Joined: Thu Jun 08, 2006 10:56 am

Re: continuous integration

Post by alexjc »

As a reference, here's a screenshot of our Trac/Bitten setup for building the AI Sandbox.
TracBitten_AISandbox.png
TracBitten_AISandbox.png (158.96 KiB) Viewed 5981 times
Alex
User avatar
tuan kuranes
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 2653
Joined: Wed Sep 24, 2003 8:07 am
Location: Haute Garonne, France
x 4
Contact:

Re: continuous integration

Post by tuan kuranes »

Thanks for sharing, Alex.

bitten + trac seems nicer to me that CC, because of the java/mono requirement, whereas python is flexible, and less intrusive (that helps getting "slave builders").

Trac could serve a "future design document repository" (still linked to/from forum), which help in work force use&repartition.

Would be an interesting move, helping having more direct contributors, using a more strict development system (like UltimateQualityDevelopmentSystem).

CI + Collaboration Tool + better DCVS would surely help giving Sinbad a higher level role (less code, more design and coordination, final patch/branch review before merge, etc.)
Shadow007
Gremlin
Posts: 185
Joined: Sat May 07, 2005 3:27 pm

Re: continuous integration

Post by Shadow007 »

I think (as I shared in the Samples GSOC) that the new Samples framework could be used to make at least a few automated visual tests...
I think a new SampleContext could be written that would render each sample and write a screenshot to a file. Then a diff with a reference image could easily be made, which would enable a automatic visual unit testing ...
For time independent samples, it would work easily, whereas for time dependant samples it would be more difficult.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Re: continuous integration

Post by sinbad »

Bitten looks good, thanks for raising it. I use Trac in my private projects already and also prefer Python system over those that require Java/Mono, simply because they tend to be more lightweight and simpler to host (because Python is so common and self-contained). Trac's wiki & bugtracking features aren't as good as dedicated systems for the 2 functions, but for private projects I find it does fine as an all-in-one.

What sort of overhead does Bitten add to the collating server? If it's not too much I might look at hosting it on ogre3d.org.
User avatar
SiENcE
Goblin
Posts: 231
Joined: Thu May 11, 2006 3:07 pm
Location: Berlin
Contact:

Re: continuous integration

Post by SiENcE »

@sinbad: Do we get Trac for Ogre repository :-) ?
alexjc
Gnoblar
Posts: 21
Joined: Thu Jun 08, 2006 10:56 am

Re: continuous integration

Post by alexjc »

Bitten doesn't have so much overhead compared to Trac (Bitten only does minor book keeping), and even Trac is reasonable. I run both on a server with many other systems (SVN, and our main site) and it's never been a problem.

As I said, installation was the only down side to Bitten, but that seems to have improved. You'll also have to set aside time to get comfortable with the system (including Trac), but now we're quite pleased with the results.

Alex
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Re: continuous integration

Post by sinbad »

SiENcE wrote:@sinbad: Do we get Trac for Ogre repository :-) ?
Probably not, or at least not forthe wiki & ticket tracking. Like I say, I use Trac regularly for my private projects and it's great as an all-in-one (very useful when you have lots of small projects each with separate trackers) but the wiki is not as good as MediaWiki, and the bug tracker isn't as good as Mantis, so for a single big public project like Ogre I wouldn't want to replace those with Trac. But I'm interested in Bitten just because it looks polished and doesn't require a hefty Java runtime like most others like CC.

Once I have some time I'll experiment with Bitten. I'll also have to see whether it will work with Mercurial since we're not planning on staying on Subversion forever.
User avatar
tdev
Silver Sponsor
Silver Sponsor
Posts: 244
Joined: Thu Apr 12, 2007 9:21 pm
Location: Germany
x 14

Re: continuous integration

Post by tdev »

*digging up old topics*

i would like to contribute a Hudson CI server to the ogre community :)
(like http://redmine.rigsofrods.com:8080/)
(if you want you get redmine as well)

example project: http://redmine.rigsofrods.com:8080/job/OISB/

who is interested in this and who is the responsible person to speak with?

cheers,
thomas
Post Reply