Using Buildbot For Continuous Integration

I have been using buildbot for continuous integration for a number of projects, including my Foswiki psgi branch and it works really well. I think it would be nice to use it for the Foswiki project.

What is Buildbot?

Buildbot is a CI server written in Python. It comes in two parts, the buildmaster and the buildslave.

Buildmaster

The buildmaster is installed on a server, and is responsible to displaying a web front end, starting builds and defining how a build is performed. It does not perform any builds itself, so shouldn't use too much resource. It could be installed on the FW.org server and available at http://ci.foswiki.org.

Buildslave

The buildslave performs the actual build, when instructed to by the buildmaster. We can have many buildslaves on a number of different OS's, different versions of perl, etc.

It runs as a daemon process, connecting to the buildmaster over tcp.

Ideally, a number of Foswiki contributes could provide one or more buildslaves for use by the project. We could provide a script to install the buildbot, in order to make it easy and to keep the base environment the same (i.e. same version of buildbot, install Perl and modules with perlbrew, etc).

Why Use Buildbot?

Buildbot is very configurable, so we are able to tailor it to our project. For example, I have written a Foswiki-specific buildstep that runs the Foswiki suite and parses the output in order to find out how many tests we have ran, and how many fail. We could do a lot more.

It has the ability to send out emails with test reports (or just failures) to a mailing list, or the blamelist (if it can get the email address). Can also send a notification to IRC.

By having multiple slaves, we will be able to test Foswiki on a much larger array of OS's.

We should be able to control the buildslave environment, so we will only get real failures, not failures due to the environment.

Potential Issues

• We probably only want to run the trunk tests when the core (or core plugins) have changed. By using a SVN post-commit script to kick off the builds, we can check this.
• Need slaves

Example Buildbots

• Chromium
• Buildbot project
• Mine...

-- AndrewJones - 22 Dec 2011

Discussion

the big reasons I didn't use buildbot -

1. its not in perl - and my partner spends quite a bit of time customising it
2. the slave uses a daemon process
3. I like simple

dumb questions:

1. is there a similar tool in perl
2. how much customisation will we need to do to aggregate TAP results from mulitple platforms and multiple branches

and to me the biggie - I'd like users to be able to run a simple script that grabs a version of foswiki, and automagically runs the tests and then reports the results and platform specifics - without them needing to install a daemon, or something else

note that the last already does happen to some degree - its the reporting and aggregation taht is missing - a user that wants to test their platform's compliance can wget http://svn.foswiki.org/trunk/core/tools/autoBuildFoswiki.pl and run that - it will automaticalle get the source using svn, use pseudo-install to create a minimal cfg and run the FoswikiSuite tests.

What I dreamed of, is a TAP aggregation system that the unit test suite could adhoc push results too - or a user could email. That way we get integration testing, package building, and a way for difficult support issues to be tested by getting them to configure install UnitTestContrib into threir running foswiki and running a unit test report.

on the other hand - don't let me stop you I really really want a magical browser integration test rig too.

-- SvenDowideit - 22 Dec 2011

Unfortunately I don't know of anything similar in Perl. Maybe Tapper, but it looks a lot more complex...

I'm not sure what you mean by aggregation of results. Buildbot will give us a waterfall, which will tell us how many tests failed on a slave. The slave can be named win7-perl5.8.8 or similar, so you know the environment. Or do you mean using something like CPAN:Smolder (which works well with Buildbot).

With the "biggie" - I like that idea. I think it should be possible to customise Buildbot to allow that, and extend autoBuildFoswiki.pl to send the results back. I don't know a lot of Python, so might take some time... I think it would be nice to have both options (slaves and user running), because I'm not sure many users will actually run the tests on a regular basis.

-- AndrewJones - 23 Dec 2011

Commenting is disabled while not logged in