Contents
Build Brigade
The Build Brigade is an effort to automate discovery and reporting of Gnome build errors and |
The Gnome Buildbot
This is a continuous integration system based on Buildbot adapted to build all Gnome modules through jhbuild:
Our Goals
Roughly in order of priority:
Find Gnome build errors quickly: when a commit breaks the build, we should be able to find out that it happened and what caused it.
Find regressions quickly: use unit tests (e.g. check), functional tests (e.g. dogtail) and integration tests (e.g. dogtail) to find out when functionality breaks.
Make life easier for testers: build RPM and Debian packages from Git so that testers can install and use them. Virtual machine images are easy to use and growing popular.
Profile performance: discover when code runs slower than before
Generate coverage information: find out gaps in our unit tests.
Upload generated API documentation.
Detailed requirements obtained mainly from Luis Villa's experience with MicroTinder can be found here:
Existing efforts
Finding build errors
Frederic Peters: JHAutoBuild -- http://jhbuild.bxlug.be/
Igalia: Gnome Buildbot
Easy testing: packages and virtual machine images
- Virtual machine images (VMware etc.)
Foresight Linux GNOME Live Demo images for various virtual machines (VMware, Microsoft VHD, Parallels/QEMU) and a Live CD image
- Software packages (RPM/DEB)
openSUSE Build Service: It can build packages for various distributions. Most of the advanced features are still missing, but the basic build functionality is there.
JhDebuild: Builds Debian packages from svn, sources are available on http://www.0d.be/bzr/jhdebuild/ (it is not yet run automatically)
- Virtual machine images (VMware etc.)
Easy testing: testing framework
Accessibility team is working in enhance a testing framework, and show interest in collaborate with the buildbrigade
Social aspects
- Building trust with maintainers
- We might get powers to revert patches to fix the build
- Use HACKING file as policy for what to do in case of breakage
- Educate maintainers in improving test coverage of their modules
Todo
This is a list of tasks that need to be done. If you think you can help with any of them, please, let us know, your help would be very welcomed ;-). After each task there is a set of attributes between parenthesis that state for (priority, difficulty, technology, asigneee). If you want to collaborate in a task which is already assigned, you can contact the people the task is assigned to. Otherwise, if you want to help in an unassigned task, you can talk to us using the mailing list and the IRC channel, or you can contact me (IagoToral) directly.
- Build error notification using a mailing-list, similar to the gnome-commits one (high)
Improve Check based unit tests integration into buildbot view. (medium, easy, check-builbot-python-html, IagoToral)
- Improve test results visualization.
- Get counts of passed and failed tests.
- dbus needs to be configured with --enable-tests --enable-asserts for the 'make check' results to be meaningful, but any production build should not have these flags
- Improve current buildbot view (low)
- Right now is synchronized with library.gnome.org view, but a review would be good to check if we missed something
- It would be good to also expose additional information, ie:
- Right now it shows the moduleset being used, it would be good to also write down the metasets (So: gnome-apps-3.0: "meta-gnome-core" "meta-gnome-apps-features")
Long term ideas
- Add a way to test patches.
- Integrate with Bugzilla for automated testing of patches.
- Build Gnome with -Wall -Werror
- Notify i18n teams of new strings. Maybe even having a string-freeze check that gets activated in freeze period.
ClaudeParoz: That's already implemented with Damned Lies. Each string-freeze break is reported to gnome-i18n ML.
- Add a Makefile target 'make bugreport' to as many modules as possible, which knows where to send the bugreport (because of the variety of bug tracking systems in use). Making sure every HACKING file has the right information would be a good start.
- Integration with SuSE Build Service
Done
John Carr improved the current jhbuild-buildbot integration. Now the buildbot is part of jhbuild, and his installation is really straightforward, see /DocsAndGuides
Improve current buildbot view GB#485125
created build-brigade-list mailing list -- The new list (note the name) list is working, please subscribe!
gcov and gcc4 work together it seems - see GStreamer (http://gstreamer.freedesktop.org/data/coverage/lcov/gstreamer/)
JoseDapenaPaz: Note about gcov in gcc4: you have to link -lgcov (add it to the LDFLAGS).
StefanKost: It needs a lcov from cvs + Makefile changes (see https://sourceforge.net/tracker/?func=detail&atid=103382&aid=1092155&group_id=3382)
- Patches for JHBuild to implement:
A DBus frontend, nice to communicate with continuous integration tools (https://bugzilla.gnome.org/show_bug.cgi?id=348487).
Configuration properties to avoid make check poisoning (https://bugzilla.gnome.org/show_bug.cgi?id=310544).
Add other checkout modes (clobber, copy, export) to jhbuild (https://bugzilla.gnome.org/show_bug.cgi?id=347114).
A hack to let user compile specific stages of a module. Shouldn't be required if you use the dbus frontend to get stages information (https://bugzilla.gnome.org/show_bug.cgi?id=344888).
A "pathrun" command, to let the user run a command in the build directory of a module (https://bugzilla.gnome.org/show_bug.cgi?id=356499).
Implemented a set of patches and scripts to integrate buildbot and jhbuild (http://community.igalia.com/twiki/bin/view/Main/JhbuildBuildbot ).
Improve RSS support in the gnome-buildbot (Iago announce)
Build Tools
Tinderbox 3
- Pro: widely used (e.g. Mozilla)
- Con: not actively maintained
JHBuild
JHbuild builds Gnome from git (before: svn/cvs) or tarballs
- Pro: can already build Gnome
- Pro: can work with multiple revision control systems
- Pro: can be used in conjunction with other systems e.g. have JHBuild do the checking out and Tinderbox/BuildBot do the build
BuildBot
- Pro: actively maintained
- Pro: flexible
- Pro: supports having dependency information
Garnome
Garnome builds Gnome from latest tarballs (both stable and unstable branch)
- Pro: actively maintained
- Pro: handles dependencies
- Pro: reliably builds Gnome
- Con: doesn't support building from SVN/Git
Requirements
- Using existing JHBuild module set information -- duplicating module information is bad
Niceties
- Don't rebuild modules that haven't changed
- Use ccache
- Support running "make check" or other test infrastructure
Questions
Should we build everything from scratch each time, or reuse existing builds?
ChristianKirbach: This cannot easily be answered. Updating the sources is faster than checking out everything each time. Keeping the old object modules, i.e. not running 'make clean', usually saves a lot of building time. However, cvs sometimes cannot update properly and gives up because of conflicts. I have sometimes experienced that after successfully updating the sources autogen must be run again for a successful build. I have also rarely seen that make clean actually was required. So, for fast building we probably want some fallback system like
- Update sources and build. (If updating fails jump to d?)
If build fails run autogen and build.
If build fails run make clean and build.
If build fails purge the sources, check them out, run autogen and build.
- If build finally fails the module is definitely broken. Reasons can be a broken module or e.g. an API change in one of its dependencies.
I assume steps b and c recover the build in 90% of all failing cases.
JoseDapenaPaz: an easy solution is combining "compiling caches" (like ccache) and a "copy" update model (see my post about implementing checkout modes in jhbuild).
When a build fails, should we use the previous build and continue, or stop?
JoseDapenaPaz: I would definitely use the previous build. We cannot taint all the modules because of every single change. Anyway, it would be interesting that Continuous Integration could warn that an error compiling a dependency has happened.
Interesting links and documentation
Meetings
- November 2006:
Background
These ideas have grown out of a presentation (CI BoF 2006) and a subsequent discussion at GUADEC 2006.
Contact us
You can contact us through:
Mailing list: http://mail.gnome.org/mailman/listinfo/build-brigade-list
- IRC: #build-brigade on irc.gnome.org