Improving the Engineering Process through Automation by Hudson - TS-6547

DISCLAIMER:
These are the tidied up notes I took from the session at the JavaOne 2008 conference in San Francisco, California. There may well be mistakes and ommissions. I will come back and correct them once the conference has completed. However my first priority is to get the information published before the backlog gets too large and swamps me. All comments welcome. Enjoy!

GOAL: Do more, with less

What is Hudson?
- Computers are getting cheaper but people are getting more expensive. We need to find ways to put computers to work for us
- Hence continuous Integration. Nightly builds? Dont be so cheap! We want software to do this
- Hudson is a java based CI server - looks for changes in VCS, runs tests and aggregates results, records all the data on one web site
- Hudson reduces communication overhead - how to build, how to test. Reduce the people dependency by removing tacit knowledge, especially for distributed teams
- Hudson is like another developer on your team (or your butler ... hence the name)
- Hudson is an OSS project on Java.net
- Hudson places particular emphasis on ease of installation and use
- Extensible architecture - 50+ community developed plugins
- Very active - 150 person years worth of code, FWIW, 3 years old, > 210 releases to date, 50+ committers
- Localised to 8 languages
- Helping all kinds of businesses - amazon, yahoo, NB, glassfish, mysql, linkedin, jboss
- Basic feature set:
- check out source code: cvs, SVN, clearcase, mercurial, etc.
- Do builds: ant, maven, shell script, Nant, ...
- record results
- close the feedback loop: RSS feeds, EMail/IM notification, Tray app to notify developers
- Tool integration: findbugs, emma, cobertura, VMWare

Demo
- hudson.dev.java.net
- download the hudson.war
- invoke java -jar hudson.war and you're running!
- or you can deploy it to your web container
- then go to localhost:8080
- Front page: see status of all jobs
- RESTful and intuitive URLS: e.g. invoke http://localhost:8080/job/your_project/build URL to start a build automatically, .../configure is the configuration page
- Test fails have "age" attribute - you can see how many builds ago the failure first occurred
- Works nicely with maven 2 projects - it can parse the POMs and figure out what you are trying to do and automatically do the reporting for you

Best Practices

• Dependency tracking - have build jobs and seperate test jobs. If test jobs take longer than builds, it is important to know what build is being tested by a job. Hudson can tell you using MD5 checksumming so you can tell what build is being tested
• Baseline - newbies do everything inside a single job (e.g. scm update, ant/maven, test, findbugs, done). => Compartmentalisation:
• full builds (before you split into pieces) (e.g. Metro)
• alongside this, have sub builds (e.g. JAXB and JAX-WS) running
• this gives faster feedback (the point of CI) and isolates your problems nicely so the rest of a team can forge ahead while the right person fixes the build
• Test / Build separation:
• tests often dominate the build time
• start test jobs once the build completes
• then you can run them in parallel
• what if there are many alternatives? e.g.: unit test with GF, unit test with tomcat, SQE test, test w/o container (3 OS * 2 JDK = 6 tests)
• Enable fingerprinting of the artifact bits: so that hudson knows which test run is testing which build
• Copy bits from the build to the test dir with wget
• "Does this build work?" - just becasue the build worked does not mean it was "good"
• QA takes time but CI should be fast
• Old way - set up "nighlies" but this is disconnected from CI builds (diff build #s etc.) and duplicates the build setup
• New Way - Build promotion
• Allied army at Omaha Bean model - throw everything at it and see how far each of the individual parts get
• Produce the builds as quickly as you can. If a build passes tests, promote it. If it fails, it is "killed"Once promoted all promoted builds are listed on the website
• then commence more expensive QA, deploy to staging server, integrate, push to maven repository, etc.
• Distributed Builds:
• Master (servers HTTP requests, stores all critical info like config) and slaves (83K single jar deployment, co ordinates local processes, can come and go easliy)
• If there are more jobs than hudson can execute in parallel, hook up more slaves
• slaves launch via java web start
• MultiConfigJobs allow Hudson to take advantages of slaves to build s/w against different environments (e.g. Operating Systems, databases, etc.)
• you set up your multiple configurations build matrix in the build config page - you just tick the build parameters you want.
• the results are reported in a single page as a traffic-light matrix, against all the different environments
• Navigation - use the search panel in hudson or in your browser search bar
  

Extensibility

• remoting API
• RESTful API to retreive / push data as XMl or JSON
• Eclipse and NB plugins are available build on this API
• Integrate with issue trackers - link to Jira and have Hudson update the Jira issues
• maven plugin for - initial skeleton creation, handle compile time annotaton processing, debug support, package, release