Notes from a Reluctant CM/CI/Release Manager

I'm in the midst of transitioning out of my role as a CM/CI/Release Manager. I've learned a lot while doing it (some things the easy way, some the hard way), and thought I'd blog my conclusions and lessons learned so I don't make the same mistakes then next time.

Grow; Gradually
There is immense value in even a small bit of SCM, build automation, or CI feedback. As long as your approach is flexible (and if your tools don't support this then seriously consider changing your tools).

What does this mean in practice?

• For SCM, choose a tool, (a free one preferably as this is quicker), get it installed, and get everyone checking in their code and other artefacts early. In my experience a distributed SCM (i.e.Git or Mercurial, or even Jazz SCM) is best as it's structure can easily be changed as the project changes over time.
• For build automation, choose anything (Ant, Maven, Rake, hell even shell scripts) but get it set up at the start and have all your bits use it. Also seperate the build from the environment right from the start. This makes it very flexible.
• For CI choose Hudson (IMHO it's the best engine at the moment by a country mile). Get it set up on a dedicated server right away (this will take 30 minutes) and get it polling your SCM, running builds, and going all red/yellow/green right from the outset.
  

Start with a Small, Simple, Core Build as Early as Possible
Continuing on from the last tip, you want to depend upon Hudson from the outset. Get the team aware of the benefit of the feedback it provides. Make it the team's friend, rather than their enemy.

To do this resist the urge to get clever to begin with. A single build will do at the begnning. You can grow it later. You'll read all about upstream and downstream builds and promotion and everything else. These are all great concepts, and you will use them, but nt yet. Implement them when the time investment warrants it. At the outset, your code base will be tiny, and you will have few Unit Tests. Don't worry about your build speed. It's not slow yet. Just get the code compiling, and the unit tests running.

Ensure the Whole Dev Team Understands how their SCM Tool Works
Your SCM is a powerful tool. If you understand it beyond the basic "checkin/checkout" dynamics it will allow you to have maximum confidence in the code you are working on both individually and as a team.

On our project we were soon taking advantage of the facilities for personal revision history, suspension and revision of change sets, creation of patches, reversals, and more. To get to this we created a training session which was run for each new joiner which took them slowly through the basics of the Jazz SCM "Change Set" system, and then build upon this so they all had a workable meta model of their changes in the context of their team, and the wider project.

Beyond this (and as a result of having to evolve the SCM model from the simple one we started with to the one we ended up with when everything went offshore) the CM/CI team had a deep understanding of how the entire system worked, which allowed us to do some really clever tricks, digging people out of messes when they did (infrequently) get into them, and making sure we could have just the right amount of seperation between bugfix, next release, and PoC / Investigation development.

Have a Sandpit
After the basic setup for both your SCM and your CI build is done you'll want to get cleverer and cleverer. This isn't a bad thing, but you will make mistakes. Especially if you want to get really clever.

We made the mistake of doing this experimentation on our "production" systems (i.e. the SCM and CI servers that the Dev Teams were using. Aside from the danger of catastrophic problems which you may cause, there is also the confusion you generate. CI is best when it is simple, clear and reliable. Red is bad. Green is good. You don't want to have to tell people to "ignore this for the minute while we get it working" or "don't worry about that, it'll work soon." This errodes trust in your tools, and gives others a source of excuses. As the CM/CI team, you need to lead by example. Hide your mistakes. Have a sandpit.

Make Sure the Architecture is "Developable"
What do I mean by "Developable"? It's my first real neologism and one I'm more and more confident of the more I work in Software Development. In reality it needs a whole post to itself (perhaps even a book), but in essence I mean that the chosen architecture and component technologies which it comprises should be easy to develop against. Perhaps a few examples are required.

• Compilable: Is it straightforward to compile your code? Do I just need the compiler, the dependencies, and the build scripts? No? I need the vendors too installed, and their server, and I need to deploy a dependency to compile downstream projects? Think again...
• Unit Testable: Is the code you write easy to unit test? Can you automate the running of these tests with your IDE and CI server and collect and view the results effectively? Is it easy to mock? No? I need to deploy in order to unit test? Perhaps you should re-consider...
• Tool Independent: Can you run your builds headlessly, without the need to install an IDE? No? Is this the 1980's? Come on...
  
• Version Controlable: Can we easily manage the code and configuration files in an SCM? You don't know SCM is? Let me explain...
  
• Manageable Dependencies: Can we automate the management of dependencies? You want me to check-in the compiled results? Are you serious? Alarm bells...
• Quick to Build: Can I poke my build scripts (on my machine, or via Hudson) and within a relatively sensible period of time get a red/green feedback? It takes 30 minutes just to compile, and its going to get longer as you add more code? Hmmmm
  
• Quick-Turnaround, Lightweight Dev Environment - Can I deploy my latest edits to my local server and see the results in a few minutes max? Can I develop on the same platform as production without having to take up hundereds of gig's of disk and have a 64-bit OS to address all the memory required to run all the servers and databases just to compile and run the unit tests? No? Have you heard of Tomcat?...
  

I'd recommend you think about (and investgate if needed) all these things and more when you get told what your architecture is and what you're going to build it in. Pushing back at the start is a lot easier than pushing back later on. Trust me...

Publicise the CI Build
Despite the fact the Hudson is a superb resource which all developers should embrace because it makes their lives easier, most of them hate it because it goes red when they screw up. In an ideal world they would love to know that the change they just made caused a problem, but the majority seem to want to forget about it until later, when the detail has been swapped out of their mental L1 cachee and it's far more of a challenge to resolve.

You can remedy this by making Hudson and the status of Jobs ubiquitous. Get Hudson to send emails; use the Hudson Tracked and Growl to pop up toast on their screens (and play a sound too); but best of all set up a build radiator. At worst, this should be a screen, widely visible to all (especially the Project Managers / clients) and displaying the colours of all key jobs. (All jobs preferably). At best, it should be a singing-dancing, attention grabbing machine which rewards good behavious and punishes the bad. Come on, get creative!

Automate the Release Process as Much as Possible - And Keep it as Simple as Possible
My final lesson learned is around making releases. You want to be able to make releases as quickly as possible, with as little manual intervention as you can get away with. I'm including the packaging, documentation, and deployment of these releases in this.

The aim is to make it easy to release super-frequently. Hudson isn't your last line of quality feedback. There are the testers too remember. The more you can give releases to them, and the quicker you can give them an update with all the fixes the more time they'll be able to sit about testing and increasing the quality of your product even more. It is all about feedback and quality after all...