Daily Builds Applied To Systems Integration Projects.

The last post has got me thinking more about the whole concept of daily builds.  I mentioned in passing that the concept is not just applicable to software development but I did not explain the comment.  I went out for a walk and started to think through how the concept could be applied to a systems integration project.  The one I chose is quite topical for me at the moment, a Windows XP desktop refresh and desktop management project.

So first let’s look at some characteristics of this sort of project:

1. A standard system image that needs to be deployed tens of thousands of times to many different types of hardware
2. The need to deploy thousands of applications on top of this standard system image, and to deploy these applications hundreds or thousands of times
3. The need to access seamlessly thousands of file, print, authentication, management and application servers
4. An environment that tens of thousands of users will use for perhaps 2-3 hours a day on average, this means hundreds of millions of pounds of deliverables depend on its usability and reliability

So let’s look at the daily, (or perhaps regular), build process and ensuring that the project team “eat their own dog food”, and how these techniques help with a project of this type:

1. It dramatically reduces risk, as deliverables evolve incrementally and at each build cycle the progress is the subject of widespread review.  Misunderstandings, omissions and errors rapidly come to light.
2. The collective intellect of the project team can be directed at improving the solution
3. The project team are forced to think, “automation, reliability, configuration control, reproducibility and quality”, on a daily basis
4. Rapid and efficient build and deployment processes are tested and refined from the beginning
5. Rapid application provisioning and deployment processes are tested and refined from the beginning, application packages are deployed many times to many different build standards and different hardware types long before they are delivered to real users
6. Architects get to verify daily that the conceptual and logical architecture are actually being implemented as envisioned, and that changes/optimisations that occur in the physical implementation get reflected back into the architecture
7. The whole provisioning, deployment, usage and management infrastructure gets integrated and exercised in an evolutionary way, greatly reducing the risk that would otherwise occur when different teams wait until integration testing in the lab.  Prior to the lab the only integration occurred through document review.
8. Every time the environment is rebuilt, (for physical desktops not every device is rebuilt every day of course, but it’s entirely practical to do this with Virtual Desktops), or a device within it is rebuilt the quality of upgrade, and migration processes gets tested.
9. Every time a device is rebuilt the success of separating system state, from user state gets tested.  Users get very unhappy when they keep loosing user state information.
10. User experience, i.e. how the whole environment fits together from the users perspective gets tested,  otherwise user experience only gets tested in pilots, which is normally too late to really make significant improvements.  This is a key issue, most projects are structured around creating bits of a solution, architects are responsible for making sure the bits of the solution work together but its rare to find sufficient focus on making sure that the bits not only work but integrate into an environment that’s effective, usable and discoverable

If you wanted to run a project this way what would you need:

1. You would need to create a “dog food” environment that the project team use.  You might not start with the whole of the project team just the core developers and architects first, then the project managers, then the programme managers etc.
2. You would also need system and integration test labs, and probably a customer reference lab, because not all test scenarios can be exercised in a dog food lab with real users with work to do.
3. You would need to implement the whole solution in this “dog food” lab, as close as possible to the user’s environment.  At first the implementation might be un-configured, i.e. products as they are installed out of the box.  Some elements may not exist at all place holders would be implemented instead just to show that a bit is missing.  For example if “end user application provisioning” is not there yet, make sure the link is on the desktop, and that a web page opens saying this bit is not done yet.
4. Make sure you tell people every day what has changed.  A set of RSS feeds is ideal for this.
5. Make it really easy for people to report problems, comments, suggestions and make sure these get logged and that the project team classify and respond.  A discussion database is ideal for this. 
6. Appoint someone in the lab team who is motivated to encourage and help the testers to identify and describe bugs, issues and comments.  Don’t worry too much about making the testers log bugs etc. in a particular way, getting input should be the priority regardless of whether it’s via the phone, one to one, logged on a web site etc.  That’s why you need someone dedicated to helping the users to log the problems.
7. Make this person also take responsibility for chasing the developers to make sure they respond.  Testers are much more motivated when they get responses.
8. Make sure that the management organisation get involved in operation and management of the environment.  It’s a great opportunity for them to learn and to test out the “management experience”, in the same way as I described the user experience above.
9. Once things start settling down consider switching to testers logging issues through the normal help desk channels.  In the early stages this would stifle the process, but in the late stages when the number of events being logged is much reduced its worth it to help test that process.
10. Make sure the lab team responsible for the “dog food lab”, think of themselves as customers of the development team.  Make sure they drag capabilities out of development, make sure they are demanding and set down standards of quality and reliability and automation, make sure they gather input from testers and chase up developers for responses.  Make sure that lab team don’t get dragged too far into development otherwise they will stop being effective customers and become too understanding of the developers challenges.

The picture is of the big wheel next to the river at York, it’s relevant because this is where Steve Townend now lives.  Steve worked with me to implement daily builds for an application we were developing and after a couple of week figuring out how to make the daily build process work it was awesome.  The application we developed progressed incredibly well using this process and every day we had a working, but not complete, system.