The road to hell is paved with potentially shippable software. Why continuous delivery at Agile Cambridge - slides and report.

Posted on October 22, 2014


I gave a talk with a value stream mapping exercise at Agile Cambridge 2014 on October 3. You can download the slides of why continuous delivery now. Below is a description of how the session came to be, how it ran, and one or two things that may not be immediately obvious from the slides. Comments welcome, I plan to do some more iterations of this session to get some more clarity on my thinking. It seems the first two iterations also provided value to the audience, not unimportant when you talk about continuous delivery.

Why a talk on the why of continuous delivery?

Marc and I ran a course at a bank last month that lead us to think that we needed to work, once again, on a story about what continuous delivery means to us. We have some practical things we do that apparently are uncommon sense, and more importantly, we found that we need to explain more clearly why early and continuos delivery of valueable software is worth doing. We needed a new story.

Stories are just like software. Sometimes the best way to find a new story is just to go out, tell it, and iterate. So shortly a few weeks later I scheduled a non-existant presentation on the whys and hows of continuous delivery for a guest lecture at the university of Bath, on October 13.

The week before Agile Cambridge 2014 I was asked if I wanted to fill a spot left by a cancellation. Hmm, that was October 3, 10 days before the planned release. The temptation to deliver a talk about releasing early and often even earlier was irresistable. It probably also meant I would not be making slides in the weekend before the guest lecture - the Agile Cambridge session was on a wednesday, the lecture on a monday. I can prepare sessions in advance, but what I say in the talk bits for a new talk often only really comes together in the days and hours before the first run.

I had initially planned to talk about the why and how (in that order) of continuous delivery, but as Steve Smith talks also enjoy the Single Responsibility Principle. For a talk of about an hour, the maximum for me is about 30 slides, and my stack of index cards for this looked well over sixty, so I decided to drop the how part.

However, I wanted the participants to leave with something practical, and make the session interactive, in the space of an hour. So I decided to ask the participants in the beginning to collect the time it takes them to ship a one line change of code in to production (it probably gets worse if you start from a customer request). Steve suggested to collect separate numbers for how long it takes to ship a fix for a defect, and how long it takes to ship a feature. That turned out to be a good idea. Most teams in the roo shipped fixes an order of magnitude faster than new features. If it takes a week to ship a feature, it takes a fix a day, if a feature takes a month, it takes a week to fix a defect.

The ‘why’ reason for this question is that the kind of feedback you can get from users or customers when you can deploy in seconds or minutes is huge. Have a look at the slides for developing in production if you want. If you develop in production (e.g. program in PHP, press save and it is live) you can make modifications while you have an actual user on the phone. This feedback cycle can be very beneficial for finding out what your users actually want. Once you grow, it can also be destructive: fix one thing, break everything else. That is why the talk has a have our cake and eat it too theme.

A ‘how’ reasons I ask this question, is that I often see continuous delivery setups that feel very complicated and wonder if we can’t do that more simply. Especially when working on a startup, or starting in an existing situation with little automation, the gap is huge.

None of the teams in the room shipped in seconds, the shortest was deploying a fix in half an hour, but that was an outlier. Mostly it was at least days, for some it was a year or a year and a half for new features. I’ve been there, and I hate the words ‘potentially shippable’ with a passion.

If you don’t ship, you don’t know anything about whether what you do has value or not to real people.

The result of long periods of not shipping, even when you iterate daily, weekly or monthly with stakeholders, is that many useless discussions happen, and you feel that most of the things discussed are besides the point, but you just can’t put your finger on it, much less stop it.

There’s also a rant that you can’t see in the slides, that might deservie its own post. To me continuous delivery is about delivering value to the customer, early and often (sometimes by adding, sometimes by removing software often through iteration). Because of the continuous delivery book it seems to be more about complicated build pipelines that often automate outdated ways of delivering software than satisfying customers.

I fall into that trap on a regular basis. Maybe the partial value stream map exercise was one of those times.

Having said that, the partial value stream maps made by the participants were enlightening. We saw complicated branching strategies, mandatory reviews by often unavailable software architects (with the expected wait time for busy architects to do the review), pipelines with many, many steps.

If you want to improve on that, pick a few problems with your way of delivery, preferably seen from the end-users’ viewpoing and attempt a few improvements on a regular basis. Simplify and remove steps, and automate the steps that you are sure that add value.

Personally, I prefer to start from production. Automate deployments to production environments first, even though it is not the easiest thing to do. Production is where users get their hands on the software. But that is often the last step, as developers in large legacy organisations have often little say on how software is deployed.

When releases go wrong, as they often do with many hand-offs and manual steps, releasing more frequently and removing manual steps seems counter-intuitive. But it’s the production deployments that carry most risk, so there is also most value to be had there.

Manual steps and handoffs cause defects, defects cause fear, and fear creates pressure to release less often

If your production deployments go wrong often, there will be pressure to release less often, thereby increasing the risk that production deployments go wrong. Practice makes perfect, delays cause ignorance.

When you then automate all your development and testing environments, you are likely to find that your more frequent releases break due to human error when deploying to production. When you start from production, you might find that your development and testing environment is overcomplicated and you can save yourself a ton of work by not automating things that don’t add as much value as actually shipping and getting feedback from actual use.

Preparing the talk was a good chance to do some reading and re-reading. One of the things that got me thinking was Jez Humble’s post Continuous Delivery versus Continuous Deployment. I don’t quite know what to make of it. On the one hand, I like the focus on customers, but ‘potentially released’ give me the shivers. The road to hell is paved with potentially shippable software that has not actually been shipped. Demonstrating to customers is cool, but still implies ceremony. I know a few places that just ship and do well with it (your context may not support this way of delivering however). The emphasis on ‘you could deploy to production if you wanted to’ is admirable, however I find that having one manual step often leads to more manual steps and workarounds.

Rant over. See you next time.

comments powered by Disqus
More posts by Willem van den Ende RSS feed Atom feed