Last winter John Allspaw joined Etsy to run tech ops. John has written several important books on web ops and is one of the experts in operating large scale web services. One of the first things John did when he arrived at Etsy is work with the dev and ops teams to put in place a continuous deployment system.
Continuous deployment is the idea that you push out changes to your code base all the time instead of doing large builds and pushing out big chunks of code. Here's an Eric Ries post on continuous deployment if you want to get a longer description of what it is and how it works.
At Etsy, they push out code about 25 times per day. It has worked out very well for Etsy and has led to faster cycles, improved morale, and a more stable and reliable web service. We were talking about all of this during the board meeting yesterday and Chad Dickerson, Etsy's CTO, invited me to push out some code after the board meeting.
I sat at Chad's computer, we pulled up the queue of code that had been reviewed and approved and was ready to push, I selected the code I was going to push out, I hit the "deploy" button, and then we watched as the code was pushed out into the production system. Then we watched all the key metrics to make sure I hadn't broken the service with my push. I pushed some small changes to the checkout system. We watched to see that checkouts continued to get processed. We looked at all the charts and graphs. Everything seemed fine. And we were done.
After we finished the deploy, I asked John (who took this picture) what would have happened if I did break the site. I asked how to roll back the changes. He said "we don't roll back, we fix the code." I asked what was the longest time it took to "fix the code" after a push had broken something. Kellan (who is on the right in the picture) told me that the longest fix for something that had taken a critical service down was about 4 minutes although they had seen longer periods of diminution of service.
I really like this model. Big changes create big problems. Little changes create little problems. I realize that this model doesn't work in desktop software and mobile apps. But cloud based web services can be operated this way and it really jives with whole culture of building and managing web apps.
The next time I am at Etsy, they are going to let me make a small change to the code base and then push that out. As Pete Miron said on Twitter yesterday, "so simple, a VC can do it."
