I work at a newspaper as a programmer for the website. Mostly my job is backend programming, some HTML and CSS work (mostly left to designers). I run our local computer infrastructure as well as manage a cluster for our online presence and assist in technology related journalism as well as assisting our CEO in managing the IT budget.

I inherited a mess of an architecture and am finally getting around to rewriting our deployment process. We buy VM services from a local outfit and the prices are basically an arm and a leg for rather small machines. Due to this my predecessor put in place an insane deployment script. It pulls the new version from github then reloads code on the running dynos, one after another. Reverting is out of the question with our current approach to VCS (something I am also fixing). Most of the time this is no problem, all we are changing really is some template code, or introducing new models and their views.

Thinking back I am quite happy we don't run into more problems than we do, but also happy that this type of insanity is soon in the rearview mirror.

The worst mistake was recently, cost us about 4 hours of downtime during the busiest time of the day.

A big feature on all news sites are lists of stories to present to the user to look at after they have read what you put in front of them at the moment. They may take the form of most viral, most read, most commented, sliced by time or category or many other factors. My predecessor had written all those lists statically, which made maintenance a nightmare and extension very fragile.

I made a function that was a generic list of items. You supply basic parameters, amongst them a QuerySet for what would construct the list and my function would check to see if it was cached and if it wasn't, generate it and cache it.

The framework I use (Django) generally uses lazy evaluation for all QuerySets and I rarely have to think about the size of the list I generate, I just take care to limit the query before I list() it. During development nothing showed up as a problem and I deployed this and all seemed to be good with the world.

A week passes by where I made at least 2 minor deploys (small changes to templates, minor tweaks to list filters) and all seemed to be good with the world.

Designer sends me a pull request, I look over the code, just some garden-variety template changes, nothing that should raise an eyebrow. Make the merge, plan to deploy and then go to lunch. Deployment done, all seems well for 2 minutes but then suddenly servers lit on fire. Pages spewed 404's and 500's like there was no tomorrow.

For 4 hours I tear my hair out, examine every piece of code I was deploying that day, call in the big gun support (the kind that costs more money than I care to think about). Everything I was looking at pointed to the caching agent not working. Too many pageviews requesting the database, too much load on the servers, reboots made them work fine for about a minute but then everything became bogged down.

The big gun support pointed something out finally that I had missed: Traffic from the database to the dynos was abnormally high. Made me take a look at code that had been there for a while and lo and behold: For some reason when you pass a QuerySet as a parameter, it seems to be evaluated for the receiving function! 2 lines of code added, one deploy, problem fixed.

I have no idea to this day how this code could be live for a week without causing problems but an unrelated change triggers the bad behavior. This is not be the first time I've seen strange behavior from code, having seen a Heisenbug in Java code.

There's a happy ending to this. I made a big mea culpa slideshow where I pointed out all the flaws and what we needed to do to prevent a re-occurence. I got support to make the changes needed and my new cluster goes live day after tomorrow. Now I can carefully change NEW dynos for a deployment, keeping the old one's around if the shit hits the fan. I got some changes instituted in how we approach VC, something that's hampered work for a while. And we save money in the long run because we will no longer be paying an arm and a leg for the VMs (AND I got to learn about clustering machines with HA, goodstuff with gravy).