April Fools and the Broken Promises of One-off Hacks
Tags: meta, sre, devops
Published: 2020-04-01 14:59:45

It's April Fool's, a special day more detested than not. I still have vivid memories of eating an Oreo cookie stuffed with toothpaste instead of icing at the age of four. It's also the day that tech companies around the world send odd emails and make product modifications—often with the underpinnings of a growth hack which will drive attention to what they're working on.

Users mostly hate this stuff, but there's another group that can really loathe the process: The implementers. One-off hacks to pull off a funny April Fool's joke are often a brutal undertaking—usually not given enough time to really polish the project, rarely resulting in sturdy production code, and oftentimes mostly delivering a healthy dose of memorable and lasting pain.

I know that pain far too well from last year, where we checked off all of these boxes. As a founder of our tech startup, I try and shoulder more of the burden of stupid hacky projects like April Fool's because I don't want to force the disruption onto other folks in the team. Sometimes, however, the disruption just happens anyway. Last year we attempted to change DEV's default font to comic sans, which seemed like a fun way to mildly infuriate some people and also show off our functionality where users can [modify their fonts and themes in settings](/settings/ux).

![Hold onto your butts](https://media.giphy.com/media/InhhjtUJoOuA0/giphy.gif)

Things did not go well. In order to make this change work we needed to clear our edge cache—something we knew to be cautious with, but also an action where we thought we knew the consequences. Narrator voice: They did not. We expected a few failed requests before the servers caught up and the cache populated again. But the traffic jam was big enough that the site just never recovered. Even in other downtime scenarios for the server, we usually rely on the fact that most of the site is still browsable *because* of the edge cache being in place.

On April 1st 2019, the site just would not come back online. I worked through the night on some fixes. Eventually we fiddled with enough knobs and got some help from Fastly support to get things up and running again. The world spun on, and some downtime on a code community site is not a life or death scenario.

That is not to say that the scenario was not a valid existential threat. If we stayed down for too long, the concern is that the almighty Google algorithms start to see our site as unworthy of their rankings, and we never recover the traffic. We employed a half dozen folks at the time and we weren't exactly built to withstand a major injury to our growth. When it's the middle of the night and your site just won't come back online, you really think the worst.

![tired](https://media.giphy.com/media/HwmDZaI4YEeZ2/giphy.gif)

The site came back online, we didn't suffer any meaningful problems, and all in all our wonderful community of empathetic developers could commiserate like no other group we could possibly be serving. Half a year later we [secured significant funding](https://dev.to/devteam/special-announcement-from-the-dev-founders-hel), [we brought on CodeNewbie](https://dev.to/devteam/special-announcement-from-dev-4oi), we announced our first DEV-organized conference with CodeLand 2019 and then later adjusted our outlook to [host it online](https://dev.to/devteam/codeland-is-going-distributed-59pi) and the response from the community has been outstanding.

The world has also entered into a crisis that puts our problems into perspective. It also brings us immense gratitude that we can continue to employ a growing team, and offer an outlet for online community when we all need it the most. Our organization's mission is to empower community. I don't think we have gotten around to mentioning that publicly, but the word "Community" is literally in our corporate name and has been the goal from day one.

You might think that the pain of April Fool's past would mean we decided not to do anything this year. That's not true. We had an idea, but it was one that didn't touch our production code at all. We have an ongoing effort to _generalize_ our open source code such that anyone can stand up a community, and this project is coming along _amazingly well_. To that end, the plan was to spin up a fun community all about sharing pictures of our pets and other animals (Our mascot is Sloan the sloth, so there promised to be plenty of sloth pics).

However, when I went to put the finishing touches on the project—registering the domain and spinning up a TLS certificate through our CDN provider Fastly, I ran into a run-of-the-mill software problem where our issuance just seemed to get stuck in purgatory.

Last night I spent a while staring at this response...

```
 ▸    Status: created
 ▸    Your cert has not yet been issued. Please try again shortly.
```

It never changed. But rather than trying too hard to get to the bottom of this, I think we'll just move on. There will be no community for our fur babies launching today. I just don't have the energy to rush to the bottom of the project.

On the other hand, spinning up this new community instance was *soooo much fun*. Thanks to incredible work from our team and hundreds of open source contributors, our underlying software for empowering community is really coming along nicely. In making this community happen I decided this community really needs more emphasis on pics in the feed, and in just about five lines of code it was possible.

![Feed with dog pics](https://dev-to-uploads.s3.amazonaws.com/i/8eiu975erzqc11flgn8p.png)

These changes will go back into the project as configuration for any new communities. We have a lot of ideas about why offering transparency and agency through open source is really good for communities around the Internet, but above all else we feel pretty excited to just keep tackling the problem about what community can be. We're just not planning to rush out a demo today.

On another note, we do have an entirely separate fun community event kicking off tomorrow right here on DEV. Check back for that.

Happy coding. ❤️