The human toll of log4j maintenance
Tags: java, log4j, opensource
Published: 2021-12-14 21:48:25

---
title: The human toll of log4j maintenance
published: true
description: How the pressure of fixing the vulnerability was dumped on a tiny team over a frantic weekend
tags: java,log4j,oss
//cover_image: https://direct_url_to_image.jpg
---
BY NOW, most of the internet knows about the famous Log4Shell exploit, and if you don't, it's easy to [get a sense](https://www.google.com/search?q=log4shell&oq=log4shell&aqs=chrome..69i57j0i3l4j0i3i10j0i3l2j0i3i10l2.1502j0j7&sourceid=chrome&ie=UTF-8) of how disastrous it's been. To drive the point home: the US Department of Homeland Security is [warning](https://www.cnn.com/2021/12/11/politics/dhs-log4j-software-flaw-warning/index.html) people about it.

There's been a lot of hand-wringing about how open source software, the lifeblood of many businesses today, is often totally unpaid and unthanked work, with some hot takes like ['Open source needs to grow the hell up.'](https://twitter.com/FiloSottile/status/1469441477642178561) and ['Open source' is broken](https://christine.website/blog/open-source-broken-2021-12-11).

What I want to touch on is something that's been bothering me for the past few days, and solidified after seeing [Bloomberg's piece](https://www.bloomberg.com/news/articles/2021-12-13/how-apache-raced-to-fix-a-potentially-disastrous-software-flaw?sref=ylv224K8)–the fact that the log4j developers had this massive security issue dumped in their laps, with the expectation that they were supposed to fix it. How did that happen? How did a group of smart, hard-working people get roped into a thankless, high-pressure situation with absolutely no upside for themselves?

Rough idea of the timeline:

> At 2:51 p.m. on Nov. 24, members of an open-source software project received an alarming email.... “I want to report a security bug,” wrote Chen Zhaojun, an employee on Alibaba Group Holding Ltd.’s cloud-security team, adding “the vulnerability has a major impact.”

This triggered a frantic rush in the log4j team to fix the issue:

> He described the conversations among the Log4j group as dispassionate and earnest. “I know these people -- they all have families and things they have to do. But they put everything aside and just sat down for the whole weekend and worked on that,” he said.

They worked liked a professional team of security-focused engineers, except with zero pay or recognition. They gave up a weekend with their families to do this for almost the entire internet. They were criticized and harassed, and their every action scrutinized: {% twitter 1469349956880408583 %}

And after all that, they delivered a fix that people have been swiftly upgrading to and breathing a sigh of relief.

Now again I ask, how did these unpaid open source maintainers get roped into this high-pressure, exploitative situation? How is it that the entire internet treated them like their personal paid security engineering department, only with no pay and no thanks? What went wrong that they got from the explicit [license terms](https://github.com/apache/logging-log4j2/blob/master/LICENSE.txt):

> `7`. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.

...to slaving away to fix it for everyone like their very lives depended on it?

## The great open source brainwash

Let's look at the Open Source Institute's [Open Source Definition](https://opensource.org/osd):

> Open source doesn't just mean access to the source code. The distribution terms of open-source software must comply with the following criteria:...(bunch of criteria that the license must satisfy to be considered open source)

Let's read Rich Hickey's (creator of Clojure) [small note](https://gist.github.com/richhickey/1563cddea1002958f96e7ba9519972d9) about being an open source maintainers after people in the Clojure community wrote posts like 'Fuck Clojure':

> Open source is a licensing and delivery mechanism, period. It means you get the source for software and the right to use and modify it. All social impositions associated with it, including the idea of 'community-driven-development' are part of a recently-invented mythology with little basis in how things actually work, a mythology that embodies, cult-like, both a lack of support for diversity in the ways things can work and a pervasive sense of communal entitlement.

It is this communal mythology I want to talk about, this great open source brainwashing that makes maintainers feel like they need to go above and beyond publishing source code under an open source license–that they need to manage and grow a community, accept contributions, fix issues, follow vulnerability disclosure best practices, and many other things.

The Alibaba software engineer who reported the issue to the log4j team subsequently (on Dec. 8th) followed up with:

> “Some WeChat security chat groups are already discussing the details of the vulnerability, and some security researchers already have the vulnerability,” Chen wrote. “We promise to keep it secret until your official release version comes out. Please hurry up.”

At this point everyone is fully bought in to the idea that the log4j team _must_ urgently fix this because everyone is relying on them. There's not even a question about doing otherwise. Why? Again–because we have this pervasive open source mythology that open source is about open community, governance, security, and all those nice-sounding ideas. I've had discussions where people have told me with a straight face:

> Well I'm pretty sure there is an entire industry of open source developers that would stand behind me in saying that open source is about more than a license, but feel free to stick your fingers in your ears and ignore all of us if that makes you happy.

They are _incredibly_ bought into this mythology, and you can't argue against it. It's like a religion.

## The wake-up call

In reality what is happening, is that open source maintainers are effectively unpaid outsourcing teams for giant corporations. The Alibaba engineer told the log4j team: 'Please hurry up'. Meanwhile, let's remember that Alibaba has a [market cap](https://finance.yahoo.com/quote/BABA/) of $348 billion (that's USD).

So what's the answer here? Seems like a lot of people are saying that corporations should fund open source. Others are pointing out that it's not that simple, because apparently corporations don't want to do that (for some reason). Yet they are totally fine with 'embracing open source' and continuing to use the software and pressure the maintainers. There must be a middle ground.

## The suggestion

Here's my suggestion–let's imagine what happens if the scenario plays out like this: the Alibaba engineer sends the warning to the log4j team, and then _nothing happens._ The log4j team does _nothing_ about it, because they have lives and families, because they're busy, they're not feeling productive, whatever. Alibaba follows up and requests the fix _urgently._ The team then sends a quote for the fix (to be made open source, of course), for $50,000. That's peanuts to Alibaba. Their market cap alone is nearly _7 million times_ that quote. They stand to lose untold amounts of money with a vulnerability like Log4Shell.

I invite you to think on how this would play out. Let's break the brainwashing and reveal the mythology as a hoax and false idol. Open source does not equate free vulnerability fixes, security best practices, open community, or any of the other nice-sounding layers that people add on top of it. Maintainers' lives don't depend on doing free labour for megacorps. It's time corporations (and everybody else) took a step back, dug their claws out of the backs of maintainers, and accepted the risks and responsibilities (and yes, expenses) that come with using open source.