Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability
Tags: azuredevopsserver, log4j, log4shell, cve202144228
Published: 2021-12-14 18:31:38

---
title: Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability
published: true
date: 2021-12-12 23:58:44 UTC
tags: AzureDevOpsServer,log4j,log4shell,cve202144228
canonical_url: https://jessehouwing.net/azure-devops-patch-for-log4j-vulnerability/
---

 ![Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability](https://jessehouwing.net/content/images/2021/12/pexels-photo-3250154.jpeg)

> Update: **[Official statement from Microsoft.](https://devblogs.microsoft.com/devops/azure-devops-and-azure-devops-server-and-the-log4j-vulnerability/?WT.mc_id=DOP-MVP-5001511)**

Azure DevOps Server ships with a pretty old version of Elastic Search and by default installs a non-auto updating JVM. I just checked the version of Elastic Search that ships with my version of Azure DevOps Server and it's 6.2.4. That version ships with log4j 2.9.1, which is unfortunately too old [for the simple fix described by Elastic to do the trick](https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476):

> Elasticsearch 6 and 7 are not susceptible to remote code execution with this vulnerability due to our use of the Java Security Manager. Elasticsearch running on **JDK8 or below is susceptible to an information leak via DNS** which is fixable by the JVM option identified below. This option is effective for Elasticsearch versions 5.6.11+, 6.4+, and 7.0+.  
>   
> **Elasticsearch 5 is susceptible to both remote code execution and an information leak via DNS.** For versions 5.6.11 - 5.6.16, this can be mitigated by setting the JVM option. Users on an earlier version of 5.x, are recommended to upgrade to 5.6.16. We are exploring other options for cases where upgrading is not possible. Please note that while we provide these remediations, Elasticsearch 5 is not a supported version, and we always recommend updating to the latest release.  
>   
> [https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476](https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476)

- For TFS 2017 RTM, Elastic Search 1.7.1, which ships with **log4j 1.2.17**, which is **not vulnerable**.
- For TFS 2017 Update 1 - TFS 2018 Update 1, Elastic Search 2.4.1, which ships with **log4j 1.2.17**, which is **not vulnerable**.
- For TFS 2018 Update 2+, Elastic Search 5.4.1, which ships with **log4j 2.8.2**, which is **vulnerable** and can not be resolved by setting the JVM option.
- For Azure DevOps Server 2019, Elastic Search 6.2.4, which ships with **log4j 2.9.1**, which is **vulnerable** and can not be resolved by setting the JVM option.
- For Azure DevOps Server 2020, Elastic Search 6.2.4, which ships with **log4j 2.9.1**, which is **vulnerable** and can not be resolved by setting the JVM option.

Luckily the default installation mode only binds Elastic Search to the local loopback network interface limiting access from the outside world, but if you've setup Search on a separate VM and didn't follow the [Securing Search guidance](https://docs.microsoft.com/en-us/azure/devops/project/search/install-configure-search?WT.mc_id=DOP-MVP-5001511&view=azure-devops#secure-search) to limit access you may want to look into that right now.  

You currently have 3 options to deal with this issue:

1. Remove Search from your installation
2. Patch the log4j-core-\*.jar files on disk
3. Patch the log4j vulnerability at runtime

I also highly recommend you update the JVM that was installed when you configured Search. The default option does not auto-update and chances are the one you're running is already quite old.

## Remove Search

[The simplest solution, and most secure solution at the moment, is to uninstall Azure DevOps Search from your Application Tiers](https://docs.microsoft.com/en-us/azure/devops/project/search/install-configure-search?view=azure-devops&WT.mc_id=DOP-MVP-5001511#uninstall-search) and disabling/uninstalling the Elastic Search service:

![Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability](https://jessehouwing.net/content/images/2021/12/image-11.png)
_Disable Elastic Search_

Or remove the service altogether:

![Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability](https://jessehouwing.net/content/images/2021/12/image-13.png)
_Remove the Elastic Search Service_

And remove the Search feature from the Azure DevOps Administration Console:

![Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability](https://jessehouwing.net/content/images/2021/12/image-15.png)
_Remove Azure DevOps Search Service from the Azure DevOps Server Admin Console_

## Patch Elastic Search files Instead

If you don't want to turn Search off, you'll need to patch `log4j-core` to [strip the vulnerable classes from the jar files](https://research.nccgroup.com/2021/12/12/log4j-jndi-be-gone-a-simple-mitigation-for-cve-2021-44228/). You can remove the vulnerable classes from the jar file with a single command. All of the docs I found give you this command which will work on Linux:

```
zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
```

On Windows that command won't work, since the `zip` command isn't available. If you've got 7-zip installed, use the following the command for Azure DevOps Server 2020:

```
cd "C:\Program Files\Azure DevOps Server 2020\Search\ES\elasticsearchv6.2\lib"
..\bin\elasticsearch-service stop
"c:\Program Files\7-Zip\7z.exe" d log4j-core-2.9.1.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
..\bin\elasticsearch-service start
```

To resolve the issue for Azure DevOps Server 2019+:

```
cd "C:\Program Files\Azure DevOps Server 2019\Search\ES\elasticsearchv6.2\lib"
..\bin\elasticsearch-service stop
"c:\Program Files\7-Zip\7z.exe" d log4j-core-2.9.1.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
..\bin\elasticsearch-service start
```

To resolve the issue for Team Foundation Server 2018 Update 2+:

```
# cd "C:\Program Files\Microsoft Team Foundation Server 15.0\Search\ES\elasticsearch-5.4.1\lib"
..\bin\elasticsearch-service stop
"c:\Program Files\7-Zip\7z.exe" d log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
..\bin\elasticsearch-service start
```

## Alternatively, patch log4j at runtime

You can also [install `log4j-jndi-be-gone`](https://github.com/nccgroup/log4j-jndi-be-gone) and instruct Elastic Search to load it to remove the vulnerability at runtime. **This will also work for older versions of Team Foundation Server and Azure DevOps Server.**

Download the [`log4j-jndi-be-gone-1.0.0-standalone.jar`](https://github.com/nccgroup/log4j-jndi-be-gone/releases/download/v1.0.0/log4j-jndi-be-gone-1.0.0-standalone.jar) from their [GitHub repository](https://github.com/nccgroup/log4j-jndi-be-gone/releases) and save it to your server.

![Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability](https://jessehouwing.net/content/images/2021/12/image-16.png)
_Save the jar file where Elastic Search can find it_

Run `elasticsearce-service.bat manager` to open the configuration panel for Elastic Search and add the jar file to the configuration:

```
-javaagent:C:\AzureDevOpsData\log4j-jndi-be-gone\log4j-jndi-be-gone-1.0.0-standalone.jar
```

![Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability](https://jessehouwing.net/content/images/2021/12/image-17.png)
_Add log4j-jndi-be-gone agent to the Elastic Search configuration_

Then restart the Elastic Search service:

![Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability](https://jessehouwing.net/content/images/2021/12/image-20.png)
_Restart Elastic Search after adding log4j-jndi-be-gone._

# Update the Java Runtime Environment for Search

**While you're at it, [update the JVM to the most recent version](https://docs.microsoft.com/en-us/azure/devops/project/search/install-configure-search?WT.mc_id=DOP-MVP-5001511&view=azure-devops#java-installation-notes). You can [find the latest Java 8 Runtime here](https://ww2.azul.com/downloads/?version=java-8-lts&os=windows&architecture=x86-64-bit&package=jre#download-openjdk).**

> **Note**  
> If you choose to use Azul Zulu OpenJDK, ensure that you [download version 8](https://ww2.azul.com/downloads/?version=java-8-lts&os=windows&architecture=x86-64-bit&package=jre#download-openjdk).

Extract the zip file into the `AzureDevOpsData\Java\` folder on your Application Tier:

![Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability](https://jessehouwing.net/content/images/2021/12/image-10.png)
_Extract the downloaded JDK to your AzureDevOpsData\Java folder_