Jenkins Upgrade from 2.1x to 2.4x
Tags: jenkins, upgrade, security, vulnerabilities
Published: 2024-02-20 06:35:14

This Article speaks about the Jenkins upgrade from 2.1X to 2.4x Due to vulnerabilities observed in the Jenkins on 24th Jan 2024.

References 
For more information on CVE-2024-23897, please refer to the following sources:]

1. https://www.jenkins.io/security/advisory/2024-01-24/
2. https://thehackernews.com/2024/01/critical-jenkins-vulnerability-exposes.html
3. https://www.bleepingcomputer.com/news/security/exploits-released-for-critical-jenkins-rce-flaw-patch-now/
4. https://github.com/jenkinsci-cert/SECURITY-3314-3315

##### Vulnerable versions
Jenkins 2.441 and earlier, LTS 2.426.2 and earlier.

###Temporary mitigation
Access to the CLI needs to be disabled. Both of the following steps must be taken:

- Remove the CLI HTTP endpoint.
- Disable the SSH Port

Both steps can be performed by executing the below script in script console of Jenkins UI(jenkins-->mange jenkins-->script-->console)


```
def removal = { lst ->
  lst.each { x -> if (x.getClass().getName()?.contains("CLIAction")) lst.remove(x) }
}
def j = jenkins.model.Jenkins.get();
removal(j.getExtensionList(hudson.cli.CLIAction.class))
removal(j.getExtensionList(hudson.ExtensionPoint.class))
removal(j.getExtensionList(hudson.model.Action.class))
removal(j.getExtensionList(hudson.model.ModelObject.class))
removal(j.getExtensionList(hudson.model.RootAction.class))
removal(j.getExtensionList(hudson.model.UnprotectedRootAction.class))
removal(j.getExtensionList(java.lang.Object.class))
removal(j.getExtensionList(org.kohsuke.stapler.StaplerProxy.class))
removal(j.actions)

println "Done!"

if (j.getPlugin('sshd')) {
  hudson.ExtensionList.lookupSingleton(org.jenkinsci.main.modules.sshd.SSHD.class).setPort(-1)
}

```

###Permanent solution/mitigation:

- Permanent mitigation can be done by upgrading the Jenkins to latest version.
- As per our current Jenkins setup automatic upgrade/migration is not possible and we have to replace the source file(i.e. Jenkins.war)
- What is Jenkins.War:  The Jenkins Web application Archive (WAR) file bundles Winstone, a Jetty servlet container wrapper, and can be started on any operating system or platform with a version of Java supported by Jenkins

> please install openjdk-17(install jdk not jre) with yum repositories before starting the upgrade which is required for 2.444 version(Latest version with fixed vulnerability at the time of writing this article

The command to install openjdk-17 is 

```shell
yum install java-17-devel
```
and please don’t create any symbolic links if you have other version of jdk in your system instead please set the config to take the java 17 by below command.

```shell
alternative --config java
```
this command will prompt for the versions which are available in the system

###Upgrade implementation steps on Linux:

Step 1: Stop the Jenkins service

```shell
sudo su -
service jenkins stop
```

Step 2: If the Jenkins is running in background, please kill the PID of the Jenkins by checking respective Jenkins port 8080

```shell
ps -ef | grep 8080
kill -9 PID
```

Step 3: Take backup of Jenkins home directory by zipping the file and move to temporary path. in my case the paths are given below,but it might different from system to system

```shell
cd /var/lib
tar -cvzf jenkins_date.tar.gz jenkins/
mv jenkins_date.tar.gz to /tmp path
```

Step 4: Take the backup of Jenkins current version binary(jenkins.war) using following commands.

```shell
cd /usr/lib/jenkins/
mv jenkins.war jenkins_old.war
```

Step 5: The webroot folder which is `/var/cache/Jenkins` has to be empty, when we are starting with new jenkins.war file. 
so take the backup of war folder(`mv war war_old`)and empty the folder (so that it will extract new configuration in war cache folder)

```shell
mv war war_old
rm -rf war/*
chown Jenkins:Jenkins war
chmod 755 war/
```

Step 6: Download The New Jenkins Version and We can check the downloaded war file of SHA by 

```shell
wget https://updates.jenkins-ci.org/latest/jenkins.war
sha256sum jenkins.war
```
Note: if the server is behind the proxy please execute the http and https proxy commands before downloading it.

step 7: Start The Jenkins Service
```shell
service jenkins start
```

There are several difficulties has been faced during this since it is a major version upgrade
1. Unable to start the Jenkins service.

- it is due to incompatible `init.d/jenkins` file where the  --daemon is not supported anymore 
- and comment the handlermaxCount, handlerMaxIdle (https://www.jenkins.io/doc/upgrade-guide/2.387/#de-duplicate-logging-implementations)

2. The Jenkins will start in the background but still the service shows as failed.

- To resolve this completely please take the Jenkins command which is used to start and create Jenkins.service file like below. 

```shell
service Jenkins status
```

> Note: the starting command can be shown while checking the status, so please take the command and convert to below service file.
- Create jenkis.service file in /etc/systemd/system with the following content(from the above copied command)
- ExecStart command should match with the above copied command
 

```shell
[Unit]
Description=Jenkins Service
After=network.target

[Service]
Type=simple
User=jenkins
Group=jenkins
ExecStart=/etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --httpPort=8080
Restart=always

[Install]
WantedBy=multi-user.target
```

- Please enable the service after creating the jenkins.service file
```shell
systemctl enable Jenkins.service
```

- And start with the below command 

```shell
systemctl start Jenkins.service
```

> Note: please delete the init.d/Jenkins file if it is present in server , that might cause multiple instance starts
> this could be different from server to server please check your server config)


- var/lib/Jenkins—JENKINS_HOME
- /usr/lib/Jenkins—Jenkins.war
- /var/cache/Jenkins/war—webroot war extraction location
- Inti.d/jenkins—manually written service file location
- /etc/system/system/--- systemctl controlled services configuration locations