Jenkins Security: Storing Secrets
Tags: devops, tutorial, productivity, opensource
Published: 2023-08-04 19:20:00

Jenkins is by far one of my favorite open source tools. Jenkins is an open source automation that that helps and provides support for building and deploying apps. Jenkins Plugins are what enable it to be customized and extended. 

> You are new to Jenkins? Quickly try it out by running its docker image, with the command
> 
> `docker run -p 8080:8080 jenkins/jenkins`

## CI/CD

Automation is fundamental to _Continuous Integration_ and _Countinuous Delivery_. Jenkins build can be setup for Continuous Integration tasks such as running integration testing. And Continuous Delivery activities such as building a final jar and pushing it to a repository or building a Docker image and pushing it to a registry. However all such integrations require credentials for authorization. Applications tend have a lot of other secrets such as Certificates for authentication for [MASSL](https://en.wikipedia.org/wiki/Mutual_authentication), credentials for upstream systems, databases and API tokens.

We cannot talk of _Continuous Integration_ and _Continuous Delivery_ (`CI/CD`) and not talk about automation. Jenkins is a CI tool and its builds can be configured to run Continuous Integration tasks such software integration testing. Not only that but Continuous Delivery actions such as creating a final jar (java app artifact) and pushing it to a repository or even creating a Docker image and pushing it to a registry are examples of Continuous Delivery activities. All of these integrations however need credentials to be able to work. Certificates for authentication for [MASSL](https://en.wikipedia.org/wiki/Mutual_authentication) credentials for upstream systems, databases, and API tokens are common secrets in applications.

## Jenkins Plugins for Secret Management

Below are some plugins that allow basic credential management: 

-   _[Credentials Binding Plugin](https://wiki.jenkins.io/display/JENKINS/Credentials+Binding+Plugin)_: These allow credentials to be bound to environment variables for use in build steps.
-   _[Credentials Plugin](https://wiki.jenkins.io/display/JENKINS/Credentials+Plugin)_: On the other hand, this plugin allows you to store your credentials in Jenkins. 

> N.B. These plugins are available in default installation of Jenkins.

Below are secrets which are supported by default:  
![test](https://malikashish8.github.io/assets/images/storing-secrets-in-jenkins/default_plugin.png "Default plugin secret options")

### Using secrets in Freestyle Job

For our test, let’s create a Free style Jenkins Project and use `Credentials` plugins to store our `Dockerhub` credentials. As seen below, select _Username and password (separated)_ and enter bindings for the credentials. Basically, these are the environment variables that username and password will be available as:

![test](https://malikashish8.github.io/assets/images/storing-secrets-in-jenkins/bindings.png "Enter bindings")

Once that is done, next click on _Add_ and select the right `scope` and enter Dockerhub credentials:

![test](https://malikashish8.github.io/assets/images/storing-secrets-in-jenkins/dockerhub_credentials.png "Enter Dockerhub credentials")

Now here is an example of how we use our credentials

![secrets](https://malikashish8.github.io/assets/images/storing-secrets-in-jenkins/use_secrets.png "use credentials")

### Using Credentials in Jenkins Pipeline

Now let us look how we can use our credentials in Jenkins Pipeline. In the following stage of our `Jenkinsfile` we are will be using `artifactory-credentials` to login to the artifactory and push our docker images to it. Also note that `usernameVariable` and `passwordVariable` are available in the `withCredentials` block:

```
    stage("docker_push") {
      withCredentials([usernamePassword(credentialsId: 'artifactory-credentials', 
        passwordVariable: 'ARTIFACTORY_KEY', 
        usernameVariable: 'ARTIFACTORY_USER')]) 
      {
        sh "echo $ARTIFACTORY_KEY | docker login -u $ARTIFACTORY_USER --password-stdin ${REGISTRY_URL}"
        sh "docker tag myapp:latest ${REGISTRY_URL}/myapp:${shortCommit}"
        sh "docker push ${REGISTRY_URL}/myapp:${shortCommit}"
      } 
```

### Retrieving Secrets

Jenkins tries to provide some sense of security by masking the credentials in logs. It does that by looking for an exact match and replaces them with asterisks (\*\*\*\*\*). But note also that this control can simply be bypassed by a user with edit permission by encoding the credentials in the pipeline and printing them in logs.

![test](https://malikashish8.github.io/assets/images/storing-secrets-in-jenkins/encode_password.png)

![test](https://malikashish8.github.io/assets/images/storing-secrets-in-jenkins/password_encoded.png)

> Please note that the first echo results in our password being `masked` since it matches our password string.

Now let us decode the encoded password and see what it gives us:

```
dev@ubu:~ $ echo YWRtaW4xMjM0Cg== | base64 -d
admin1234
```

Let us explain this. Actually, Jenkins credentials are stored in Jenkins master [encrypted by Jenkins instance id](https://jenkins.io/doc/book/using/using-credentials/#credential-security). This file usually exists in your linux server at `/var/lib/jenkins/credentials.xml`. It is therefore very easy to decrypt credentials saved in Jenkins for Jenkins administrators. Moreover, do note that any user that can SSH to Jenkins can read the encrypted credentials file since it has read permissions for everyone. Let us look at that:

```
dev@ubu:/var/lib/jenkins$ ls -la credentials.xml 
-rw-r--r-- 1 jenkins jenkins 4650 Mar 26 09:28 credentials.xml
```

This file has passwords encrypted with Jenkins Id:

```
<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
    <scope>GLOBAL</scope>
    <id>artifactory-credentials</id>
    <description></description>
    <username>admin</username>
    <password>{AQAAABAAAAAQom3LN7ei0wdm9cdOlGOa4GxDHzpndn0BUPeI4biARto=}</password>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
```

Our favorite CI-tool, Jenkins provides a handy utility at `/script` of the URL that can be used to decrypt passwords with the following script:

```
encryptedPassword = '{AQAAABAAAAAQom3LN7ei0wdm9cdOlGOa4GxDHzpndn0BUPeI4biARto=}'
passwd = hudson.util.Secret.decrypt(encryptedPassword)
println(passwd)
```

![test](https://malikashish8.github.io/assets/images/storing-secrets-in-jenkins/jenkins_script.png)

**In Conclusion**, it is very clear that secrets entered in Jenkins will not be disclosed. Any user that can use a secret in a build can decode and see it. All and any user that can SSH into Jenkins master server can decrypt all secrets using `/script` utility if they can login to Jenkins Web UI.

## Management Issues with Hardcoding Credentials

As we just saw, the `principle of least privilege` cannot be effectively employed when just using Jenkins `Credentials` Plugin. Therefore, there has to be a better solution for auditable credential sharing within team, credential rotation and automatic build provisioning that we can use. For that, I would suggest a number of credential management products such as [Hashicorp Vault](https://www.vaultproject.io/), [Cyberark Vault](https://www.cyberark.com/products/privileged-account-security-solution/enterprise-password-vault/) and [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/). Some open source products like [CredStash](https://github.com/fugue/credstash) for AWS help with credential management.