Introduction To Jenkins Shared Libraries
Tags: jenkins, cicd, testing, groovy
Published: 2023-06-26 02:21:53

Shared libraries in Jenkins are a great way to organize the build process in modular components. The [following repository](https://github.com/cwgem/pythonsample) contains base code that will be used in this tutorial. We'll look at the basics of how shared libraries can be utilized.

## Shared Library Preparation

Jenkins shared libraries have a specific structure to work with as follows (taken from the wonderful [shared libraries documentation](https://www.jenkins.io/doc/book/pipeline/shared-libraries/)):

```
(root)
+- src                     # Groovy source files
|   +- org
|       +- foo
|           +- Bar.groovy  # for org.foo.Bar class
+- vars
|   +- foo.groovy          # for global 'foo' variable
|   +- foo.txt             # help for 'foo' variable
+- resources               # resource files (external libraries only)
|   +- org
|       +- foo
|           +- bar.json    # static helper data for org.foo.Bar
```

`src` and `resources` are tied together while `vars` can act more independently. For simple calls `vars` is good enough. `src` will let you handle things in a more java class like layout. To get things started I'll be creating a gitolite repo (setup details can be [found here](https://dev.to/cwprogram/windows-jenkins-server-running-wsl-docker-agents-1kcj)) to hold the shared libraries. Under `conf/gitolite.conf` in the `gitolite-admin` repository:

```
repo jenkins-shared-library
        RW+ = private_git_server
```

After checking out I'll create a sample directory structure like this:

```
.
└── vars
    └── buildTest.groovy

1 directory, 1 file
```

Where `buildTest.groovy` looks like this:

```groovy
def call() {
  sh 'python3 --version'
}
```

As indicated by the extension Jenkins shared libraries are primarily written in the [Groovy language](https://groovy-lang.org/). It's a JVM based language with a more friendlier syntax in dealing with the scripted like nature of Jenkins pipelines. Finally I'll commit and push the changes to the repository.

## Declarative vs Scripted Pipeline

In terms of shared libraries, scripted pipelines are more beneficial if you want to fully utilize them. Declarative pipelines can work with them, it's just that they tend to be more verbose when written out than scripted pipelines. More so if your end goal is to use libraries for standardizing the build process for a large number of Jenkinsfiles by keeping things compact. With this in mind I'll mostly be utilizing scripted pipelines but will point out any gotchas if they come up.

## Defining and Using Shared Libraries

Now Jenkins by default doesn't know this shared library exists so we have to setup the instance for that. First go to the Jenkins top page then:

1. Click on "Manage Jenkins"
2. Click on "Configure System" under "System Configuration" towards the top
3. Scroll down until you see "Global Pipeline Libraries"
4. Click "Add"
5. Enter an appropriate Name which will be used to identify the library for import purposes (JenkinsShared as an example)
6. "Default Version" in this case is what git identifier will be used, in this case I'll put "main" which is my default HEAD identifier for the repo
7. I'll ignore the other options for now and instead fill out "Project Repository" which in my case will be `git@gitserver:jenkins-shared-library`
8. For "Credentials" I'll select my appropriate git connection credentials
9. Then I'll click "Save" at the bottom since there's nothing else to do

After coming back to the settings again `Default Version` should show a banner under it that looks like:

"Currently maps to revision: 5e7125183cc7525ae1669ca721349277824dbdde"

Showing that your repository mapping looks okay. Now it's time to actually use this code. I'll use this `Jenkinsfile` for that purpose:

```
@Library('JenkinsShared') _
node('jenkins-agent')
{
    buildTest()
}
```
**Note:** The `_` is being used as a replacement for a long import statement and if left out will cause a syntax error

The `@Library('JenkinsShared') _` is pulling in the library for use and `buildTest()` is calling the actually code in the library. I'll now commit and push this code, which will proc the build via [gitolite hooks](https://dev.to/cwprogram/gitolite-hooks-for-automated-jenkins-build-start-of-multibranch-pipelines-3930):

```
> git fetch --no-tags --force --progress -- git@gitserver:jenkins-shared-library +refs/heads/*:refs/remotes/origin/* # timeout=10
Checking out Revision 5e7125183cc7525ae1669ca721349277824dbdde (main)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5e7125183cc7525ae1669ca721349277824dbdde # timeout=10
Commit message: "Initial Commit"
 > git rev-list --no-walk 5e7125183cc7525ae1669ca721349277824dbdde # timeout=10
<snip>
Checking out Revision a2b59c5987014dbf40fbda6e1be19dc3ca68b7e9 (master)
 > /usr/bin/git config core.sparsecheckout # timeout=10
 > /usr/bin/git checkout -f a2b59c5987014dbf40fbda6e1be19dc3ca68b7e9 # timeout=10
Commit message: "Fix steps missing"
 > /usr/bin/git rev-list --no-walk 9d4dd034d22e307970102045f6de7c71c6b507c5 # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Build)
[Pipeline] node
Running on Jenkins Agent-0007fndg4vl6o on docker in /home/jenkins/agent/workspace/GitoliteTest_master
[Pipeline] {
[Pipeline] sh
+ python3 --version
Python 3.9.2
```

The two important things are that Jenkins is both pulling the shared library from its repo using the latest version, and the appropriate code is being executed from the shared library. Now this kind of library import method is good if you want more control over the version of the library you want to use or be more specific on imports. If you're just using the default though the imports are less valuable. It turns out that Jenkins has an option in the Global Pipeline config section called "Load implicitly", which as the help describes "If checked, scripts will automatically have access to this library without needing to request it via @Library". So after selecting this option, the `@Library` header can be removed and still work:

```
node('jenkins-agent')
{
    buildTest()
}
```

## Working With vars

The vars directory holds what are often known as global variables. These provide a more imperative experience when working with pipelines and the most straightforward to work with. Now there are two ways that you can work with vars: encapsulation of methods and as calls. Looking at the first method:

`vars/python.groovy` (shared libs)
```groovy
def checkoutCode() {
    checkout scm
}

def poetryInstall() {
    sh '''
    pip install poetry
    ~/.local/bin/poetry install
    '''
}

def pytestRun() {
    sh '~/.local/bin/poetry run python -m pytest'
}
```

`Jenkinsfile` (project)
```
node('jenkins-agent')
{
    python.checkoutCode()
    python.poetryInstall()
    python.pytestRun()
}
```

In this case, `python` represents an encapsulation of `poetryInstall` and `pytestRun`. The one issue for declarative pipelines is that they must be encapsulated in a `script{}` block to work properly:

```
pipeline {
    agent { 
        label 'jenkins-agent'
    }
    stages {
        stage('Build') {
            steps {
                script {
                    python.poetryInstall()
                }
            }
        }
        stage('Test') {
            steps {
                script {
                    python.pytestRun()
                }
            }
        }
    }
}
```

That said, declarative pipelines also don't require the `checkoutCode()` portion to work properly as that's done by default. The advantage here is that this method allows for a more namespace like format if you wish that level of explicit declaration. On the other hand it's also requiring slightly more work being done on the `Jenkinsfile` side. Now let's take a look at an alternative:

`vars/pythonBuild.groovy` (shared libs)
```groovy
def call() {
    checkout scm
    sh '''
    pip install poetry
    ~/.local/bin/poetry install
    ~/.local/bin/poetry run python -m pytest
    '''
}
```

**Note**: Realistically poetry should be part of the docker agent image, but explaining that would have complicated the article a bit so I decided to simply use this workaround

`Jenkinsfile`
```
node('jenkins-agent')
{
    pythonBuild()
}
```

In this case the step separation is missing (something we'll get into fixing later) and the install and test steps are no longer broken out in the `Jenkinsfile`. The `script{}` block is no longer needed as well when doing declarative pipelines:

```
pipeline {
    agent { 
        label 'jenkins-agent'
    }
    stages {
        stage('Build') {
            steps {
                pythonBuild()
            }
        }
    }
}
```

This makes things a lot more compact. We can also separate out the install and build process on the shared libs side:

```groovy
def call() {
    checkoutCode()
    installPoetry()
    runTests()
}

def checkoutCode() {
    checkout scm
}

def installPoetry() {
    sh '''
    pip install poetry
    ~/.local/bin/poetry install
    '''
}

def runTests() {
    sh '~/.local/bin/poetry run python -m pytest'
}
```

This is a lot more cleaner on the organization side. 

## src Libraries

Another method is using a more class style method similar to how traditional Java imports work. It also allows multiple instances of a class along with inheritance versus the singleton nature of the vars solution. Simple classes not related to pipeline steps are rather simple:

`src/org/foo/Human.groovy`
```groovy
package org.foo

class Human {
    String name
}
```

`Jenkinsfile`
```
node('jenkins-agent')
{
    human = new org.foo.Human()
    human.setName('John Smith')
    sh "echo ${human.name}"
}
```

Sample output of the result:

```
[Pipeline] {
[Pipeline] sh
+ echo John Smith
John Smith
```

Now if I try to abstract out the `sh` line to the library:

`src/org/foo/Human.groovy`
```groovy
package org.foo

class Human {
    String name

    void sayName() {
        sh "echo ${name}"
    }
}
```

`Jenkinsfile`
```
node('jenkins-agent')
{
    human = new org.foo.Human()
    human.setName('John Smith')
    human.sayName()
}
```

I'll get this nasty error:

```
Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: de622af5-c724-46ba-af61-e3a70a709322
hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: org.foo.Human.sh() is applicable for argument types: (java.lang.String) values: [$name]
```

This is because the class as is isn't aware of steps and thinks you're trying to call an `sh` method of the `Human` class. To fix this, we'll change the class as follows:

```groovy
package org.foo

class Human implements Serializable {
  def steps
  String name

  Human(steps) {this.steps = steps}
  def sayName() {
    steps.sh "echo ${name}"
  }
}
```

Serialization is a method of taking an object into something that can be written externally to a text file, passed over the network, and still produce the object in question. Jenkins needs this as it needs the ability to suspend and resume tasks, which having a serializable form of an object enables this. From there `this` is passed into the constructor which contains the script environment to allow access to the steps. This roundabout way of doing things is again why I recommend using the global variable method as much as possible (it's also easier to reason with if you're used to CircleCI/GitHub actions workflows).

## Environment Variables

Environment variables contain a wealth of information about several useful properties of the system. For example:

- A user's home directory on the local system
- A custom set environment
- The name of a git branch that triggered a build

These variables also can come from several sources:

- OS level
- Project level environment variables
- Cloud/node agent level environment variables
- Global environment variables
- Environment variables defined via `withEnv()`
- Environment variables exposed via Jenkins plugins

This means that what environment variables are available to you will be up to the Jenkins setup and other factors. You may need to have additional plugins installed as well. Accessing an environment variable is fairly simple:

```groovy
sh "echo ${env.HOME}"
```

You can also get an environment variable via a key known at runtime via:

```groovy
def myEnvVariable = env."${someKeyHere}"
```

Using `withEnv` is also very useful, for example the situation with poetry PATH:

```groovy
newPath = env.PATH + ':' + env.HOME + '/.local/bin'
withEnv(["PATH=${newPath}"]) {
  sh '''
  pip install poetry
  poetry install
  '''
}
```

Here path has the home directory (`/home/jenkins` in this case) which is an OS variable, followed by `/.local/bin` where poetry lies. Now `poetry` can be used as-is without needing to prepend the path. As withEnv takes a list of strings, you can set multiple ones if you want (though that might be better handled at the project/global/agent level declaration).

## Steps and Stages

It's possible with scripted pipelines to declare stages within shared libraries. This is useful when checking the overall build to see how long things are taking and make logs easier to work with. Without that your stage view will throw back something like this:

![Jenkins warning about no stages defined](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/94bjj8tqi4i0wvcqjmox.png)

which  isn't very useful. Stage declaration is pretty straightforward though, much like how a standard Jenkins pipeline would define it. For example:

`vars/pythonBuild.groovy`
```groovy
def call() {
    stage('Checkout') {
        checkoutCode()
    }
    newPath = env.PATH + ':' + env.HOME + '/.local/bin'
    withEnv(["PATH=${newPath}"]) {
        stage('Install Poetry') {
            installPoetry()
        }
        stage('Run Tests') {
            runTests()
        }
    }
}

def installPoetry() {    
    sh '''
    pip install poetry
    poetry install
    '''
}

def checkoutCode() {
    checkout scm
}

def runTests() {
    sh 'poetry run python -m pytest'
}
```

Here different stages are declared for checking out the code, installing poetry, and running tests. This gives a nice visual overview of the different processes:

![Stage view with stages defined broken into three phases](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bilcy52v8xhizoevft1c.png)

Logs also get broken up so there's less to search:

![Logs specific to the run phase test](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s1udeytbct7ymg1d31bl.png)

While it's possible to return an entire declarative pipeline for handling steps, the result can be a bit noisy and you still have to do it at the level of what's declaring the pipeline. You also can't declare pipelines once. With that in mind it's recommended to stick with scripted pipelines if you plan to use 

An interesting feature is the ability to have a custom step like setup. As an example:

`vars/poetry.groovy`
```groovy
def call(Closure body) {
    newPath = env.PATH + ':' + env.HOME + '/.local/bin'
    withEnv(["PATH=${newPath}"]) {
        sh '''
        pip install poetry
        poetry install
        '''
        body()
    }
}
```

So a few things are going on here. First is the `Closure body` being passed to the call. This will execute the closure passed in when `body()` is called. Then from here I add poetry's binary location to `PATH` using `withEnv`. This means I no longer need to run poetry via `~/.local/bin/poetry`, including everything in the Closure passed in. Before calling that though a poetry installation and poetry install run are executed, so the closure statements won't have to worry about that. As an example use:

`vars/pythonBuild.groovy`
```groovy
def call() {
    checkoutCode()
    poetry {
        runTests()
    }
}

def checkoutCode() {
    checkout scm
}

def runTests() {
    sh 'poetry run python -m pytest'
}
```

In this case the Closure that will be passed in to the `poetry.groovy` call is:

```groovy
{
   runTests()
}
```

If you'll notice `runTests` runs without needing the `~/.local/bin/poetry` call. Poetry's install is also abstracted out by the poetry block's backend code.

## Parameters

Global variables declared via the call syntax are also able to utilize parameters. For example:

`vars/pythonBuild.groovy`
```groovy
def call(boolean shouldRunTests) {
    stage('Checkout') {
        checkoutCode()
    }
    newPath = env.PATH + ':' + env.HOME + '/.local/bin'
    withEnv(["PATH=${newPath}"]) {
        stage('Install Poetry') {
            installPoetry()
        }
        if( shouldRunTests ) {
            stage('Run Tests') {
                runTests()
            }
        }
    }
}

def installPoetry() {    
    sh '''
    pip install poetry
    poetry install
    '''
}

def checkoutCode() {
    checkout scm
}

def runTests() {
    sh 'poetry run python -m pytest'
}
```

`Jenkinsfile`
```
node('jenkins-agent')
{
    pythonBuild(false)
}
```

In this case, a boolean parameter is available to indicate whether or not tests should be run. This might be useful to break out unit tests from trunk branches. The stage view shows that the "Run Tests" stage is clearly skipped:

![Run Test stage being skipped](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5hklaurhrsty4r7zajke.png)

It's also not uncommon for parameters to be passed in as maps. This way another config can validate them for required parameters and set default ones. For example:

`vars/pythonBuild.groovy`
```groovy
def call(Map config = [:]) {
    validateConfig(config)

    stage('Checkout') {
        checkoutCode()
    }
    newPath = env.PATH + ':' + env.HOME + '/.local/bin'
    withEnv(["PATH=${newPath}"]) {
        stage('Install Poetry') {
            installPoetry()
        }
        if( config.runTests ) {
            stage('Run Tests') {
                runTests()
            }
        }
    }
}

def validateConfig(Map config) {
    requiredKeys = ['runTests']

    for ( requiredKey in requiredKeys ) {
        if ( ! config.containsKey(requiredKey) ) {
            throw new Exception("Missing required key: ${requiredKey}")
        }
    }
}

def installPoetry() {    
    sh '''
    pip install poetry
    poetry install
    '''
}

def checkoutCode() {
    checkout scm
}

def runTests() {
    sh 'poetry run python -m pytest'
}
```

`Jenkinsfile`
```
node('jenkins-agent')
{
    pythonBuild()
}
```

I now have a check to ensure that a map is passed in with `runTests` defined. If I run this without any parameters:

```
[Pipeline] End of Pipeline
Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 533cc6df-bcde-47ff-91d5-da8b1537eca4
java.lang.Exception: Missing required key: runTests
```

Providing the parameter will make this work:

```
node('jenkins-agent')
{
    pythonBuild(['runTests': true])
}
```

I can also do it without providing the map by having validation support a default for the parameter:

`vars/pythonBuild.groovy`
```groovy
def call(Map config = [:]) {
    config = validateConfig(config)

    stage('Checkout') {
        checkoutCode()
    }
    newPath = env.PATH + ':' + env.HOME + '/.local/bin'
    withEnv(["PATH=${newPath}"]) {
        stage('Install Poetry') {
            installPoetry()
        }
        if( config.runTests ) {
            stage('Run Tests') {
                runTests()
            }
        }
    }
}

def validateConfig(Map config) {
    requiredKeys = ['runTests']
    defaultKeys = ['runTests': true]

    for ( requiredKey in requiredKeys ) {
        if ( ! config.containsKey(requiredKey) ) {
            if ( defaultKeys.containsKey(requiredKey) ) {
                config[requiredKey] = defaultKeys[requiredKey]
            }
            else {
                throw new Exception("Missing required key: ${requiredKey}")
            }
        }
    }

    return config
}

def installPoetry() {    
    sh '''
    pip install poetry
    poetry install
    '''
}

def checkoutCode() {
    checkout scm
}

def runTests() {
    sh 'poetry run python -m pytest'
}
```

This now allows for `pythonBuild()` to be called without an error as the defaults will set it to true. The `Map config = [:]` sets an empty default for the map so we don't have to validate if the map is empty or not. 

## Resources

Resource are fairly simple so this section will be brief. Such resources are stored in the `resources` folder of the shared library root. They can then be accessed via the `libraryResource` call which  returns text data for the resource. As an example:

`resources/util/shell/test.sh`
```sh
echo "Hello World"
echo "Hello Again"
```

`vars/buildTest.groovy`
```groovy
def call() {
    sh libraryResource('util/shell/test.sh')
}
```

`Jenkinsfile`
```
node('jenkins-agent)
{
  buildTest()
}
```

When called this will print out both "Hello World" and "Hello Again". This is also very useful when bundled with the [Pipeline Utility Steps plugin](https://www.jenkins.io/doc/pipeline/steps/pipeline-utility-steps/) which works with several file formats. As an example with JSON:

`resources/util/json/test.json`
```json
{
    "foo": "bar"
}
```

`vars/buildTest.groovy`
```groovy
def call() {
    def result = readJSON text: libraryResource('util/json/test.json')
    sh "echo ${result.foo}"
}
```

## Conclusion

This concludes a look into the power of Jenkins shared library in managing code. I recommend working with scripted pipelines for easy of use and flexibility with stage declarations, and global variables to handle the backend code. You can also find a summary of some of these points for quick reference and the [Jenkins documentation page for shared libraries](https://www.jenkins.io/doc/book/pipeline/shared-libraries/).