How to set up Jenkins and a Pipeline on AWS
Tags: aws, jenkins, tutorial
Published: 2022-12-31 23:00:00

## Table of Contents

1. [Introduction](#Intro)
2. [Create an IAM User](#User)
3. [Create a Key pair](#KP)
4. [Create a Security Group](#SG)
5. [Create an EC2 instance](#EC2)
6. [Install and Configure Jenkins](#Jenkins)
7. [Create a Pipeline](#Pipeline)
8. [Clean Up](#cleanup)
9. [Summary](#summary)
10. [Referrals](#referrals)

<a name="Intro"></a>

### Introduction

In this blog we are going to set up Jenkins on an EC2 instance and then set up a pipeline to copy a file from S3.
We are going to start something very basic and enhance it as we move along.

### What is Jenkins?

Jenkins offers a simple way to set up a continuous integration or continuous delivery (CI/CD) environment for almost any combination of languages and source code repositories using pipelines, as well as automating other routine development tasks. While Jenkins doesn’t eliminate the need to create scripts for individual steps, it does give you a faster and more robust way to integrate your entire chain of build, test, and deployment tools than you can easily build yourself.

### What is Jenkins Pipeline?

Jenkins Pipeline is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins.

Pipeline provides an extensible set of tools for modeling simple-to-complex delivery pipelines `as code` via the [Pipeline domain-specific language (DSL) syntax](https://www.jenkins.io/doc/book/pipeline/syntax).

---

### Demo

Let's get started with the demo.

<a name="User"></a>

#### Step 1. Create an IAM User

1. Navigate to IAM > Users > Click on **Add users**
2. Enter User Name as JenkinsUser
3. Select Access key - Programmatic access and Click **Next: Permissions**
4. Click Attach existing policies directly and then **Create policy**
5. In the Create policy window, Click on **JSON** and add the following _JenkinsEC2Policy_ and save the policy as **JenkinsEC2Policy**
6. Select AmazonS3ReadOnlyAccess and  JenkinsEC2Policy under **Attach existing policies directly**
7. Click **Next:Tags**, **Next:Review** and then **Create user**
8. Download the credentials **.csv** file as we will need this during Jenkins configuration at Step 6. 

    _JenkinsEC2Policy_

    ```json
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1312295543082",
            "Action": [
                "ec2:DescribeSpotInstanceRequests",
                "ec2:CancelSpotInstanceRequests",
                "ec2:GetConsoleOutput",
                "ec2:RequestSpotInstances",
                "ec2:RunInstances",
                "ec2:StartInstances",
                "ec2:StopInstances",
                "ec2:TerminateInstances",
                "ec2:CreateTags",
                "ec2:DeleteTags",
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceTypes",
                "ec2:DescribeKeyPairs",
                "ec2:DescribeRegions",
                "ec2:DescribeImages",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "iam:ListInstanceProfilesForRole",
                "iam:PassRole",
                "ec2:GetPasswordData"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
    }
    ```
![IAM_JenkinsUser1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ga6tlhu1j86dq60kmvv0.png)
> IAM_JenkinsUser1

![IAM_JenkinsUser2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r8ayag4dtjl9yxj45gg7.png)
> IAM_JenkinsUser2

![IAM_JenkinsUser3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jvvvukxtn6q70ve32af9.png)
> IAM_JenkinsUser3

---

<a name="KP"></a>

#### Step 2. Create a key pair

1. Navigate to EC2 > Key Pairs (under Network & Security).
2. Click `Create key pair`.
3. Navigate to the folder where the key pair is downloaded and run.
   
   ```bash
   chmod 400 <key_pair_name>.pem
   ```

![Jenkins_Keypair](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vv4s38p2f973r366acce.png)
> Jenkins_Keypair

---

<a name="SG"></a>

#### Step 3. Create a Security Group

1. We are going to create a Security Group for SSH and Jenkins web access.
2. Navigate to EC2 > Security Groups > Create a new security group for your ALB, and set the following values:
   * Name: `JenkinsSG`.
   * Add an Inbound rule to allow `SSH (TCP 22)` traffic from `My IP`.
   * Add another Inbound rule to allow `Custom (TCP 8080)` traffic from `My IP`.


![Jenkins_SG](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iyd59lgalp2vw509nhhm.png)
> Jenkins_SG

---


<a name="EC2"></a>

#### Step 4. Create an EC2 instance

1. Navigate to EC2 > EC2 Dashboard > Click on **Launch instance**.
2. Launch an instance with the following values as shown in the screenshots.
3. When the instance state is Running, select the instance and click on Connect and then copy the connections details.
4. Connect to the instance.

    ```bash
    ssh -i "Jenkins-Keypair.pem" ec2-user@ec2-54-211-70-130.compute-1.amazonaws.com
    ```


![Jenkins_EC1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kqrb3o7w58bcsmhsj5o0.png)
> Jenkins_EC1

![Jenkins_EC2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/voh8taye0u2m7cupxf9w.png)
> Jenkins_EC2

-----

<a name="Jenkins"></a>

#### Step 5. Install and Configure Jenkins

---

##### Install Jenkins

1. Ensure that the software packages are up to date on the instance by executing the following command:
    ```
    [ec2-user ~]$ sudo yum update –y
    ```

2.  Add the Jenkins repo using the following command:
    ```
    [ec2-user ~]$ sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
    ```

3. Import a key file from Jenkins-CI to enable installation from the package:

    ```
    [ec2-user ~]$ sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key

    [ec2-user ~]$ sudo yum upgrade
    ```

4. Install Java:

    ```
    [ec2-user ~]$ sudo amazon-linux-extras install java-openjdk11 -y
    ```

5. Install Jenkins:

    ```
    [ec2-user ~]$ sudo yum install jenkins -y
    ```

6. Enable Jenkins service to auto start at boot:

    ```
    [ec2-user ~]$ sudo systemctl enable jenkins
    ```

7. Start Jenkins as a service:

    ```
    [ec2-user ~]$ sudo systemctl start jenkins
    ```

8. You can check the status of the Jenkins service using the command:

    ```
    [ec2-user ~]$ sudo systemctl status jenkins
    ```

9. Just in case we ever need to restart Jenkins during this setup/configuration.

    ```
    [ec2-user ~]$ sudo systemctl restart jenkins
    ```

---

##### Configure Jenkins

1. Copy the **Public IPv4 DNS** of EC2 instance and paste the URL as following in the browser
For Example:  http://ec2-54-211-70-130.compute-1.amazonaws.com:8080
2. Enter the initialAdminPassword from `/var/lib/jenkins/secrets/initialAdminPassword` and Click **Continue**

    ```
    [ec2-user@ip-172-31-89-157 ~]$ sudo cat /var/lib/jenkins/secrets/initialAdminPassword
        671x0x5x3xxx46xxxxxx099x1xf0149x
    ```

3. Select Install suggested plugins.
4. Once the installation is complete, Create First Admin User will open. Enter your information, and then select **Save** and **Continue**.
5. Click **Dashboard**, select **Manage Jenkins**, and then select **Manage Plugins**.
6. Click **Available plugins**, Search and Select **Amazon EC2** and then ***Install without restart**
7. Once the installation is complete, Navigate back to **Dashboard**, select **Manage Jenkins**, select **Manage nodes and clouds** and then click on **Configure Clouds**.
8. Select **Add a new cloud**, and select **Amazon EC2**. A new pop up window opens with more fields.
9. Click Add under Amazon EC2 Credentials and Select **Jenkins**
    + From the **Jenkins Credentials Provider: Jenkins**, select **AWS Credentials** as the Kind.
    + Enter **Access Key ID**, **Secret Access Key** from the key pair and Click **Add**.
10. Scroll down to Region and select your Region.
11. Click Add under **EC2 Key Pair's Private Key** and Select **Jenkins**.
    + From the **Jenkins Credentials Provider: Jenkins**, select `SSH Username with private key` as the Kind and set the Username to `ec2-user`.
    + Select `Enter Directly` under **Private Key**, then select Add.
    + Open the private key pair you created in the creating a key pair step and paste in the contents from `-----BEGIN RSA PRIVATE KEY-----` to `-----END RSA PRIVATE KEY-----`. Select **Add** when completed.
12. Scroll down to **Test Connection** and ensure it states `Success` and then Click **Save**.


![Jenkins_Configure1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t07lvv80n4ojez5dbexg.png)
> Jenkins_Configure1

![Jenkins_Configure2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cy8tzfxlohsgh547fl7k.png)
> Jenkins_Configure2

![Jenkins_Configure3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jbtt6p7u5zqbn2yo5xww.png)
> Jenkins_Configure3

![Jenkins_Configure4](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pfwtd3drjvixrcz2gfq0.png)
> Jenkins_Configure4

![Jenkins_Configure5](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cz56xkuvt3j8rjexotux.png)
> Jenkins_Configure5

![Jenkins_Configure6](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/utg05l3slf9ehvy709i1.png)
> Jenkins_Configure6

![Jenkins_Configure7](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iirlukkodtzj843zkbtu.png)
> Jenkins_Configure7

![Jenkins_Configure8](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4lijcvyowwk1x6mmkrxk.png)
> Jenkins_Configure8

![Jenkins_Configure9](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mjoqh28y1ujnjsrbnju0.png)
> Jenkins_Configure9

![Jenkins_Configure10](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fcdie7hgad701yxct5ry.png)
> Jenkins_Configure10

![Jenkins_Configure10.1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ekadyu6on2j1er4orw46.png)
> Jenkins_Configure10.1

![Jenkins_Configure10.2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wpaxchqzq9aeez6tv63c.png)
> Jenkins_Configure10.2

![Jenkins_Configure11](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e93se75zqsxrwan4zzgp.png)
> Jenkins_Configure11


![Jenkins_Configure11.1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/41ta102qob85j999xaf3.png)
> Jenkins_Configure11.1

![Jenkins_Configure11.2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ly7937rs78c1pw9fhgmy.png)
> Jenkins_Configure11.2

![Jenkins_Configure12](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z98zpohy6nheunwc8kjx.png)
> Jenkins_Configure12

-----

<a name="Pipeline"></a>

#### Step 6. Create a Pipeline

1. Navigate to **Dashboard** > **Manage Jenkins** and copy the **ID** of **IAM User** created in Step 1, which we need to replace `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` with the **ID**.
2. Navigate to **Dashboard** > **Manage Jenkins** > **Plugin Manager** > Click on **Available plugins**
3. Search for **AWS Steps** and **Install without restart**.
4. Navigate to **Dashboard** > **New Job** > Enter `download-a-file-from-s3` > Select **Pipeline** and Click **OK**.
5. Scroll down to **Pipeline**, add the following `Pipeline script` and Click **Save**
6. Create an S3 bucket and copy some files to the bucket.
7. Replace the `s3bucket` with your bucket name and `filename` with one of the files in your S3 bucket.
8. Navigate to **Dashboard** > **download-a-file-from-s3** and Click on **Build Now**.
9. Navigate to the latest **Build History** link and check out the **Console Output**.

    ```groovy
    pipeline 
    {
    agent any
        stages
        {
        stage('S3download') 
            {      
                steps {
                    withAWS(region:'us-east-1',credentials:'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')\
                    {
                        s3Download(file: "filename", bucket: 's3bucket', path: '')
                    }
                }
            }
        }
    }
    ```


![Jenkins_Pipeline1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fr298gqzb3rppfo0gx3m.png)
> Jenkins_Pipeline1

![Jenkins_Pipeline2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/18iubvi3jnsvrtqa20es.png)
> Jenkins_Pipeline2

![Jenkins_Pipeline3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t8xax56wosfjrma4tfr5.png)
> Jenkins_Pipeline3

![Jenkins_Pipeline4](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t516tiu5jkr7arapavwx.png)
> Jenkins_Pipeline4

The following is the console log, which shows the pipeline has been successful. :white_check_mark:

```
Started by user Sri
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/download-a-file-from-s3
[Pipeline] {
[Pipeline] stage
[Pipeline] { (S3download)
[Pipeline] withAWS
Constructing AWS CredentialsSetting AWS region us-east-1 
[Pipeline] {
[Pipeline] s3Download
Downloading s3://s3bucket/ to file:/var/lib/jenkins/workspace/download-a-file-from-s3/receiveMessages.sh 
Finished: Downloading from s3bucket/
Download complete
[Pipeline] }
[Pipeline] // withAWS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
```

The file that was copied from S3 to the EC2 instance. :white_check_mark:

```
[ec2-user@ip-172-31-89-157 ~]$ ls -lrt /var/lib/jenkins/workspace/download-a-file-from-s3/
total 0
drwxr-xr-x 2 jenkins jenkins 55 Dec 31 03:45 receiveMessages.sh
```

Let's run the build again, this time the build has failed with the following error :question:
The error message suggests to use `set force=true`.

```
Started by user Sri
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/download-a-file-from-s3
[Pipeline] {
[Pipeline] stage
[Pipeline] { (S3download)
[Pipeline] withAWS
Constructing AWS CredentialsSetting AWS region us-east-1 
[Pipeline] {
[Pipeline] s3Download
Downloading s3://s3bucket/ to file:/var/lib/jenkins/workspace/download-a-file-from-s3/receiveMessages.sh/ 
Download failed due to existing target file; set force=true to overwrite target file
[Pipeline] }
[Pipeline] // withAWS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
java.lang.RuntimeException: Target exists: file:/var/lib/jenkins/workspace/download-a-file-from-s3/receiveMessages.sh/
    at de.taimos.pipeline.aws.S3DownloadStep$Execution.run(S3DownloadStep.java:146)
    at de.taimos.pipeline.aws.S3DownloadStep$Execution.run(S3DownloadStep.java:113)
    at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Finished: FAILURE
```

So we need to add `force:true` as per the [Pipeline: AWS Steps documentation](https://plugins.jenkins.io/pipeline-aws/#plugin-content-s3download)

```groovy
pipeline 
{
agent any
    stages
    {
    stage('S3download') 
        {      
            steps {
                withAWS(region:'us-east-1',credentials:'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')\
                {
                    s3Download(file: "filename", bucket: 's3bucket', path: '', force:true)
                }
            }
        }
    }
}
```

The build succeeded after adding `force:true` and now we can run the build multiple times.

##### Now let's parameterise the destination foldername.

1. Navigate to Dashboard > download-a-file-from-s3 and click on **Configure**
2. Select **This project is parameterised** and then Add **String** parameter
3. Enter **Name** as `foldername`, **Default Valu** as `foldername` and **Save**
4. Click **Build with Parameters**
5. Navigate to the latest **Build History** link and check out the **Console Output**.

```groovy
pipeline 
{
agent any
    stages
    {
    stage('S3download') 
        {      
            steps {
                withAWS(region:'us-east-1',credentials:'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')\
                {
                    echo "${foldername}" 
                    s3Download(file: "${foldername}", bucket: 's3bucket', path: '', force:true)
                }
            }
        }
    }
}
```

![Jenkins_Pipeline_Parameter](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1xyk8yztvoehas13a3r7.png)
> Jenkins_Pipeline_Parameter

![Jenkins_Pipeline_Build-with-Parameters](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pg9btgp61wxrp4u2e5lq.png)
> Jenkins_Pipeline_Build-with-Parameters

The following is the console log, which shows the pipeline has been successful. :white_check_mark:

```
Started by user Sri
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/download-a-file-from-s3
[Pipeline] {
[Pipeline] stage
[Pipeline] { (S3download)
[Pipeline] withAWS
Constructing AWS CredentialsSetting AWS region us-east-1 
[Pipeline] {
[Pipeline] echo
s3files
[Pipeline] s3Download
Downloading s3://s3bucket/ to file:/var/lib/jenkins/workspace/download-a-file-from-s3/s3files 
Finished: Downloading from sqssri/
Download complete
[Pipeline] }
[Pipeline] // withAWS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
```

The files were copied to **s3files** directory on the EC2 instance. :white_check_mark:

```
[ec2-user@ip-172-31-89-157 ~]$ ls -lrt /var/lib/jenkins/workspace/download-a-file-from-s3/s3files 
total 8
-rw-r--r-- 1 jenkins jenkins 952 Dec 31 03:40 receiveMessages.sh
-rw-r--r-- 1 jenkins jenkins 646 Dec 31 03:40 sendMessages.sh
```

---

<a name="cleanup"></a>

#### Clean Up

1. Delete the `EC2` instance.
2. Delete the `JenkinsUser`.
3. Delete the `key pair`.
4. Delete the Security Group `JenkinsSG`.

---

<a name="summary"></a>

#### Summary

- We learned how to install and configure Jenkins.
- We also learned about setting up the pipeline.

---

<a name="referrals"></a>

### Referrals

- [Jenkins User Documentation](https://www.jenkins.io/doc/#jenkins-user-documentation)
- [Pipeline: AWS Steps](https://plugins.jenkins.io/pipeline-aws/#plugin-content-s3download)