AWS: IAM users keys rotation, EC2 IAM Roles and Jenkins
Tags: aws, jenkins, security
Published: 2019-05-30 16:00:34

---
title: AWS: IAM users keys rotation, EC2 IAM Roles and Jenkins
published: true
tags: AWS,Jenkins,Security
canonical_url: https://rtfm.co.ua/en/aws-iam-users-keys-rotation-ec2-iam-roles-and-jenkins/
---

![](https://rtfm.co.ua/wp-content/uploads/2014/11/aws-logo-square-02-e1417012834176-1.png)Today I checked our IAM-users and “suddenly” recalled that it’s good to update their credentials sometimes:

[![](https://rtfm.co.ua/wp-content/uploads/2019/05/Screenshot_20190530_160010.png)](https://rtfm.co.ua/wp-\content/uploads/2019/05/Screenshot_20190530_160010.png)

Well, that’s good to do but here is a question: it’s simple enough to set an expire for keys in IAM, but what to do with all scripts which are used in our Jenkins and which are using those IAM ACCESS/SECRET keys?

For example – our backend provisioning is done from Jenkins using Ansible [`cloudformation`](https://rtfm.co.ua/ansible-modul-cloudformation/) module which uses IAM user with IAM policy with EC2/RDS/CloudFormation etc Allow rules.

Our data analytics has their own jobs which makes some ETL jobs and stores results in AWS S3 buckets using IAM user with a policy which allows access to specific S3 buckets.

Now what – need to update all ACCESS/SECRET keys in all those jobs every time they will be rotated in IAM?

As a solution – [Hashicorp Vault](https://www.vaultproject.io/) could be used, which can grant access tokens to those scripts/jobs. But at first – it’s some things (and time) to set up Vault itself, its backups etc and as the second thing – it’s a piece of work to do with scripts/Jenkins jobs to update them to use Vault instead of variables with ACCESS/SECRET keys.

A better solution can be to use IAM roles for EC2 instead, as any AWS SDK will look for it during authentication, for example, [boto3 documentation says](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html):

1. Passing credentials as parameters in the <tt><span>boto.client()</span></tt> method
2. Passing credentials as parameters when creating a <tt><span>Session</span></tt> object
3. Environment variables
4. Shared credential file (<tt><span>~/.aws/credentials</span></tt>)
5. AWS config file (<tt><span>~/.aws/config</span></tt>)
6. Assume Role provider
7. Boto2 config file (<tt><span>/etc/boto.cfg</span></tt> and <tt><span>~/.boto</span></tt>)
8. Instance metadata service on an Amazon EC2 instance that has an IAM role configured.

So – let’s try to:

1. create an IAM role with a necessary policy
2. attach it to an EC2
3. use AWS CLI without configuration to test access

### Create an IAM role

Create a role, chose EC2 type:

[![](https://rtfm.co.ua/wp-content/uploads/2019/05/Screenshot_20190530_132113.png "AWS: ротация ключей IAM пользователей, EC2 IAM Roles и Jenkins")](https://rtfm.co.ua/wp-content/uploads/2019/05/Screenshot_20190530_132113.png)

Attach any policy, here is _AmazonRoute53ReadOnlyAccess_ for example:

[![](https://rtfm.co.ua/wp-content/uploads/2019/05/Screenshot_20190530_132142.png "AWS: ротация ключей IAM пользователей, EC2 IAM Roles и Jenkins")](https://rtfm.co.ua/wp-content/uploads/2019/05/Screenshot_20190530_132142.png)

Save your new role:

[![](https://rtfm.co.ua/wp-content/uploads/2019/05/Screenshot_20190530_132219.png "AWS: ротация ключей IAM пользователей, EC2 IAM Roles и Jenkins")](https://rtfm.co.ua/wp-content/uploads/2019/05/Screenshot_20190530_132219.png)

### Run EC2

Create an EC2 instance and set the IAM role created above:

[![](https://rtfm.co.ua/wp-content/uploads/2019/05/Screenshot_20190530_132333.png "AWS: ротация ключей IAM пользователей, EC2 IAM Roles и Jenkins")](https://rtfm.co.ua/wp-content/uploads/2019/05/Screenshot_20190530_132333.png)

### Testing

Log in to the instance:

```
$ ssh admin@34.244.125.163 -i setevoy-testing-eu-west-1.pem
```

Check IAM in the [instance meta-data](https://docs.aws.amazon.com/en_us/AWSEC2/latest/UserGuide/ec2-instance-metadata.html):

```
root@ip-172-31-42-77:/home/admin# curl http://169.254.169.254/latest/meta-data/iam/info
{
  "Code" : "Success",
  "LastUpdated" : "2019-05-30T10:54:26Z",

  "InstanceProfileArn" : "arn:aws:iam::534***385:instance-profile/ec2-example-role"
...
```

Install AWS CLI:

```
root@ip-172-31-42-77:/home/admin# apt update && apt -y install awscli
```

And get Route53 hosted zones without any ACCESS/SECRET keys configuration:

```
root@ip-172-31-42-77:/home/admin# aws route53 list-hosted-zones --output text
HOSTEDZONES     33C2D264-***-***-3052BEA607A9    /hostedzone/Z30***LB6      example.com. 104
CONFIG  DME sites       False
...
```

_“It works!”_ (c).

### Jenkins

Let’s go further and check if this will work on Jenkins, as at first – Jenkins itself is running in a Docker container, and as a second thing – its jobs like Ansible tasks also are started using Docker containers “inside” Jenkins.

Update existing EC2 instance:

[![](https://rtfm.co.ua/wp-content/uploads/2019/05/Screenshot_20190530_132804.png "AWS: ротация ключей IAM пользователей, EC2 IAM Roles и Jenkins")](https://rtfm.co.ua/wp-content/uploads/2019/05/Screenshot_20190530_132804.png)

Attach role created previously:

[![](https://rtfm.co.ua/wp-content/uploads/2019/05/Screenshot_20190530_132822.png "AWS: ротация ключей IAM пользователей, EC2 IAM Roles и Jenkins")](https://rtfm.co.ua/wp-content/uploads/2019/05/Screenshot_20190530_132822.png)

Create own Docker image with AWS CLI – create simple `Dockerfile`:

```
FROM python:3.7-stretch
RUN apt-get update -y
RUN pip install awscli
```

Build image:

```
root@jenkins-dev:/opt/jenkins# docker build -t setevoy/awscli .
```

Check it:

```
root@jenkins-dev:/opt/jenkins# docker run -ti setevoy/awscli aws --version
aws-cli/1.16.168 Python/3.7.3 Linux/4.9.0-8-amd64 botocore/1.12.158
```

Go to Jenkins and create a test-job.

Here is [Jenkins Pipeline](https://jenkins.io/doc/book/pipeline/docker/) used:

```
node {
    docker.image('setevoy/awscli:latest').inside('-v /var/run/docker.sock:/var/run/docker.sock') {
        stage('List zones') {
            sh "aws route53 list-hosted-zones --output text"
        }
    }
}
```

[![](https://rtfm.co.ua/wp-content/uploads/2019/05/Screenshot_20190530_141355.png "AWS: ротация ключей IAM пользователей, EC2 IAM Roles и Jenkins")](https://rtfm.co.ua/wp-content/uploads/2019/05/Screenshot_20190530_141355.png)

Run this job:

[![](https://rtfm.co.ua/wp-content/uploads/2019/05/Screenshot_20190530_141456.png "AWS: ротация ключей IAM пользователей, EC2 IAM Roles и Jenkins")](https://rtfm.co.ua/wp-content/uploads/2019/05/Screenshot_20190530_141456.png)

Great!

And let’s check if unauthorized API-call will work.

In our policy only _AmazonRoute53ReadOnlyAccess_ is allowed – let’s try to execute S3 list-buckets:

```
node {
    docker.image('setevoy/awscli:latest').inside('-v /var/run/docker.sock:/var/run/docker.sock') {
        stage('List zones') {
            sh "aws s3api list-buckets"
        }
    }
}
```

Run:

[![](https://rtfm.co.ua/wp-content/uploads/2019/05/Screenshot_20190530_143720.png "AWS: ротация ключей IAM пользователей, EC2 IAM Roles и Jenkins")](https://rtfm.co.ua/wp-content/uploads/2019/05/Screenshot_20190530_143720.png)

> + aws s3api list-buckets  
> An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied

_AccessDenied_ – nice!

But there is one issue using EC2 IAM Roles as a jenkins jobs authorization mechanism: actually, its EC2 instance will get access to every resource under your AWS account.

I.e. if somebody will get SSH access to this host – he will be able to access any other resource in our AWS.

From one hand – this can be eliminated by using EC2 spot-instances as Jenkins workers with dedicated restricted policies attached.

From another hand, if somebody will get access to your Jenkins – you’ll have big problems anyway, so just do not forget to keep it upgraded and restrict access to it via authorizations and AWS Security Groups.

* * *

> On April 9, 2019, security researcher Jaikey Sarraf alerted Matrix to existing vulnerabilities in Jenkins, which Matrix said it used for continuous integration. “The version of Jenkins we were using had a vulnerability (CVE-2019-1003000, CVE-2019-1003001, CVE-2019-1003002) which allowed an attacker to hijack credentials (forwarded ssh keys), giving access to our production infrastructure.”

[https://www.infosecurity-magazine.com/news/matrix-compromised-through-known-1/](https://www.infosecurity-magazine.com/news/matrix-compromised-through-known-1/)

### Similar posts

- <small>04/16/2019</small> [Jenkins: a job to check a Github organization’s public repositories list](https://rtfm.co.ua/en/jenkins-a-job-to-check-a-github-organizations-public-repositories-list/)<small> (0)</small>  
- <small>02/08/2017</small> [Jenkins: sandbox.RejectedAccessException: Scripts not permitted to use](https://rtfm.co.ua/jenkins-sandbox-rejectedaccessexception-scripts-not-permitted-to-use/)<small> (0)</small>