Streamlining AMIs using Packer, Vault & GitHub Actions
Tags: aws, packer, ami, hashicorp
Published: 2022-11-27 05:33:37

Nowadays, if you want to minimize human errors and maintain a consistent process for how software is released then you are going to rely on Continuous integration and continuous deployment (CI/CD). It's really hard to imagine how much productivity they bring into the plate.

In this tutorial, we are going to take entire AWS instance backup using tools like Packer and see how it solves our problem and make our life much easier.

## Amazon Machine Image (AMI)

![AMI_LOGO](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vi9rgoqia61u5nyxzp3u.png)
---

An **Amazon Machine Image (AMI)** is a special type of virtual appliance that is used to create a virtual machine within the Amazon Elastic Compute Cloud ("EC2"). It serves as the basic unit of deployment for services delivered using EC2. -- _Wikipedia_

An AMI includes the following:

- A template for the root volume for the instance (for example, an operating system, an application server, and applications)
- Launch permissions that control which AWS accounts can use the AMI to launch instances.
- A block device mapping that specifies the volumes to attach to the instance when it's launched.

---

## What is Packer ?


![PACKER_LOGO](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sa0eilmrxdyb3a19vt89.jpg)
---


Packer is a tool for building identical machine images for multiple platforms from a single source configuration.



![Packer_HashiCorp](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v28v984u83ixlemdicw4.jpg)
Image Source : https://www.hashicorp.com/


Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer comes out of the box with support for many platforms.

To know more about Packer, visit : https://developer.hashicorp.com/packer


---

## Project Structure 

GitHub Repository : https://github.com/mukulmantosh/Packer-Exercises


![project_structure](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vemswbdot9b5hqb92tzp.png)

- **.github** - Workflow files for GitHub Actions
- **packer** - Contains HCL2 Packer templates, Shell Scripts etc.
- **Dockerfile** - Building Docker Image
- **main.py** - FastAPI Routes handling two endpoints
- **requirements.txt** - listing all the dependencies for a specific Python project

---

## Let's Begin

![project_structure_2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fnrnk8tf7ek8b0nixjor.png)

I have used Amazon Linux 2 with arm64 architecture as our base AMI.


![ami](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2ifcn28rcnw5xs8k1qkz.png
)

The custom AMI name is `FastAPI_Base_Image`. It's a clean AMI without any OS/Software dependencies.


![custom_ami_fastapi](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m0j0zw3ju0n5nfxkgjnx.png)


If you are not sure how to create an AMI, follow this link : https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/tkv-create-ami-from-instance.html

### Dockerfile

I will create a container from the Dockerfile which is taking Python 3.9 as the base image and followed with python dependencies installation and starting the uvicorn server.

![dockerfile](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/57ymyzofqzjqmbk1v5ms.png)

The image is already hosted in DockerHub.

URL : https://hub.docker.com/r/mukulmantosh/packerexercise


![dockerhub](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5drmxnd3muyct6ay6wla.png)

We have compiled for three architectures. Thanks to [Docker Buildx](https://docs.docker.com/build/).

- `amd64`
- `arm64`
- `arm/v7`

---

## Packer Template

**packer/build.pkr.hcl**

```
variable "ami_name" {
  type        = string
  description = "The name of the newly created AMI"
  default     = "fastapi-nginx-ami-{{timestamp}}"
}

variable "security_group" {
  type        = string
  description = "SG specific for Packer"
  default     = "sg-064ad8064cf203657"
}

variable "tags" {
  type = map(string)
  default = {
    "Name" : "FastAPI-NGINX-AMI-{{timestamp}}"
    "Environment" : "Production"
    "OS_Version" : "Amazon Linux 2"
    "Release" : "Latest"
    "Creator" : "Packer"
  }
}
source "amazon-ebs" "nginx-server-packer" {
  ami_name          = var.ami_name
  ami_description   = "AWS Instance Image Created by Packer on {{timestamp}}"
  instance_type     = "c6g.medium"
  region            = "ap-south-1"
  security_group_id = var.security_group
  tags              = var.tags

  run_tags        = var.tags
  run_volume_tags = var.tags
  snapshot_tags   = var.tags


  source_ami_filter {
    filters = {
      name                = "FastAPI_Base_Image"
      root-device-type    = "ebs"
      virtualization-type = "hvm"
    }

    most_recent = true
    owners      = ["self"]
  }
  ssh_username = "ec2-user"



}

build {
  sources = [
    "source.amazon-ebs.nginx-server-packer"
  ]

  provisioner "shell" {
    inline = [
      "sudo yum update -y",
    ]
  }

  provisioner "shell" {
    script       = "./scripts/build.sh"
    pause_before = "10s"
    timeout      = "300s"
  }

  provisioner "file" {
    source      = "./scripts/fastapi.conf"
    destination = "/tmp/fastapi.conf"
  }


  provisioner "shell" {
    inline = ["sudo mv /tmp/fastapi.conf /etc/nginx/conf.d/fastapi.conf"]
  }

  error-cleanup-provisioner "shell" {
    inline = ["echo 'update provisioner failed' > packer_log.txt"]
  }

}
```

**[User Variables](https://developer.hashicorp.com/packer/docs/templates/legacy_json_templates/user-variables)**


User variables allow your templates to be further configured with variables from the command-line, environment variables, Vault, or files. This lets you parameterize your templates so that you can keep secret tokens, environment-specific data, and other types of information out of your templates. This maximizes the portability of the template.

**[Builders](https://developer.hashicorp.com/packer/docs/builders)**

Builders create machines and generate images from those machines for various platforms (EC2, GCP, Azure, VMware, VirtualBox) etc. Packer also has some builders that perform helper tasks, like running provisioners.


**[Provisioners](https://developer.hashicorp.com/packer/docs/provisioners)**

Provisioners use built-in and third-party software to install and configure the machine image after booting. Provisioners prepare the system, so you may want to use them for the following use cases:

- installing packages
- patching the kernel
- creating users
- downloading application code


**[Post-Processors
](https://developer.hashicorp.com/packer/docs/post-processors)**


Post-processors run after builders and provisioners. Post-processors are optional, and you can use them to upload artifacts, re-package files, and more.


**[On Error Provisioner](https://developer.hashicorp.com/packer/docs/templates/hcl_templates/blocks/build/provisioner#on-error-provisioner)**

You can optionally create a single specialized provisioner called an error-cleanup-provisioner. This provisioner will not run unless the normal provisioning run fails. If the normal provisioning run does fail, this special error provisioner will run before the instance is shut down. This allows you to make last minute changes and clean up behaviors that Packer may not be able to clean up on its own.


The **amazon-ebs** Packer builder is able to create Amazon AMIs backed by EBS volumes for use in EC2.

```
source "amazon-ebs" 
```

This builder builds an AMI by launching an EC2 instance from a source AMI, provisioning that running machine, and then creating an AMI from that machine. This is all done in your own AWS account. The builder will create temporary keypairs, security group rules, etc. that provide it temporary access to the instance while the image is being created. This simplifies configuration quite a bit.

The builder does not manage AMIs. Once it creates an AMI and stores it in your account, it is up to you to use, delete, etc. the AMI.

To know more, visit this link : https://developer.hashicorp.com/packer/plugins/builders/amazon/ebs



In the “source_ami_filter” section, We are filtering based on the base AMI which we created earlier.

```
  source_ami_filter {
    filters = {
      name                = "FastAPI_Base_Image"
      root-device-type    = "ebs"
      virtualization-type = "hvm"
    }

    most_recent = true
    owners      = ["self"]
  }
```

**most_recent** - Selects the newest created image when true. 
**owners** - You may specify one or more AWS account IDs, "self" (which will use the account whose credentials you are using to run Packer)

We are using a Packer function called **“timestamp”** to generate UNIX timestamp, which helps to get a unique AMI name on every build.

By default the AMI’s you create will be private. If you want to share the AMI’s with other accounts you can make use of the **“ami_users”** option in packer.

If you want to build images in multi-region, you can specify the below code in the source section.

```
  ami_regions   = ["us-west-2", "us-east-1", "eu-central-1"]
```

In the provisioner section we will be updating the OS along-with installing scripts and copy nginx configuration.

**packer/scripts/build.sh**

Installing Docker, NGINX, and pulling latest application image from DockerHub and starting the container.

```
#!/bin/bash
sudo yum install jq -y
sudo yum install -y git

sudo yum install -y docker
sudo usermod -a -G docker ec2-user
sudo systemctl enable docker.service
sudo systemctl start docker.service

sudo amazon-linux-extras install nginx1 -y
sudo systemctl enable nginx.service
sudo systemctl start nginx.service

IMAGE_TAG=`curl -L -s 'https://hub.docker.com/v2/repositories/mukulmantosh/packerexercise/tags'|jq '."results"[0]["name"]' | bc`

sudo docker pull mukulmantosh/packerexercise:$IMAGE_TAG
sudo docker run -d --name fastapi --restart always -p 8080:8080 mukulmantosh/packerexercise:$IMAGE_TAG
```

**packer/scripts/fastapi.conf**

Copy the configuration to NGINX configuration folder. So, NGINX will proxy the request to backend.

```
upstream fastapi {
    server 127.0.0.1:8080;
}
server {

    listen 80;

    location / {
        proxy_pass http://fastapi;
        proxy_set_header X-Forwarded-For 
        $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;
    }

}
```
---

## Building Template

Before you begin to build, make sure you have setup the following keys in your system and **aws-cli** is installed in your machine.

- `AWS_ACCESS_KEY_ID`
- `AWS_SECRET_ACCESS_KEY`


There are two commands which you need to run before you execute build.

`packer fmt build.pkr.hcl`

The packer [fmt](https://developer.hashicorp.com/packer/docs/commands/fmt) Packer command is used to format HCL2 configuration files to a canonical format and style

`packer validate build.pkr.hcl`

The packer [validate](https://developer.hashicorp.com/packer/docs/commands/validate) Packer command is used to validate the syntax and configuration of a template

### Starting the Build

`packer build build.pkr.hcl`

The packer [build](https://developer.hashicorp.com/packer/docs/commands/build) command takes a template and runs all the builds within it in order to generate a set of artifacts. 

![animate_packer_build](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ahmfrd9vqq09emfxo6gy.gif)


You can see the new AMI has been successfully created and tag has been assigned.


![new_ami_1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dgb7oh343tzp6i1xl487.png)


![new_ami_2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4hnqnvf9avtsg0nwoh3t.png)


You must have observed in the packer template, that we are using a custom security group. By default, Packer creates security group which access port 22 (0.0.0.0) from anywhere. 

This posses security risk and to minimize that, I created a custom security group (**Packer_SG**) which allows only My IP.

```
variable "security_group" {
  type        = string
  description = "SG specific for Packer"
  default     = "sg-064ad8064cf203657"
}
```

![custom_sg](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5uway4o6ol8lwawbp143.png)

You can add more security by taking leverage of Session Manager Connections.

**[Session Manager Connections](https://developer.hashicorp.com/packer/plugins/builders/amazon/ebs#session-manager-connections)**
Support for the AWS Systems Manager session manager lets users manage EC2 instances without the need to open inbound ports, or maintain bastion hosts.

---

## GitHub Actions

GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline.


### Self-hosted runners

For our setup we will be using self-hosted Github runners. 

Self-hosted runners offer more control of hardware, operating system, and software tools than GitHub-hosted runners provide. With self-hosted runners, you can create custom hardware configurations that meet your needs with processing power or memory to run larger jobs, install software available on your local network, and choose an operating system not offered by GitHub-hosted runners. Self-hosted runners can be physical, virtual, in a container, on-premises, or in a cloud.

Don't know how to setup ? Follow the below link :

- [Deploying Self-hosted Runners for GitHub Actions](https://www.youtube.com/watch?v=G6nBM3NxBDc)
- [About self-hosted runners](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners)


As from security standpoint, we will make sure **"Packer_SG"** security group allow inbound port 22 for Github Action IP.


![allow_sg_gh_action_1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uuozxdxrapd0ha1ny41n.png)


![allow_sg_gh_action_2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bv3wf9dtvlpqu9601spy.png)


![allow_sg_gh_action_3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rxy6ao73x0widttsil98.png)


### Execute Pipeline

Before proceeding, make sure to create the secrets which will be required in the build process.

`AWS_ACCESS_KEY_ID`
`AWS_SECRET_ACCESS_KEY`


![action_secrets](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vcrteutc7hq2oo5pr169.png)

**.github/workflows/build-packer.yml**

```
name: Packer

on:
  push:
    branches: main

jobs:
  packer:
    runs-on: self-hosted
    name: packer

    steps:
      - name: Checkout Repository
        uses: actions/checkout@v2

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1-node16
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ap-south-1


      # validate templates
      - name: Validate Template
        uses: hashicorp/packer-github-actions@master
        with:
          command: validate
          arguments: -syntax-only
          target: build.pkr.hcl
          working_directory: ./packer


      # build artifact
      - name: Build Artifact
        uses: hashicorp/packer-github-actions@master
        with:
          command: build
          arguments: "-color=false -on-error=abort"
          target: build.pkr.hcl
          working_directory: ./packer


``` 

On inspecting the YAML file, you can clearly observe that we will be validating packer templates and then followed by building the artifact.

Let me make a small change in main branch. So, the pipeline will get triggered.


![packer_github_action_flow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7gc791kcxld37dvnlqg0.gif)

You can see now, the new AMI is created.


![packer_github_action_ami](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rp0knxpwzmmuo2u7n6cv.png)

---

## Vault


![hashicorp_vault](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rhppf5n5eycmxtiuk6s9.png)

**HashiCorp Vault** tightly controls access to secrets and encryption keys by authenticating against trusted sources of identity such as Active Directory, LDAP, Kubernetes, Cloud Foundry, and cloud platforms. Vault enables fine grained authorization of which users and applications are permitted access to secrets and keys.

To know more about Vault, visit this link : 

- https://developer.hashicorp.com/vault


The reason we are using Vault over here is to create dynamic user credentials.  

This helps us to avoid setting up environment variables for 

`export AWS_ACCESS_KEY_ID="XXXXXXXXXXXXXXX"`
`export AWS_SECRET_ACCESS_KEY="XXXXXXXXXXXXXXX"`

Putting this keys in local machine, might expose some risks. So, I would recommend trying out **[AWS Secrets Engine](https://developer.hashicorp.com/vault/docs/secrets/aws)**.

### AWS Secrets Engine

The AWS secrets engine generates AWS access credentials dynamically based on IAM policies. This generally makes working with AWS IAM easier, since it does not involve clicking in the web UI. Additionally, the process is codified and mapped to internal auth methods (such as LDAP). The AWS IAM credentials are **time-based** and are **automatically revoked** when the Vault lease expires.

I have already setup Vault in my local machine.

Follow the below link for setting up Vault.

- https://developer.hashicorp.com/vault/downloads

You can either setup in your local machine or use [HashiCorp Cloud](https://cloud.hashicorp.com/).


Let's now begin by enabling the AWS secrets engine in our Vault server which is running locally.


![create_vault_aws_1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ocbq7dmxiivvboh60y9n.png)


![create_vault_aws_2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lv5uf6196iyda1u0v505.png)

![create_vault_aws_3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wtt4yu2bu0hwbc6niwsw.png)

Now, click on **Configuration** to setup our credentials.


![vault_configure](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/91u0v6s9xtrqlzs47km2.png)

Provide the AWS credentials and region which will be used to create user and attach role to them. 

![create_vault_aws_4](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vqxz41zk0ofobb637mbs.png)


Next, I will modify lease time to 15 minutes. So, once the user is created it will be deleted automatically after 15 minutes.


![create_vault_aws_5](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bltuc5jce5382r80z39b.png)

Click on Save.

I have configured the AWS credential. Now, I will create the role which is going to be attached to the new user.


![create_vault_aws_6](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2ibcvnp0v3of5u3j93fb.png)


**Policy Document**

```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:AttachVolume",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:CopyImage",
        "ec2:CreateImage",
        "ec2:CreateKeypair",
        "ec2:CreateSecurityGroup",
        "ec2:CreateSnapshot",
        "ec2:CreateTags",
        "ec2:CreateVolume",
        "ec2:DeleteKeyPair",
        "ec2:DeleteSecurityGroup",
        "ec2:DeleteSnapshot",
        "ec2:DeleteVolume",
        "ec2:DeregisterImage",
        "ec2:DescribeImageAttribute",
        "ec2:DescribeImages",
        "ec2:DescribeInstances",
        "ec2:DescribeInstanceStatus",
        "ec2:DescribeRegions",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSnapshots",
        "ec2:DescribeSubnets",
        "ec2:DescribeTags",
        "ec2:DescribeVolumes",
        "ec2:DetachVolume",
        "ec2:GetPasswordData",
        "ec2:ModifyImageAttribute",
        "ec2:ModifyInstanceAttribute",
        "ec2:ModifySnapshotAttribute",
        "ec2:RegisterImage",
        "ec2:RunInstances",
        "ec2:StopInstances",
        "ec2:TerminateInstances"
      ],
      "Resource": "*"
    }
  ]
}

```

I would recommend follow defense in depth and principle of least privilege.

Most of them don't encourage that policy document should contain delete permissions.

I came across an interesting article for tightening your policy document and make it more secure. So, it won't interfere with other instances.

Please checkout the below link :

- **[Towards a more Restricted Set of AWS IAM Permissions for Packer](https://blog.stefan-koch.name/2021/05/16/restricted-packer-aws-permissions)** 


![create_vault_aws_7](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vv079i4j6urp73n4v7z3.png)

Now, I will click on **Generate Credentials**.


![create_vault_aws_8](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1gpmlyh9q1si6jn9feta.png)

Now, it's going to create a IAM user which is valid for 15 minutes (900 seconds)

You can see below, the new user is appearing in the IAM User section.


![create_vault_aws_9](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wk2iw70pesw3yukkcqmo.png)

The **PackerRole** with assigned permissions are also being reflected.

![create_vault_aws_10](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ij8n8tbd095q3epu4ts5.png) 

Now, we are going to make sure that Packer should generate this credentials automatically.

Let's begin by editing the **build.pkr.hcl** file.

You need to add this line before closing of the source block.

```
  vault_aws_engine {
    name = "PackerRole"
  }
```


![vault_aws_engine_packer_template](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bt6y1ayy6gbf0v9fgjr8.png)


Next, you need to setup environment variables.

Windows :

 `set VAULT_ADDR=http://127.0.0.1:8200` 
 `set VAULT_TOKEN=XXXXXXXXXXXXXXXXXXXXX`

Linux : 

 `export VAULT_ADDR=http://127.0.0.1:8200` 
 `export VAULT_TOKEN=XXXXXXXXXXXXXXXXXXXXX`


Once, we are done setting up our environment variables. We need to validate everything is working as expected by running the `validate` command.

```
packer validate build.pkr.hcl
```

If you receive this message `The configuration is valid` then you are good to proceed.

To initiate the build run the below command : 

```
packer build build.pkr.hcl
```

* Note : Make sure before your begin build. The security group **Packer_SG** allows inbound access to port 22 from MyIP, as you are running the build from local machine.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m6d831f6lnu1ffzhaxyb.png)

Observe the message : `You're using Vault-generated AWS credentials`

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3hawzgzphoxjyukkpck2.png)

This is going to pick the credentials from Vault, which is going to dynamically create a new user and attach the role.

The user will get automatically deleted based on the expiry specified.


![vault_iam_user_expire](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n3v61koxqulu9sd9sd44.png)


Once, the build is complete. You will find the new image appearing in the AMI section.


![ami_images_list](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w9ws0qftkfp0p4r0pysg.png)


## Final Destination

Congratulations !!! You did it 🏆🏆🏆

![conclusion](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uflct952hlxfsktlsk3m.jpg)

If you liked this tutorial 😊, make sure to share across your friends and colleagues. 

#### References

- [Build a Golden Image Pipeline with HCP Packer
 ](https://developer.hashicorp.com/packer/tutorials/cloud-production/golden-image-with-hcp-packer)
- [How to Set Up Continuous Golden AMI Vulnerability Assessments with Amazon Inspector](https://aws.amazon.com/blogs/security/how-to-set-up-continuous-golden-ami-vulnerability-assessments-with-amazon-inspector/)
- [Packer Docs
 ](https://developer.hashicorp.com/packer/docs)