EC2 instance as self-hosted GItHub runners
Tags: githubactions, ec2, selfhostedrunner, githubrunner
Published: 2022-10-11 05:56:39

By default, GitHub offers [hosted](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners) runners to run our workloads inside. For their Linux runners, GitHub offers runners with specs: 2-core CPU, 7 GB of RAM, 14 GB of SSD disk space.

Some of your CI/CD workloads may require more powerful hardware than GitHub-hosted runners could provide. In such case, you can configure an EC2 instance to act as your GitHub runner instead.

For example, you may run a `c5.4xlarge` instance type for some of your workloads, or even a `r5.xlarge` for workloads that process large data sets in-memory.

Let us begin:

## #1. Create a GitHub personal access token 
- Go to your GitHub Settings
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/puped7o6kxqmf5r65dle.png)

- Click the `Developer settings`

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ss0669h84v9h12i9b267.png)
- Click `Personal access token` to create one

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r1hkgb7kqizbbry5wdyl.png)

- Create a new GitHub personal access token with the `repo` scope in the name of `GH_PERSONAL_ACCESS_TOKEN`. The action will use the token for self-hosted runners management in the GitHub account on the repository level.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3citqwmnviw24jlyxfzm.png)

## #2. Create an AMI image

- Create a new [EC2 instance](https://docs.aws.amazon.com/efs/latest/ug/gs-step-one-create-ec2-resources.html) based on a Linux distribution of your choice
- Connect to the instance
   ```sh
   ssh -i <path_to_pemkey> user_name@IP_ADDRESS
   ```
- Install packages and configure the instance according to your workflow requirements
- After that, land on the EC2 console, select your instance, right-click on it to find `Image and templates` , hover to see `Create image`, click it

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/buiy3gqonksd0yabhsxg.png)
- Enter the `Image name`, `Image description`, and check `No reboot`, and at last , hit `Create`
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k00gnvipndhmh2avxaor.png)
-  The AMI should show up in the AMI console 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/83mf1vod1nav7k5h2rhv.png)
- You can safely terminate the instance as it's of no use any more, of course, you could just keep it running if you plan on making any changes to it, however
 
## #3. Create VPC with subnet and security group
- Create a new [VPC and Subnet](https://docs.aws.amazon.com/vpc/latest/userguide/working-with-vpcs.html) or use an existing
- Create a new [Security group](https://docs.aws.amazon.com/cli/latest/userguide/cli-services-ec2-sg.html) for the runners in the VPC and only allow outbound traffic to port 443, for pulling jobs from GitHub. No port opening for inbound traffic is necessay.

## #4. Configure the environment secrets
- The following environment variables must be set for this action 

| Secrets | Description |
| ------- | ----------- |
| `GH_PERSONAL_ACCESS_TOKEN` | Add the GitHub pat token in secret |
| `AWS_REGION` | Add the region in secret |
| `AWS_ACCESS_KEY_ID` | Add the access key in secret |
| `AWS_SECRET_ACCESS_KEY` | Add the secret key in secret |
| `AMI` | Add the EC2 AMI ID in secret |
| `INSTANCE_TYPE` | Provide the instance type like t2.medium, t2.micro |
| `SUBNET` | Add the subnet ID in secret |
| `SECURITY_GROUP` | Add the security group ID in secret |

- Go to your project `repo settings` to add the above Secrets

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bpbvapzu5qh3qckwpfqr.png)

- Expand the `Secrets` drop-down and click `Actions`

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y8041s4x1caywdahqwvo.png)

- In the _Actions secrets_ tab click `New repository secret` 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5fmybdpgemqq2vv6s1ci.png)

- Name your secrets and add their values to be consumed by the action

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3kx83orynk8n3mz16vwr.png)

- A complete listing of the required secrets should look something like

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gjd7ez1jzkxgwqrwmkul.png)


## #5. Configure the GitHub workflow

- Create a new GitHub Action with the below example workflow
- Please don't forget to set up a job for removing the EC2 instance at the end of the workflow execution.
   Otherwise, the EC2 instance won't be removed and continue to run even after the workflow execution is finished.

Now you're ready to go!

### Inputs

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Name&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Required                                   | Description                                                                                                                                                                                                                                                                                                                           |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `mode`                                                                                                                                                                       | Always required.                           | Specify here which mode you want to use: <br> - `start` - to start a new runner; <br> - `stop` - to stop the previously created runner.                                                                                                                                                                                               |
| `GitHub-token`                                                                                                                                                               | Always required.                           | GitHub Personal Access Token with the `repo` scope assigned.                                                                                                                                                                                                                                                                          |
| `ec2-image-id`                                                                                                                                                               | Required if you use the `start` mode.      | EC2 Image Id (AMI). <br><br> The new runner will be launched from this image. <br><br> The action is compatible with Amazon Linux 2 images.                                                                                                                                                                                           |
| `ec2-instance-type`                                                                                                                                                          | Required if you use the `start` mode.      | EC2 Instance Type.                                                                                                                                                                                                                                                                                                                    |
| `subnet-id`                                                                                                                                                                  | Required if you use the `start` mode.      | VPC Subnet Id. <br><br> The subnet should belong to the same VPC as the specified security group.                                                                                                                                                                                                                                     |
| `security-group-id`                                                                                                                                                          | Required if you use the `start` mode.      | EC2 Security Group Id. <br><br> The security group should belong to the same VPC as the specified subnet. <br><br> Only the outbound traffic for port 443 should be allowed. No inbound traffic is required.                                                                                                                          |
| `label`                                                                                                                                                                      | Required if you use the `stop` mode.       | Name of the unique label assigned to the runner. <br><br> The label is provided by the output of the action in the `start` mode. <br><br> The label is used to remove the runner from GitHub when the runner is not needed anymore.                                                                                                   |
| `ec2-instance-id`                                                                                                                                                            | Required if you use the `stop` mode.       | EC2 Instance Id of the created runner. <br><br> The id is provided by the output of the action in the `start` mode. <br><br> The id is used to terminate the EC2 instance when the runner is not needed anymore.                                                                                                                      |
| `iam-role-name`                                                                                                                                                              | Optional. Used only with the `start` mode. | IAM role name to attach to the created EC2 runner. <br><br> This allows the runner to have permission to run additional actions within the AWS account, without having to manage additional GitHub secrets and AWS users. <br><br> Setting this requires additional AWS permissions for the role launching the instance (see above). |
| `aws-resource-tags`                                                                                                                                                          | Optional. Used only with the `start` mode. | Specifies tags to add to the EC2 instance and any attached storage. <br><br> This field is a stringified JSON array of tag objects, each containing a `Key` and `Value` field (see example below). <br><br> Setting this requires additional AWS permissions for the role launching the instance (see above).                         |
| `runner-home-dir`                                                                                                                                                              | Optional. Used only with the `start` mode. | Specifies a directory where pre-installed actions-runner software and scripts are located.<br><br> |

### Outputs

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Name&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Description                                                                                                                                                                                                                               |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `label`                                                                                                                                                                      | Name of the unique label assigned to the runner. <br><br> The label is used in two cases: <br> - to use as the input of `runs-on` property for the following jobs; <br> - to remove the runner from GitHub when it is not needed anymore. |
| `ec2-instance-id`                                                                                                                                                            | EC2 Instance Id of the created runner. <br><br> The id is used to terminate the EC2 instance when the runner is not needed anymore.                                                                                                       |

### Example workflow

`ci.yml`

```yml
name: ci
on: pull_request
jobs:
  start-runner:
    name: Start self-hosted EC2 runner
    runs-on: ubuntu-latest
    outputs:
      label: ${{ steps.start-ec2-runner.outputs.label }}
      ec2-instance-id: ${{ steps.start-ec2-runner.outputs.ec2-instance-id }}
    steps:
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.AWS_REGION }}
      - name: Start EC2 runner
        id: start-ec2-runner
        uses: machulav/ec2-github-runner@v2
        with:
          mode: start
          github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
          ec2-image-id: ${{ secrets.AMI }}
          ec2-instance-type: ${{ secrets.INSTANCE_TYPE }}
          subnet-id: ${{ secrets.SUBNET }}
          security-group-id: ${{ secrets.SECURITY_GROUP }}
          iam-role-name: my-role-name # optional, requires additional permissions
          aws-resource-tags: > # optional, requires additional permissions
            [
              {"Key": "Name", "Value": "ec2-github-runner"},
              {"Key": "GitHubRepository", "Value": "${{ github.repository }}"}
            ]
  do-the-job:
    name: Do the job on the runner
    needs: start-runner # required to start the main job when the runner is ready
    runs-on: ${{ needs.start-runner.outputs.label }} # run the job on the newly created runner
    steps:
      - name: Hello World
        run: echo 'Hello World!'
  stop-runner:
    name: Stop self-hosted EC2 runner
    needs:
      - start-runner # required to get output from the start-runner job
      - do-the-job # required to wait until the main job is done
    runs-on: ubuntu-latest
    if: ${{ always() }} # required to stop the runner even if the error happened in the previous jobs
    steps:
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.AWS_REGION }}
      - name: Stop EC2 runner
        uses: machulav/ec2-github-runner@v2
        with:
          mode: stop
          github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
          label: ${{ needs.start-runner.outputs.label }}
          ec2-instance-id: ${{ needs.start-runner.outputs.ec2-instance-id }}
```
After a successful first workflow execution, you should see this

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s0nwfoke8pz006yhfrs6.png)