Infinitely scalable GitHub Actions runners on AWS with Cirun
Tags: cirun, github, aws, cicd
Published: 2022-12-29 15:17:30


![Cirun](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/p4js59col2lauidbe62i.png)
Using the features of cloud providers like AWS and GCP, we can create Infinitely scalable GitHub Actions runners on Cloud but why do we need them?
## Understanding The Problem
In the tech world, everyone wants CI/CD. As we all know GitHub Actions is a CI/CD platform used to automate the builds, test, and deployment process. GitHub Actions uses workflows to automate the whole process. We can run the builds and tests on every push and pull request, or even on merging the pull request with workflows. For all this GitHub Actions use its cloud machines but also they have some limitations.

1. GitHub Actions does not provide support for custom images, or customized configurations of hardware. Sometimes users need a custom image to complete their build and run tests on it according to their use case. Also, GitHub Actions provides a fixed configuration of the systems.![GitHub Actions Hardware](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kujjc2yff7ptdxs5hdkq.png)

2. It does not provide users with GPU instances. When working on stuff like MLops that may requires GPU to build. 

3. GitHub Actions provide you with limited execution time for each run in a workflow that is 6 hours. If a job takes more than this time to execute it will automatically terminate. GitHub provides you with 2000 minutes/month and 500 MB of free storage. To use beyond these limits we have to pay some amount. You can see the pricing [here](https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions).

### What if we can automate this whole process on our cloud and have minimum cost?
 
Definitely, we can do this on our cloud using GitHub actions feature self-hosted runners and Cirun. Also, we can reduce the cost by doing **Infinitely scalable GitHub Actions runners on the cloud with Cirun**. 

## About Cirun and Infinitely scalable runners
Cirun provides self-hosted GitHub Actions Runners on your clouds. 
![Flowchart](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xkhuckytdhil450hfrpg.png)
Before understanding the flowchart let's understand:
### How does Infinitely scalable Runners works?
Sometimes we may don't have limit of time and we want the minimum cost of our process. Clouds like AWS and GCP allow us to do so with an option. In the case of AWS, it is **SPOT**. A spot instance allows us to make a machine whenever capacity is available with no time constraints. To automate this whole process, Cirun provide us with a flag named `preemptible`. When we pass value **true**, It creates the cheapest self-hosted VM.  

### From Flowchart
We can see whenever an event is triggered on the repository. GitHub sends a webhook to cirun. Cirun authenticates the cloud using credentials provided by you. It takes Machine configurations and `preemptible` flag defined in `.cirun.yml` from your GitHub repository and sends a request to the cloud for **Spot instance**. Cloud keeps checking for available capacity. Whenever it meets its conditions, the cloud will create a spot instance using `.cirun.yml` file, and then all workflow files defined on the GitHub repo will run on the instance. Besides all of this, Cirun continuously checks whether the workflow is completed and the machine is idle or not? If condition is satisfied then cirun delete's the VM. By doing this we have minimum cost for our desired machine.

This blog explains how to set up Infinitely scalable GitHub Actions runners on AWS with Cirun. For reference, you can check cirun [documentation](https://docs.cirun.io/reference/examples#amazon-web-services).

The steps are almost the same for other cloud providers which provide the scalable option.

1. In Cirun, log in with GitHub
2. Install Cirun on your Repository
3. Toggle on the Repository to activate.
4. Pass your cloud credentials and connect AWS with Cirun
5. Add workflow file and `.cirun.yml` file on your repository
6. In `.cirun.yml` along with machine config also specify the flag `preemptible: true`
7. Now push a trigger in your Repository

It's done! :rocket:

#### Demo

#### In Cirun login with GitHub
![Login with GitHub](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8ulwhwo6a3bkri0q95ug.png)
##### Install Cirun on your Repository
![Install github to repo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/udl92fc7hq7rb4l9ihyl.png)
1. Configure Cirun
![Install](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v3drqeohdc7mgphugowo.png)
2. Give access to repositories
![Access to repo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rrwv5dtkcq38mp28zvzc.png)
3. To activate the repository toggle the button.

![Toggle on the repo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0c27p9mp26l3dorxyskc.png)



##### Adding AWS to Cirun

1. Navigate to IAM user.

![AWS IAM](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rxlv3jx12e7cp4bitl3t.png)


![AWS user console](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5barawn6p3hao5hvbwd8.png)

- Add users
![Adding user](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o895wtt4xdy9atie7wwc.png)

- Add user with credential type programmatic access

![Access key type](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rrn9p7a4qh7chsk8al2r.png)

- Add user with permission **AmazonEC2FullAccess** 
![AWS permission](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/p6xpyskoyi6jn3qh0w93.png)

- Generate Access key ID and Access Secret key

![Paste keys in cirun dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rdxw1sblktci8kd2gu9r.png)

- Paste your Access key Id and Access Secret key in Cirun dashboard

![Paste your keys](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d554282e5nguzsu24wui.png)

### Add Workflow file on your Repository
Create a `aws.yml` file inside `.github/workflows`. In workflow you can put jobs you want. For example paste the yml file given below to print the system info, memory, architecture. For latest file you can visit [GitHub repo](https://github.com/vishal9629/Infinitely-scalable-runners-on-aws).
```yml
name: AWS

on: [push]

jobs:
  build:

    runs-on: [self-hosted, cirun-aws-runner]
    steps:
    - uses: actions/checkout@v3
    - name: System Info
      run: |
        uname -a
    - name: Print Memory
      run: |
        free -m
    - name: Print Architecture
      run: |
        dpkg --print-architecture
```

#### Now add a `.cirun.yml` file in your Repository

In this yml, we define the configuration and flags we want on your clouds (AWS, GCP, Oracle, etc.), instance_type, and machine_image. For Infinitely scalable GitHub Actions runners we must have pass flag `preemptible: true`.
To explore more you can visit cirun [documentation](https://docs.cirun.io/reference/examples#amazon-web-services).

In this blog, we are creating `.cirun.yml` file for the AWS cloud with the infinitely scalable runner. 
```yml
runners:
  - name: "aws-runner"
    cloud: "aws"
    instance_type: "t2.micro"
    machine_image: "ami-05e786af422f8082a"
    preemptible: "true"
    workflow: .github/workflows/aws.yml
    labels:
      - "cirun-aws-runner"
```

### Pushing a trigger

- To trigger an event we must commit something in our repo.


![New trigger](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yjb9rd0fycxocvkdc4ck.png)

- With a new commit, cirun sends a request to the cloud then the cloud makes a spot instance and runs the workflow


![Running workflow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lhy5uktnd1ir40uro48j.png)

- Printed the system info, memory info, system architecture.

![Printed jobs](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mjgxtsn4ottwdyz4wmrv.png)

### Conclusion

GitHub Actions provide us with self-hosted runners for your repository but setting up them is a tedious task. You can automate the set-up process using cirun. Cirun charges some amount for private repositories and you have to pay for what you use on your cloud. Cirun supports all major cloud providers at this time. (AWS, GCP, Digital Ocean, Azure, Openstack, Oracle).
For more information you can visit [cirun.io](https://cirun.io/).

### References 

- [Cirun](https://cirun.io/)
- [Cirun documentation](https://docs.cirun.io/)
- [Repository used for demo](https://github.com/vishal9629/Infinitely-scalable-runners-on-aws)
- [Spot instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html)

If you have any questions and feedback about this blog, feel free to comment or reach out [Cirun.io](https://cirun.io/).