Building APOD color search part II: Computing in cloud ☁️
Tags: webdev, devops, cloud, githubactions
Published: 2023-03-12 12:25:58

If you're following along, this is the second part in a series about how I built [APOD color search](https://bryce.io/apod-color-search/).

In this post I'll cover how everything was built with the intent to run via a hosted service instead of a local environment/database. Making things cloud-first puts more effort up front but saves a tremendous amount of time when deploying and running remotely.

## Everything via GitHub Actions 🛰️

Since processing images for this project involves iterating over a large dataset, it was clear early on that the amount of computation was going to be immense. This would require processing things over a long period of time and ensuring consistency when there are any errors.

I already had some experience with [GitHub Actions](https://github.com/features/actions) (their native CI/CD framework) and after finding [rust-toolchain](https://github.com/dtolnay/rust-toolchain) I wanted to experiment with whether it'd be feasible to use Actions as a compute service.

Turns out: works great! ✨ The hardware is more than sufficient to do CPU-bound image processing, and the only issue that I encountered related to builds stopping due to hitting my account limit of [50 compute hours/month](https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions). But with the Actions UI I was able to trigger workflows and process batches of APODs one year at a time over the course of a few months.

## Rust in action(s) ⚡️

The Rust-based utility to process images from the [previous article](https://blog.bryce.io/building-apod-color-search-part-i-image-analysis-in-rust) can be invoked via CLI. To be able to do this in an Action, the environment must have the right dependencies installed and appropriate environment variables.

A safe way to handle environment variables is via GitHub's [encrypted secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets) - it requires manually copying values from a local `.env` file used for development but it's better than storing unencrypted values in your repository. For this project, the Rust script fetches from apod-api and saves to a supabase Postgres instance, hence the keys for `SUPABASE_*` and `APOD_API_*`. 

For the environment to have Rust installed, including the `dtolnay/rust-toolchain@stable` toolchain will ensure it can be used in the action:

```yml
name: Process images for month
env:
  SUPABASE_REST_URL: ${{ secrets.SUPABASE_REST_URL }}
  SUPABASE_URL: ${{ secrets.SUPABASE_URL }}
  SUPABASE_PUBLIC_API_KEY: ${{ secrets.SUPABASE_PUBLIC_API_KEY }}
  APOD_API_URL: ${{ secrets.APOD_API_URL }}
  APOD_API_KEY: ${{ secrets.APOD_API_KEY }}
jobs:
  process-images-for-month:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout 🛎️
        uses: actions/checkout@v3

      - name: Install Rust 🦀
        uses: dtolnay/rust-toolchain@stable
        with:
          components: rustfmt, clippy
          toolchain: stable
```

## Dispatching via workflow UI 🛎️

The only remaining piece is to have a way to trigger the workflow to process images. The `on` field is the [entrypoint](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows) for when a workflow runs, and thankfully there's a way to trigger directly from GitHub's UI.

Adding an `on: workflow_dispatch` will provide an entrypoint to [run the workflow via the Action UI](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch) with given inputs. These arguments can then be referenced via `github.event.inputs` in the `run` command:

```yml
...

on: 
  workflow_dispatch:
    inputs:
      year:
        description: 'Year'     
        required: true
      month:
        description: 'Month'     
        required: true

...

jobs:

    ...

    - name: Run 🤖
      run: cargo run -- ${{ github.event.inputs.year }} ${{ github.event.inputs.month }}
```

Once that is pushed to the main branch for the repository, the option to dispatch should be available. Simply navigate to the  workflow under the 'Actions' tab:

![Actions tab](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0uyegetobicj97jhcksr.png)

And on the right side of the table of workflow runs will be a 'Run workflow' option (will inform you that this is because the "workflow has a `workflow_dispatch` event trigger"):

![Run workflow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r1acrpwwcpxru21cq706.png)

Assuming all secrets are in order, triggering a run will kick of a build to process the images! 

Lastly, I added a couple additional jobs to avoid formatting errors or regressions when the workflow runs with new commits:

```yml
...

jobs:

    ...

      - name: Lint 🧹
        run: |
          cargo fmt --all -- --check
          cargo clippy -- -D warnings

      - name: Test 🔨
        run: cargo test
    ...
```

---

This made it super easy to trigger a bunch of workflows in parallel to process images over time. Would highly recommend using them for small projects like this, assuming your requirements aren't too resource-intensive.