Using NX Distributed Execution In Github Actions CI
Tags: nx, github, ci
Published: 2022-01-07 18:28:36

<sub><sup>Photo by <a href="https://unsplash.com/@ruslanbardash?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Ruslan Bardash</a> on <a href="https://unsplash.com/s/photos/vietnam?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a></sub></sup>
  

## Scenario

In building and testing a monorepo in CI, there are problems regarding splitting the tasks in order to lower the time of the CI run.

## NX Solution

[Nx](https://nx.app/) has a feature - [distributed task execution](https://nx.app/docs/distributed-execution) to run tasks parallel in CI that helps speed up building and testing a monorepo.

Unlike manually splitting the build and tests into different jobs, in NX distributed execution the tasks are distributed by a automated manager. The user decided the number of workers to give and the manager utilizes them.

## TLDR

### `CI.yaml` - Main Section

```yaml
name: CI
on:
  push:
    branches:
      - main
  pull_request:
  workflow_dispatch:

env:
  NX_CLOUD_DISTRIBUTED_EXECUTION: true
```

### `CI.yaml` - Job - Main

```yaml
jobs:
  main:
    runs-on: ubuntu-latest
    if: ${{ github.event_name != 'pull_request' }}
    steps:
      - uses: actions/checkout@v2
        name: Checkout [main]
        with:
          fetch-depth: 0

      - name: Derive appropriate SHAs for base and head for `nx affected` commands
        uses: nrwl/nx-set-shas@v2

      - uses: pnpm/action-setup@v2.0.1
        with:
          version: 6.22.2

      - uses: actions/setup-node@v2
        with:
          node-version: '17'
          cache: 'pnpm'
      - run: pnpm install
      - run: pnpm exec nx-cloud start-ci-run
      - run: pnpm exec nx affected --target=build --parallel --max-parallel=3
      - run: pnpm exec nx affected --target=lint --parallel --max-parallel=2
      - run: pnpm exec nx affected --target=test --parallel --max-parallel=2

      - name: Clean Agents
        if: ${{ always() }}
        run: pnpm exec nx-cloud stop-all-agents
```

### `CI.yaml` - Job - PR

```yaml
  pr:
    runs-on: ubuntu-latest
    if: ${{ github.event_name == 'pull_request' }}
    steps:
      - uses: actions/checkout@v2
        with:
          ref: ${{ github.event.pull_request.head.ref }}
          fetch-depth: 0

      - name: Derive appropriate SHAs for base and head for `nx affected` commands
        uses: nrwl/nx-set-shas@v2

      - uses: pnpm/action-setup@v2.0.1
        with:
          version: 6.22.2

      - uses: actions/setup-node@v2
        with:
          node-version: '17'
          cache: 'pnpm'

      - run: pnpm install
      - run: pnpm exec nx-cloud start-ci-run
      - run: pnpm exec nx affected --target=build --parallel --max-parallel=3
      - run: pnpm exec nx affected --target=lint --parallel --max-parallel=2
      - run: pnpm exec nx affected --target=test --parallel --max-parallel=2
      - name: Clean Agents
        if: ${{ always() }}
        run: pnpm exec nx-cloud stop-all-agents
```

### `CI.yaml` - Job - Agents

```yaml
  agents:
    runs-on: ubuntu-latest
    name: Agent - General
    timeout-minutes: 15
    strategy:
      matrix:
        agent: [1, 2]
    steps:
      - uses: actions/checkout@v2'

      - uses: pnpm/action-setup@v2.0.1
        with:
          version: 6.22.2

      - uses: actions/setup-node@v2
        with:
          node-version: '17'
          cache: 'pnpm'

      - run: pnpm install
      - name: Start Nx Agent ${{ matrix.agent }}
        run: npx nx-cloud start-agent
```

## Solution Walkthrough

### Activating the Distributed Execution Manager

In order for the NX commands to know this they should use the manager for work dispatching, they have to have the environment variable [NX_CLOUD_DISTRIBUTED_EXECUTION](https://nx.app/docs/distributed-execution) to `true`.

We put this variable environment wide for all jobs and steps.

```yaml
env:
  NX_CLOUD_DISTRIBUTED_EXECUTION: true
```

If we want to disable the usage of distribute tasks for a specific NX command we can set this variable to false temporarily for a specific step for instance.

```yaml
      - name: Publish artifacts
        run: pnpm exec nx affected --target=custom-publish --parallel=2 --commitid=$GITHUB_SHA
        env:
          NX_CLOUD_DISTRIBUTED_EXECUTION: false
```

NX distributed execution can only run commands that are specified as `cachable` in the NX comnfig. Commands that are not specified as `cachable` gives error.

### Checking out Branch 

To be able to determine which packages were changed - meaning needing build and test - on the `main` and `pr` branches, we want to retrieve all the repo commits. Otherwise, [NX Affected](https://nx.dev/using-nx/affected) command will not be able to compare commits regarding packages changes.

We do this by specifying to download all commits in the checkout action.

[fetch-depth: 0](https://www.git-scm.com/docs/git-clone#Documentation/git-clone.txt---depthltdepthgt) - Specify to download all commits

For determine changes of `pr` branches, we want to checkout not the [special branch](https://docs.github.com/en/actions/learn-github-actions/events-that-trigger-workflows#pull_request) Github creates which is the merge between the PR branch and the base it is against, but the PR branch head only.

This is done by specifying `ref: ${{ github.event.pull_request.head.ref }}` 

The [event.pull_request.head.ref](https://docs.github.com/en/rest/reference/pulls#get-a-pull-request) contains the branch name which is the reference to the [head commit](https://git-scm.com/book/en/v2/Git-Internals-Git-References)

The tests and builds themselves will be performed in the agents which will checkout the merged PR commit as usual.

Job - `main`

```yaml
      - uses: actions/checkout@v2
        name: Checkout [main]
        with:
          fetch-depth: 0
```

Job - `pr`

```yaml
      - uses: actions/checkout@v2
        with:
          ref: ${{ github.event.pull_request.head.ref }}
          fetch-depth: 0
``` 

### Helping NX Affect Command Comparison

For PR's we want to compare the head of the PR branch against the base of the PR. 

For `main` branch commits, the desire is to find changes against the latest commit that was succesful. 

NX provide an [action utility](https://github.com/nrwl/nx-set-shas) that calculate those `head` and `base` commits and set those to [environment variables](https://nx.dev/using-nx/affected) - `NX_BASE` and `NX_HEAD` - used by the NX Affected command.

```yaml
      - name: Derive appropriate SHAs for base and head for `nx affected` commands
        uses: nrwl/nx-set-shas@v2
```

### Starting the Distributed Execution Manager

To start the manager daemon, we execute `nx-cloud` utility with `start-ci-run` command.

```yaml
      - run: pnpm exec nx-cloud start-ci-run
```

### Dispatching a Distributed Task

The NX Affect command run without customization.

The parallel/max-parallel does not specify the number of workers/agents but specify the number of processes to run parallel on each agent and on the manager.

```yaml
      - run: pnpm exec nx affected --target=build --parallel --max-parallel=3
```

### Cleanup Agent After Success and Failure

After the job is done or after a build error, we want to signal the agents to shutdown and not keep waiting for future tasks to execute.

This is done by activating `nx-cloud stop-all-agents`.

In order to do this step even on error, the step needs to specify that it will run always and not just on success.

`if: ${{ always() }}` - [Do the step always](https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#example-not-requiring-dependent-jobs-to-be-successful)

```yaml
      - name: Clean Agents
        if: ${{ always() }}
        run: pnpm exec nx-cloud stop-all-agents
```

### Agent Job Configuration

The Agent has one shared configuration. To parallel and multiple the number of agents node, we use the [matrix](https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix) feature of the job. 

It will create parallel worker nodes, each with different `agent` name variable value.

```yaml
    strategy:
      matrix:
        agent: [1, 2]
```

Rest of the agent job configuration is standard. Checking out the repo merged commit, setting up nodejs, and starting up the agent server that listen for the tasks manager task dispatches.

```yaml
      - name: Start Nx Agent ${{ matrix.agent }}
        run: npx nx-cloud start-agent
```

## Conclusion

For monorepo scalable testing and building, checkout [NX](https://nx.dev/getting-started/intro) and [NX Cloud](https://nx.app/docs/add-nx-cloud-to-workspace)

It provide an interesting solution for efficient parallel task execution to speedup monorepos CI workflows.