Setting up CI for microservices in monorepo using GitHub Actions
Tags: devops, github, go, actionshackathon
Published: 2020-08-31 10:09:11

In this article, I'll describe how to set up **Continuous Integration** for an example application consisting of multiple microservices structured as **monorepo** and written in **Go**.

<br>
### **What is monorepo?**
In the context of microservices, structuring an application as a monorepo means having a single repository for all microservices with each microservice residing in its own separate directory. 

<br>
### **Why monorepo?**
To be clear, monorepo isn't a silver bullet for microservices structuring, having its own benefits and downsides. Generally, benefits include *simplicity*, *consistency*, *stability*, and *code reuse*, while downsides are considered to be *tight coupling*, *git repo scalability*, and *lack of code access control*. I will dive more deeply into the benefits and downsides of structuring microservices as monorepo in a separate article. 

<br>
### **Example monorepo**
Let's take a look at an [example application](https://github.com/stackdumper/monorepo-actions-ci) structured as monorepo. This is a simple HTTP API that consists of three routes: `auth`, `users`, and `articles`.

```
monorepo-actions-ci
└── routes
    ├── articles
    │   └── main.go
    ├── auth
    │   └── main.go
    └── users
        └── main.go
```

The important detail is that each route is an independent application with its own entrypoint, `main.go`.

{% collapsible <code>routes/auth/main.go</code> %}
```go
package main

import "github.com/gin-gonic/gin"

func main() {
	router := gin.Default()

	router.GET("/auth", func(c *gin.Context) {
		c.String(200, "auth")
	})

	router.Run()
}
```
{% endcollapsible %}

{% collapsible <code>routes/users/main.go</code> %}
```go
package main

import "github.com/gin-gonic/gin"

func main() {
	router := gin.Default()

	router.GET("/users", func(c *gin.Context) {
		c.String(200, "users")
	})

	router.Run()
}
```
{% endcollapsible %}

{% collapsible <code>routes/articles/main.go</code> %}
```go
package main

import "github.com/gin-gonic/gin"

func main() {
	router := gin.Default()

	router.GET("/articles", func(c *gin.Context) {
		c.String(200, "articles")
	})

	router.Run()
}
```
{% endcollapsible %}

<br>
### **Setting up GitHub Actions**
Let's create a simple workflow for the auth route, `.github/workflows/auth.yaml`.

```yaml
name: "auth"

on:
  push:
    # run the workflow only on changes
    # to the auth route and auth workflow
    paths:
      - "routes/auth/**"
      - ".github/workflows/auth.yaml"

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1

      # run tests for route
      - name: Run tests
        run: |-
          cd routes/auth
          go test ./...
```

This workflow runs `go test` when changes are made to the auth route. Additionally, it runs when changes are made to the workflow file, which is useful for debugging the workflow itself.

<br>
### **Automating workflow generation**
Now, the next thing would be to add the same workflow for `routes/users` and `routes/articles`. However, while in this case just duplicating the workflow seems a reasonable approach, in case of a big application with a complex workflow and a large number of routes this becomes problematic.

Instead, it would be better to automatically generate workflows from a single workflow template. Let's determine which parts of the workflow need to be adjusted when we move from one endpoint to another.

```yaml
name: "{{ROUTE}}"

on:
  push:
    paths:
      - "routes/{{ROUTE}}/**"
      - ".github/workflows/{{ROUTE}}.yaml"

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1

      - name: Run tests
        run: |-
          cd routes/{{ROUTE}}
          go test ./...

```

Looks simple. Let's put this template into `.github/workflow-template.yaml` and create a bash script that generates workflows for all endpoints from this template. Let's put that script into `workflows.sh`:

```bash
# read the workflow template
WORKFLOW_TEMPLATE=$(cat .github/workflow-template.yaml)

# iterate each route in routes directory
for ROUTE in $(ls routes); do
	echo "generating workflow for routes/${ROUTE}"

	# replace template route placeholder with route name
	WORKFLOW=$(echo "${WORKFLOW_TEMPLATE}" | sed "s/{{ROUTE}}/${ROUTE}/g")
	
	# save workflow to .github/workflows/{ROUTE}
	echo "${WORKFLOW}" > .github/workflows/${ROUTE}.yaml
done
```

Don't forget to make the script executable after creating the file:

```bash
chmod +x ./workflows.sh
```

Now let's run the script and see what happens:

```bash
monorepo-actions-ci
❯ ./workflows.sh
generating workflow for routes/articles
generating workflow for routes/auth
generating workflow for routes/users
```

After taking a quick look at `.github/workflows` you'll see that the script automatically generated a workflow for each route present in `/routes`.

{% collapsible tree %}
```bash
├── .github
│   ├── workflow-template.yaml
│   └── workflows
│       ├── articles.yaml
│       ├── auth.yaml
│       └── users.yaml
├── routes
│   ├── articles
│   │   └── main.go
│   ├── auth
│   │   └── main.go
│   └── users
│       └── main.go
└── workflows.sh
```
{% endcollapsible %}

{% collapsible <code>.github/workflow/auth.yaml</code> %}
```yaml
name: "auth"

on:
  push:
    paths:
      - "routes/auth/**"
      - ".github/workflows/auth.yaml"

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1

      - name: Run tests
        run: |-
          cd routes/auth
          go test ./...
```
{% endcollapsible %}

{% collapsible <code>.github/workflow/users.yaml</code> %}
```yaml
name: "users"

on:
  push:
    paths:
      - "routes/users/**"
      - ".github/workflows/users.yaml"

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1

      - name: Run tests
        run: |-
          cd routes/users
          go test ./...

```
{% endcollapsible %}

{% collapsible <code>.github/workflow/articles.yaml</code> %}
```yaml
name: "articles"

on:
  push:
    paths:
      - "routes/articles/**"
      - ".github/workflows/articles.yaml"

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1

      - name: Run tests
        run: |-
          cd routes/articles
          go test ./...
```
{% endcollapsible %}

### **Does it work?**
Let's push our code to GitHub repository and see what happens.

<sub>Workflows are starting:</sub>
![Workflows are starting running](https://dev-to-uploads.s3.amazonaws.com/i/srwie9q8b3h65lz1d9td.png)

<sub>Workflows have finished running successfully:</sub>
![Workflows have finished running successfully](https://dev-to-uploads.s3.amazonaws.com/i/wluirz2il4up8qbtwofm.png)

<sub>Auth workflow details:</sub>
![Workflow steps](https://dev-to-uploads.s3.amazonaws.com/i/kmlzqx7axju1wna25zo6.png)

<br>
Each of the workflows has successfully finished running. Success! :tada:

---

I hope this article will come handy to you one day. Let me know your thoughts in the comments below or on [Twitter](https://twitter.com/stackdumper). 🖖