Building GitHub Apps with Golang
Tags: go, github, api, tutorial
Published: 2022-01-17 19:11:57


If you're using GitHub as your version control system of choice then GitHub Apps can be incredibly useful for many tasks including building CI/CD, managing repositories, querying statistical data and much more. In this article we will walk through the process of building such an app in Go including setting up the GitHub integration, authenticating with GitHub, listening to webhooks, querying GitHub API and more.

_TL;DR: All the code used in this article is available at <https://github.com/MartinHeinz/go-github-app>._

## Choosing Integration Type

Before we jump into building the app, we first need to decide which type of integration we want to use. GitHub provides 3 options - _Personal Access Tokens_, _GitHub Apps_ and _OAuth Apps_. Each of these 3 have their pros and cons, so here are some basic things to consider:

- _Personal Access Token_ is the simplest form of authentication and is suitable if you only need to authenticate with GitHub as _yourself_. If you need to act on behalf of other users, then this won't be good enough
- _GitHub Apps_ are the preferred way of developing GitHub integrations. They can be installed by individual users as well as whole organizations. They can listen to events from GitHub via webhooks as well as access the API when needed. They're quite powerful, but even if you request all the permissions available, you won't be able to use them to perform all the actions that a user can.
- _OAuth Apps_ use OAuth2 to authenticate with GitHub on behalf of user. This means that they can perform any action that user can. This might seem like the best option, but the permissions don't provide the same granularity as GitHub Apps, and it's also more difficult to set up because of OAuth.

If you're not sure what to choose, then you can also take a look at [diagram in docs](https://docs.github.com/en/developers/apps/getting-started-with-apps/about-apps#determining-which-integration-to-build) which might help you decide. In this article we will use GitHub App as it's very versatile integration and best option for most use cases.

## Setting Up

Before we start writing any code, we need to create and configure the GitHub App integration:

1. As a prerequisite, we need a tunnel which we will use to deliver GitHub webhooks from internet to our locally running application. You will need to install _localtunnel_ tool with `npm install -g localtunnel` and start forwarding to your localhost using `lt --port 8080`.

2. Next we need to go to <https://github.com/settings/apps/new> to configure the integration. Fill the fields as follows:
  - _Homepage URL_: Your _localtunnel_ URL
  - _Webhook URL_: `https://<LOCALTUNNEL_URL>/api/v1/github/payload`
  - _Webhook secret_: any secret you want (and save it)
  - _Repository Permissions_: Contents, Metadata (Read-only)
  - _Subscribe to events_: Push, Release

3. After creating the app, you will be presented with the settings page of the integration. Take note of _App ID_, generate a private key and download it.

4. Next you will also need to install the app to use it with your GitHub account. Go to _Install App_ tab and install it into your account.
   
5. We also need _installation ID_, which we can find by going to _Advanced_ tab and clicking on latest delivery in the list, take a note of installation ID from request payload, it should be located in `{ "installation": { "id": <...>} }`.

If you've got lost somewhere along the way, refer to the guide [GitHub docs](https://docs.github.com/en/developers/apps/getting-started-with-apps/setting-up-your-development-environment-to-create-a-github-app) which shows where you can find each of the values.

With that done, we have the integration configured and all the important values saved. Before we start receiving events and making API requests we need to get the Go server up and running, so let's start coding!

## Building the App

To build the Go application, we will use the template I prepared in <https://github.com/MartinHeinz/go-github-app>. This application is ready to be used as GitHub app and all that's missing in it, are a couple of variables which we saved during setup in previous section. The repository contains convenience script which you can use to populate all the values:

```shell
git clone git@github.com:MartinHeinz/go-github-app.git && cd go-github-app
./configure_project.sh \
    APP_ID="54321" \
    INSTALLATION_ID="987654321" \
    WEBHOOK_SECRET="verysecret" \
    KEY_PATH="./github_key.pem" \
    REGISTRY="ghcr.io/<GITHUB_USERNAME>/go-github-app"
```

The following sections will walk you through the code but if you're inpatient, then the app is good to go. You can use `make build` to build a binary of the application or `make container` to create a containerized version of it.

First part of the code we need to tackle is authentication. It's done using `ghinstallation` package as follows:

```go
func InitGitHubClient() {
	tr := http.DefaultTransport
	itr, err := ghinstallation.NewKeyFromFile(tr, 12345, 123456789, "/config/github-app.pem")

	if err != nil {
		log.Fatal(err)
	}

	config.Config.GitHubClient = github.NewClient(&http.Client{Transport: itr})
}
```


This function, which is invoked from `main.go` during _Gin_ server start-up, takes App ID, Installation ID and private key to create a GitHub client which is then stored in global config in `config.Config.GitHubClient`. We will use this client to talk to the GitHub API later.

Along with the GitHub client, we also need to set up server routes so that we can receive payloads:

```go
func main() {
    // ...
	v1 := r.Group("/api/v1")
	{
		v1.POST("/github/payload", webhooks.ConsumeEvent)
		v1.GET("/github/pullrequests/:owner/:repo", apis.GetPullRequests)
		v1.GET("/github/pullrequests/:owner/:repo/:page", apis.GetPullRequestsPaginated)
	}
	utils.InitGitHubClient()
	r.Run(fmt.Sprintf(":%v", config.Config.ServerPort))
}
```

First of these is the payload path at `http://.../api/v1/github/payload` which we used during GitHub integration setup. This path is associated with `webhooks.ConsumeEvent` function which will receive all the events from GitHub.

For security reasons, the first thing the `webhooks.ConsumeEvent` function does is verify request signature to make sure that GitHub is really the service that generated the event:

```go
func VerifySignature(payload []byte, signature string) bool {
	key := hmac.New(sha256.New, []byte(config.Config.GitHubWebhookSecret))
	key.Write([]byte(string(payload)))
	computedSignature := "sha256=" + hex.EncodeToString(key.Sum(nil))
	log.Printf("computed signature: %s", computedSignature)

	return computedSignature == signature
}

func ConsumeEvent(c *gin.Context) {
	payload, _ := ioutil.ReadAll(c.Request.Body)

	if !VerifySignature(payload, c.GetHeader("X-Hub-Signature-256")) {
		c.AbortWithStatus(http.StatusUnauthorized)
		log.Println("signatures don't match")
	}
	// ...
```

It performs the verification by computing a HMAC digest of payload using webhook secret as a key, which is then compared with the value in `X-Hub-Signature-256` header of a request. If the signatures match then we can proceed to consuming the individual events:

```go
func ConsumeEvent(c *gin.Context) {
    // ...
	event := c.GetHeader("X-GitHub-Event")

	for _, e := range Events {
		if string(e) == event {
			log.Printf("consuming event: %s", e)
			var p EventPayload
			json.Unmarshal(payload, &p)
			if err := Consumers[string(e)](p); err != nil {
				log.Printf("couldn't consume event %s, error: %+v", string(e), err)
				// We're responding to GitHub API, we really just want to say "OK" or "not OK"
				c.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"reason": err})
			}
			log.Printf("consumed event: %s", e)
			c.AbortWithStatus(http.StatusNoContent)
			return
		}
	}
	log.Printf("Unsupported event: %s", event)
	c.AbortWithStatusJSON(http.StatusNotImplemented, gin.H{"reason": "Unsupported event: " + event})
}
```

In the above snippet we extract the event type from `X-GitHub-Event` header and iterate through a list of events that our app supports. In this case those are:

```go
const (
	Install     Event = "installation"
	Ping        Event = "ping"
	Push        Event = "push"
	PullRequest Event = "pull_request"
)

var Events = []Event{
	Install,
	Ping,
	Push,
	PullRequest,
}
```

If the event name matches one of the options we proceed with loading the JSON payload into a `EventPayload` struct, which is defined in [`cmd/app/webhook/models.go`](https://github.com/MartinHeinz/go-github-app/blob/master/cmd/app/webhooks/models.go). It's just a struct generated using <https://mholt.github.io/json-to-go/> with unnecessary fields stripped.

That payload is then sent to function that handles the respective event type, which is one of the following:

```go
var Consumers = map[string]func(EventPayload) error{
	string(Install):     consumeInstallEvent,
	string(Ping):        consumePingEvent,
	string(Push):        consumePushEvent,
	string(PullRequest): consumePullRequestEvent,
}
```

For example for _push_ event one can do something like this:

```go
func consumePushEvent(payload EventPayload) error {
	// Process event ...
	// Insert data into database ...
	log.Printf("Received push from %s, by user %s, on branch %s",
		payload.Repository.FullName,
		payload.Pusher.Name,
		payload.Ref)

	// Enumerating commits
	var commits []string
	for _, commit := range payload.Commits {
		commits = append(commits, commit.ID)
	}
	log.Printf("Pushed commits: %v", commits)

	return nil
}
```

That being in this case - checking the receiving repository and branch and enumerating the commits contained in this single push. This is the place where you could for example insert the data into database or send some notification regarding the event. 

Now we have the code ready, but how do we test it? To do so, we will use the tunnel which you already should have running, assuming you followed the steps in previous sections.

Additionally, we also need to spin up the server, you can do that by running `make container` to build the containerized application, followed by `make run` which will start the container that listens on port `8080`.

Now you can simply push to one of your repositories and you should see a similar output in the server logs:

```shell
[GIN] 2022/01/02 - 14:44:10 | 204 |     696.813µs |   123.82.234.90 | POST     "/api/v1/github/payload"
2022/01/02 14:44:10 Received push from MartinHeinz/some-repo, by user MartinHeinz, on branch refs/heads/master
2022/01/02 14:44:10 Pushed commits: [9024da76ec611e60a8dc833eaa6bca7b005bb029]
2022/01/02 14:44:10 consumed event: push
```

To avoid having to push dummy changes to repositories all the time, you can redeliver payloads from _Advanced_ tab in your GitHub App configuration. On this tab you will find a list of previous requests, just choose one and hit the _Redeliver_ button.

## Making API Calls

GitHub apps are centered around webhooks to which you can subscribe and listen to, but you can also use any of the GitHub REST/GraphQL API endpoints assuming you requested the necessary permissions. Using API rather than push events is useful - for example - when creating files, analyzing bulk data or querying data which cannot be received from webhooks.

For demonstration of how to do so, we will retrieve pull requests of specified repository:

```go
func GetPullRequests(c *gin.Context) {
	owner := c.Param("owner")
	repo := c.Param("repo")
	if pullRequests, resp, err := config.Config.GitHubClient.PullRequests.List(
	    c, owner, repo, &github.PullRequestListOptions{
		State: "open",
	}); err != nil {
		log.Println(err)
		c.AbortWithStatus(resp.StatusCode)
	} else {
		var pullRequestTitles []string
		for _, pr := range pullRequests {
			pullRequestTitles = append(pullRequestTitles, *pr.Title)
		}
		c.JSON(http.StatusOK, gin.H{
			"pull_requests": pullRequestTitles,
		})
	}
}
```

This function takes 2 arguments - `owner` and `repo` - which get passed to `PullRequests.List(...)` function of GitHub client instance. Along with that, we also provide `PullRequestListOptions` struct to specify that we're only interested in pull requests with state set to `open`. We then iterate over returned PRs and accumulate all their titles which we return in response.

The above function resides on `.../api/v1/github/pullrequests/:owner/:repo` path as specified in `main.go` so we can query it like so:

```shell
curl http://localhost:8080/api/v1/github/pullrequests/octocat/hello-world | jq .
```

It might not be ideal to query API as shown above in situations where we expect a lot of data to be returned. In those cases we can utilize _paging_ to avoid hitting rate limits. A function called `GetPullRequestsPaginated` that performs the same task as `GetPullRequests` with addition of `page` argument for specifying page size can be found in [`cmd/app/apis/github.go`](https://github.com/MartinHeinz/go-github-app/blob/master/cmd/app/apis/github.go).

## Writing Tests

So far we've been testing the app with _localtunnel_, which is nice for quick ad-hoc tests against live API, but it doesn't replace proper unit tests. To write unit tests for this app, we need to mock-out the API to avoid being dependent on the external service. To do so, we can use [`go-github-mock`](https://github.com/migueleliasweb/go-github-mock):

```go
func TestGithubGetPullRequests(t *testing.T) {
	expectedTitles := []string{ "PR number one", "PR number three" }
	closedPullRequestTitle := "PR number two"
	mockedHTTPClient := mock.NewMockedHTTPClient(
		mock.WithRequestMatch(
			mock.GetReposPullsByOwnerByRepo,
			[]github.PullRequest{
				{State: github.String("open"), Title: &expectedTitles[0]},
				{State: github.String("closed"), Title: &closedPullRequestTitle},
				{State: github.String("open"), Title: &expectedTitles[1]},
			},
		),
	)
	client := github.NewClient(mockedHTTPClient)
	config.Config.GitHubClient = client

	gin.SetMode(gin.TestMode)
	res := httptest.NewRecorder()
	ctx, _ := gin.CreateTestContext(res)
	ctx.Params = []gin.Param{
		{Key: "owner", Value: "octocat"},
		{Key: "repo", Value: "hello-world"},
	}

	GetPullRequests(ctx)
	body, _ := ioutil.ReadAll(res.Body)

	assert.Equal(t, 200, res.Code)
	assert.Contains(t, string(body), expectedTitles[0])
	assert.NotContains(t, string(body), closedPullRequestTitle)
	assert.Contains(t, string(body), expectedTitles[1])
}
```

This test starts by defining mock client which will be used in place of normal GitHub client. We give it list of pull request which will be returned when `PullRequests.List` is called. We then create test context with arguments that we want to pass to the function under test, and we invoke the function. Finally, we read the response body and assert that only PRs with `open` state were returned. 

For more tests, see the full [source code](https://github.com/MartinHeinz/go-github-app/blob/master/cmd/app/apis/github_test.go) which includes examples of tests for pagination as well as handling of errors coming from GitHub API.

When it comes to testing our webhook methods, we don't need to use a mock client, because we're dealing with basic API requests. Example of such tests including generic API testing setup can be found in [`cmd/app/webhooks/github_test.go`](https://github.com/MartinHeinz/go-github-app/blob/master/cmd/app/webhooks/webhook_test.go).

## Conclusion

In this article I tried to give you a quick tour of both GitHub apps, as well as the [GitHub repository](https://github.com/MartinHeinz/go-github-app/) containing the sample Go GitHub project. In both cases, I didn't cover everything, the Go client package has much more to offer and to see all the actions you can perform with it, I recommend skimming through the [docs index](https://pkg.go.dev/github.com/google/go-github/v41/github#pkg-index) as well as looking at the source code itself where GitHub API links are listed along each function. For example, like the earlier shown `PullRequests.List` [here](https://github.com/google/go-github/blob/master/github/pulls.go#L147).

As for the repository, there are couple more things you might want to take a look at, including Makefile targets, [CI/CD](https://github.com/MartinHeinz/go-github-app/tree/master/.github/workflows) or additional tests. If you have any feedback or suggestions, feel free to create an issue or just star it if it was helpful to you. &#x1F642;