Promise & async/await-like concurrency for API requests in Go
Tags: go, concurrency
Published: 2024-01-16 02:04:18

---
title: Promise & async/await-like concurrency for API requests in Go
published: true
description: How to implement patterns like JavaScript's async/await and Promise.all() for HTTP requests in Golang.
tags: go, concurrency
# cover_image: https://direct_url_to_image.jpg
# Use a ratio of 100:42 for best results.
# published_at: 2024-01-15 20:12 +0000
---

I'm building a trivial Go app that requests data from multiple API endpoints and renders them in a template. The requests are completely independent from each other: neither request relies on data from the other.

I've written a lot of JavaScript over the past 7 years, and JavaScript's promises and async/await make concurrency trivial. **I wanted to see if I could use Go's concurrency primitives to implement JS-inspired async/await patterns in Go.**

_**Note:** This post requires some familiarity with Go's concurrency primitives, and this isn't a recommendation for how to write concurrent Go code._

## JavaScript concurrency

Here's an example of concurrent requests using **promises** and **`async`**/**`await`**:

```js
// Async/await-based concurrency:

async function makeRequests() {
  let postsResponse, tagsResponse;

  try {
    const postsPromise = fetch("https://example.com/api/posts");
    const tagsPromise = fetch("https://example.com/api/tags");

    postsResponse = await postsPromise;
    tagsResponse = await tagsPromise;
  } catch (e) {
    console.error("At least one request failed", e);
  }

  // Do more stuff...
}
```

Here's what it looks like to make concurrent requests using **`promise.all()`**:

```js
// promise.all()-based concurrency:

Promise.all([
  fetch("https://example.com/api/posts"),
  fetch("https://example.com/api/tags"),
]).then(([postsResponse, tagsResponse]) => {
  // Do more stuff...
}).catch((error) => {
  console.error("At least one request failed", error);
});
```

Both of these methods appeal to me because they're simple and require very little code.

## Implementing awaitable HTTP requests in Go

My attempt at implementing a promise & async/await-like API called `async.Get()` for HTTP requests in Go is below.

```go
// main.go

package main

import (
	"app/async"
)

func main() {
	postPromise := async.Get("https://example.com/api/posts")
	tagPromise := async.Get("https://example.com/api/tags")

	postRes, postErr := postPromise.AwaitResponse()
	tagRes, tagErr := tagPromise.AwaitResponse()

	// Do more stuff...
}
```

The following code powers this API.

```go
// async/get.go

package async

import (
	"net/http"
	"sync"
)

type response struct {
	res *http.Response
	err error
}

type ResponsePromise struct {
	channel chan *response
	result  *response
	once    *sync.Once
}

func (r *ResponsePromise) AwaitResponse() (resp *http.Response, err error) {
	r.once.Do(func() {
		r.result = <-r.channel
	})

	return r.result.res, r.result.err
}

func Get(url string) ResponsePromise {
	c := make(chan *response)
	go func(url string) {
		resp, err := http.Get(url)
		c <- &response{resp, err}
	}(url)

	return newResponsePromise(c)
}

func newResponsePromise(c chan *response) ResponsePromise {
	return ResponsePromise{
		channel: c,
		once:    &sync.Once{},
	}
}

```

## How it works

Here's what happens when you call `async.Get(url)`:

1. A channel is created within `Get()` to store the result of the http request.
1. A goroutine asynchronously fires off an `http.Get()` request to the URL provided as an argument to `Get()`. Its return values will be added to a `request` struct when the request completes, and these will be sent the the channel.
1. A `ResponsePromise` struct with a reference to the channel is returned to the caller of `Get()`.
1. The first time that the `AwaitResponse()` method is called on the `ResponsePromise` struct, all other consumers of the `AwaitResponse()` method will be blocked by `sync.Once.Do` until the channel that was created within `Get()` resolves a `result` struct. The `result` will be cached on the `ResultPromise`.

     Caching the `result` on the `ResponsePromise` is wrapped in `sync.Once` to ensure that unsafe concurrent writes to the `ResultPromise` are impossible, and therefore won't crash the app.
1. Once the cached result is stored on the `ResultPromise` struct, it will return both the response and error from the previously called `http.Get()` request.

This implementation has a little bit of indirection, but it has some nice properties. Because the channel isn't returned directly by `async.Get()`, it's impossible for Go to block permanently by waiting for another channel value if the consumer so happens to call `<- returnedChannel` twice. Also, calling `AwaitResponse()` multiple times is idempotent after the first call: the result is cached so no further requests will be made.

One noteworthy part of the implementation is the intermediate `response` struct. This is necessary because Go does not have a tuple type, so either multiple channels or an intermediate struct were necessary to return both the response and error from `http.Get()`.

## Should you do this? Maybe not.

I really like my implementation: I think that the code reads well. But the code hides inefficiency and rigidness.

My code allocates more memory than is strictly necessary. Behind the scenes, each call to the `async.Get()` function allocates a channel and 3 structs (`ResponsePromise`, `response` and `sync.Once`) that the caller doesn't care about: the caller only wants the response and error from each request.

Compare my awaitable HTTP request to a WaitGroup-based implementation:

```go
// Concurrent requests using a WaitGroup:

package main

import (
	"net/http"
	"sync"
)

func main() {
	var pRes, tRes *http.Response
	var pErr, tErr error

	wg := sync.WaitGroup{}
	wg.Add(2)

	go func() {
		defer wg.Done()
		pRes, pErr = http.Get("https://example.blog/api/posts")
	}()

	go func() {
		defer wg.Done()
		tRes, tErr = http.Get("https://example.blog/api/tags")
	}()

	wg.Wait()

	// Do more stuff...
}
```

Compared to my solution, this allocates 2 fewer channels and 6 fewer structs to the heap. It's _slightly_ more verbose, but **much** more flexible, allowing other concurrent processes to happen that might not be HTTP requests.

If you had a database call that you wanted to make concurrently with the awaitable HTTP requests above, you'd need to wrap that call in a similar awaitable structure, or you'd need to do something hacky to make it play nice with the awaitable HTTP requests.

If you instead use a WaitGroup for the HTTP requests, it's trivial to add a concurrent database call using that same WaitGroup:

```go
// Add concurrent database call with imaginary
// `app/db` and `app/models` packages.

func main() {
	var pRes, tRes *http.Response
	var pErr, tErr error

	// NEW!
	var q []models.Quote
	var qErr error

	wg := sync.WaitGroup{}
	wg.Add(3) // CHANGED!

	// NEW!
	go func() {
		defer wg.Done()
		q, qErr = db.GetInspirationalQuotes()
	}()
	
	go func() {
		defer wg.Done()
		pRes, pErr = http.Get("https://example.blog/api/posts")
	}()

	go func() {
		defer wg.Done()
		tRes, tErr = http.Get("https://example.blog/api/tags")
	}()

	wg.Wait()

	// Do more stuff...
}
```

## Closing thoughts

This was fun to play with, and I learned a lot about Go's concurrency primitives in the process. I should mention that there are other Gophers who have played with creating general-purpose libraries for promises ([here](https://github.com/chebyrash/promise) and [here](https://github.com/ulasakdeniz/go-typed-promise)). The Go community's reaction towards these general purpose promise libraries hasn't been particularly positive, with my favorite unhelpful reaction being "[Please, God… No!!](https://www.reddit.com/r/golang/comments/1039czg/comment/j30ma5h/?utm_source=share&utm_medium=web2x&context=3)" 

My biggest criticism of general purpose promise libraries in Go is that they really don't reduce _that much_ boilerplate code compared to WaitGroups. It's always fun to experiment though, and with any luck you may have learned something from this post.

If you enjoyed this article (or hated it!), please hit the like button and leave a comment below. Thanks!