Test your Go web apps with httptest
Tags: go, testing, codenewbie, webdev
Published: 2021-07-23 14:50:05

---
title: Test your Go web apps with httptest
published: true
description: 
tags: #go, #testing, #codenewbie, #webdev
series: Intro to automated testing in Go
---

In [part 1 of this series](https://dev.to/salesforceeng/intro-to-automated-testing-in-go-4mjl), we looked at the basics of writing tests in Go with the `testing.T` type, and in [part 2](https://dev.to/salesforceeng/subtesting-skipping-and-cleanup-in-the-go-testing-t-49ea), we looked at how with just the `testing.T` type, you can organize your tests with its `Run`, `Skip`, and `Cleanup` methods. Even with a good grasp of just `testing.T`, you're ready to write professional test coverage in your Go codebase.

But `testing.T` and isn't all that the Go standard library provides for writing Go tests! One of the most popular uses of Go is building the server code for web apps, using its detailed [net/http](https://golang.org/pkg/net/http/) package. So the Go Team provided us with a sweet additional package for testing web apps in addition to the main `testing` package: `net/http/httptest`!

In this tutorial we'll look at:

* 📋 how to test an HTTP handler with httptest
* 💻 how to use httptest to test against a real server

This tutorial is for you if you're interested in Go web development or do webdev already, you're familiar with the basics of Go testing and structs, and you have some familiarity with the HTTP protocol's concepts of requests, responses, headers. If you've used the `net/http` package, that will help you follow along, but if you're new to `net/http`, this tutorial does have an overview of some Go web app concepts.

## 📶 A quick overview of Go HTTP handlers

> If you already are familiar with Go HTTP handlers, feel free to read the code sample and then skip ahead to the next section. If you're new to Go web development or just want a recap, read on!

Let's start with a recap of one of its core concepts: when you get an HTTP request, you process it using a **handler**.

Handlers look something like the `handleSlothfulMessage` function in this code sample (if you're following along, save this to a file titled `server.go`):

```go
package main

import (
    "net/http"
)

func handleSlothfulMessage(w http.ResponseWriter, r *http.Request) {
    w.Header().Add("Content-Type", "application/json")
    w.Write([]byte(`{"message": "Stay slothful!"}`))
}

func appRouter() http.Handler {
    rt := http.NewServeMux()
    rt.HandleFunc("/sloth", handleSlothfulMessage)
    return rt
}

func main() { http.ListenAndServe(":1123", appRouter()) }
```

All Go HTTP handlers take in an implementation of the `ResponseWriter` interface, and a `Request` struct. Those objects have the following responsibilities:

* The `Request` contains the data of the HTTP request that the HTTP client (ex a browser or cURL), sends to your web server, like:
    * The **URL** that was requested, like the path and query parameters.
    * **Request headers**, for example a `User-Agent` header saying whether the request comes from Firefox, Chrome, a command-line client, or a punched card someone delivered to a card reader.
    * The request **body** for POST and PUT requests.
* The `ResponseWriter` is in charge of formulating the HTTP response, so it handles things like:
    * Writing **status codes**, like 200 for a successful request, or the familiar "404 file not found" websites make corny webpages for.
    * Writing **Response headers**, such as the `Content-Type` to say what format our response is in, like HTML or JSON.
    * Writing the response **body**, which can be any response format, like an HTML webpage, a JSON object, or picture of a sloth 🌺!

In our `handleSlothfulMessage` function, we first add a header to indicate that our response is JSON with the line:

```go
w.Header().Add("Content-Type", "application/json")
```

And then we write out the bytes of a JSON message that has a header saying "Stay slothful!" with the line:

```go
w.Write([]byte(`{"message": "Stay slothful!"}`))
```

Note that because we didn't call `w.WriteHeader` to explicitly select a status code for our HTTP response, the response's code will be 200/OK. If we had some kind of error scenario we'd need to handle in one of our web app's handlers, for example issues talking to a database, then in the handler function we might have code like this to give a 500/internal server error response:

```go
if errorScenarioOccurs {
    w.WriteHeader(http.StatusInternalServerError)
    w.Write([]byte(`{"error_message": "description of the error"}`))
    return
}

// rest of HTTP handler
```

In the `appRouter` function, we make a router so requests to our `/sloth` endpoint are handled with the `handleSlothfulMessage` function:

```go
func appRouter() http.Handler {
    rt := http.NewServeMux()
    rt.HandleFunc("/sloth", handleSlothfulMessage)
    return rt
}
```

Finally, in the `main` function, we start an HTTP server on port 1123 with `http.ListenAndServe`, causing all requests to `localhost:1123` to be handled by the HTTP router we made in `appRouter`.

Now if you start this Go program and then go to `localhost:1123/sloth` in your browser or send a request to it via a client like cURL or Postman, you can see that we got back a simple JSON object!

## 📋 Testing your handler with httptest

As you can see, you can start using your HTTP handler in a real web server without a lot of code. When you're running a `net/http` server with `http.ListenAndServe`, Go does the work behind the scenes for you of making `http.Request` and `ResponseWriter` objects when a request comes in from a client like your browser.

But that does raise the question, where do we get a `ResponseWriter` and `Request` in our Go code when we're testing our HTTP handlers inside `go test`?

Luckily, the standard library has convenient code for testing that in the [net/http/httptest](https://golang.org/pkg/net/http/httptest/) package to facilitate writing test coverage for that, with functions and types like:

* The `NewRequest` function for making the `*http.Request`.
* a `ResponseRecorder` type that both implements the `http.ResponseWriter` interface and lets you replay the HTTP response.
* a `Server` type for testing Go code that sends HTTP requests by setting up a **real HTTP server** to send them to.

To see this httptest in action, let's see how we would test `handleSlothfulMessage`. If you're following along, save this code to `server_test.go`:

```go
package main

import (
    "net/http"
    "net/http/httptest"
    "strings"
    "testing"
)

func TestHandleSlothfulMessage(t *testing.T) {
    wr := httptest.NewRecorder()
    req := httptest.NewRequest(http.MethodGet, "/sloth", nil)

    handleSlothfulMessage(wr, req)
    if wr.Code != http.StatusOK {
        t.Errorf("got HTTP status code %d, expected 200", wr.Code)
    }

    if !strings.Contains(wr.Body.String(), "Stay slothful!") {
        t.Errorf(
            `response body "%s" does not contain "Stay slothful!"`,
            wr.Body.String(),
        )
    }
}
```

Run `go test -v`, and you should see a passing test!

Let's take a look at what happened, and how we're using httptest in our code:

First, there's making the ResponseWriter and Request:

```go
    wr := httptest.NewRecorder()
    req := httptest.NewRequest(http.MethodGet, "/sloth", nil)
```

We make our net/http `ResponseWriter` implementation with `NewRecorder`, and a `Request` object pointed at our `/sloth` endpoint using `NewRequest`.

Then, we run our HTTP handler:

```go
handleSlothfulMessage(wr, req)
```

Remember that Go HTTP handler functions are just regular old Go functions that happen to take in specialized `net/http` objects. That means we can run a handler without any actual HTTP server if we have a ResponseWriter and Request to pass into it. So we run our handler by passing `wr` and `req` into a call to our `handleSlothfulMessage` function. Or if we wanted to test our web app's entire router rather than just one endpoint, we could even run `appRouter().ServeHTTP(wr, req)`!

Then, in the next piece of code, we check out the results of running `handleSlothfulMessage`:

```go
    if wr.Code != http.StatusOK {
        t.Errorf("got HTTP status code %d, expected 200", wr.Code)
    }
```

An httptest `ResponseRecorder` implements the `ResponseWriter` interface, but that's not all it gives us! It also has struct fields we can use for examining the response we get back from our HTTP request. One of them is `Code`; we expect our response to be a 200, so we have an assertion comparing our status code to `http.StatusOK`.

Additionally, a `ResponseRecorder` makes it easy to look at the body of our response. It gives us a `bytes.Buffer` field titled `Body` that recorded the bytes of the response body. So we can test that our HTTP response contains the string "Stay slothful!", having our test fail if it does not.

```go
    if !strings.Contains(wr.Body.String(), "Stay slothful!") {
        t.Errorf(
            `response body "%s" does not contain "Stay slothful!"`,
            wr.Body.String(),
        )
    }
```

By the way, this technique also works with POST requests. If we had an endpoint that took in a POST request with an encoded JSON body, then sending the request in the test for that endpoint would look something like this:

```go
var b bytes.Buffer
err := json.NewEncoder(b).Encode(objectToSerializeToJSON)
if err != nil {
    t.Fatal(err)
}

wr := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodPost, "/post-endpoint", b)
handlePostEndpointRequest(wr, req)
```

First we set up a `bytes.Buffer` to use as our POST request's body. This is because a net/http Request's body needs to be an implementation of the [io.Reader interface](https://pkg.go.dev/io#Reader). `bytes.Buffer` conveniently has the `Read` method, so it implements that interface. We then use [json.NewEncoder(b).Encode](https://pkg.go.dev/encoding/json#Encoder.Encode) to convert a Go struct into bytes of JSON that get stored in the buffer.

We make our POST request by passing `MethodPost`, rather than `MethodGet`, into `httptest.NewRequest`. Our `bytes.Buffer` is passed in as the last argument to `NewRequest` as the request body. Finally, just like before, we call our HTTP request handler using our ResponseRecorder and Request.

## 💻 The httptest server

Not only does `httptest` provide us a way to test our handlers with requests and responses, it even provides ways to test your code with a real HTTP server!

A couple scenarios where this is useful are:

* When you're doing functional tests or integration tests for a web app. For example, testing that communication goes as expected between different microservices.
* If you are implementing clients to other web servers, you can define an httptest server giving back a response in order to test that your client can handle both sending the correct request and processing the server's response correctly.

Let's try out the latter of these scenarios, by making a client struct to send an HTTP request to our `/sloth` endpoint, and deserialize the response into a struct.

First, import `fmt` and `encoding/json` (and `net/http` if you're putting the client code in its own file) and then write this code for the client. If you're newer to JSON deserialization, no worries if the code doesn't click 100% for you. The main things you need to know are:

* The client we're making has a `GetSlothfulMessage` that sends an HTTP request to the `/sloth` of its `baseURL`.
* Using Go's awesome `encoding/json` package, the HTTP response body is converted to a `SlothfulMessage` struct, which is returned if the request and JSON deserialization are successful. We are using [json.NewDecoder(res.Body).Decode](https://pkg.go.dev/encoding/json#Decoder.Decode) for reading the response body into our `SlothfulMessage` struct. 
* If we get a non-200 HTTP status code sending the request, or there's a problem deserializing the JSON response, then `GetSlothfulMessage` instead returns an error.

```go
type Client struct {
    httpClient *http.Client
    baseURL string
}

type SlothfulMessage struct {
    Message string `json:"message"`
}

func NewClient(httpClient *http.Client, baseURL string) Client {
    return Client{
        httpClient: httpClient,
        baseURL:    baseURL,
    }
}

func (c *Client) GetSlothfulMessage() (*SlothfulMessage, error) {
    res, err := c.httpClient.Get(c.baseURL + "/sloths")
    if err != nil {
        return nil, err
    }

    if res.StatusCode != http.StatusOK {
        return nil, fmt.Errorf(
            "got status code %d", res.StatusCode,
        )
    }

    var m SlothfulMessage
    if err := json.NewDecoder(res.Body).Decode(&m); err != nil {
        return nil, err
    }
    return &m, nil
}
```

We've got our client, so let's see how we can test it with an `httptest.Server`:

```go
func TestGetSlothfulMessage(t *testing.T) {
    router := http.NewServeMux()
    router.HandleFunc("/sloth", handleSlothfulMessage)

    svr := httptest.NewServer(router)
    defer svr.Close()

    c := NewClient(http.DefaultClient, svr.URL)
    m, err := c.GetSlothfulMessage()
    if err != nil {
        t.Fatalf("error in GetSlothfulMessage: %v", err)
    }
    if m.Message != "Stay slothful!" {
        t.Errorf(
            `message %s should contain string "Sloth"`,
            m.Message,
        )
    }
}
```

Here's what happens:

First, we start our server:

```go
    svr := httptest.NewServer(appRouter())
    defer svr.Close()
```

We pass in our web app's HTTP router as a request handler for the server. When we run `NewServer`, a server is set up to run on your localhost, on a randomized port. In fact, if you had your test run `time.Sleep` to pause for a while, you could actually go to that server in your own browser!

Now that we've got our server, we set up our client and have it test an HTTP roundtrip to our `/sloth` endpoint:

```go
    c := NewClient(http.DefaultClient, svr.URL)
    m, err := c.GetSlothfulMessage()
```

The base URL we give to the `Client`, is the URL of the server, which is the randomized port I mentioned earlier. So a request might go out to somewhere like "localhost:1123/sloths", or "localhost:5813/sloths". It all depends on which port `httptest.NewServer` picks!

Finally, we check that we didn't get an error, and that the response is what we expected. If we run go test -v, we'll get:

```
=== RUN   TestHandleSlothfulMessage
--- PASS: TestHandleSlothfulMessage (0.00s)
=== RUN   TestGetSlothfulMessage
    webapp_test.go:37: error in GetSlothfulMessage: got status code 404
--- FAIL: TestGetSlothfulMessage (0.00s)
```

A failing test, because we got a 404 response, not the 200 we expected. So that means there's a bug in our client.

The part of `GetSlothfulMessage` that was for sending our HTTP request was:

```go
func (c *Client) GetSlothfulMessage() (*SlothfulMessage, error) {
    res, err := c.httpClient.Get(c.baseURL + "/sloths")
    if err != nil {
        return nil, err
    }
```

As you can see, we're sending the request to `c.baseURL + "/sloths"`. We wanted to send it to `/sloth`, not `/sloths`. So fix that code, run `go test -v`, and now...

```
=== RUN   TestHandleSlothfulMessage
--- PASS: TestHandleSlothfulMessage (0.00s)
=== RUN   TestGetSlothfulMessage
--- PASS: TestGetSlothfulMessage (0.00s)
```

Your test should pass!

As you can see, with the `httptest` package's `ResponseRecorder` and `Server` objects, we've got the ability to take the concepts we were already working with for writing tests using the `testing` package, and then start using functionality to test both receiving and sending HTTP requests. Definitely a must-know package in a Go web developer's toolbelt!

> Thank you to my coworker Aaron Taylor for peer-reviewing this blog post!