Build a web API client in Go, Part 2: Deserializing the data
Tags: go, codenewbie, webdev, tutorial
Published: 2020-04-20 18:00:24

---
title: Build a web API client in Go, Part 2: Deserializing the data
published: true
description: 
series: Build a web API client in Go
tags: #go, #codenewbie, #webdev, #tutorial
---

In [part 1](https://dev.to/andyhaskell/build-a-web-api-client-in-go-part-1-connecting-to-the-api-5a4o) of this tutorial, we got an API key for the ClimaCell API, used it to connect to ClimaCell in cURL to get back some JSON weather data, and then set up the same thing in Go. Where we left off in [commit 1](https://github.com/andyhaskell/tea-temperature/commit/347257fba1485e7d8a8cbde0cf29d9a2811e2c57), we made an `http.Request` to a URL on the ClimaCell API's hourly forecast endpoint, got back some data in an `http.Response`, and read it into a byte slice.

The next thing we need to do in order to build the Go app the Cambridge Fresh Pond sloths are building, is to take that HTTP response data and convert it to a Go struct that our code can work with, so we need to **deserialize** it.

## 📚 Understanding the API data we're deserializing

The idea of deserialization, is that you're taking data encoded in some format like JSON, XML, CSV, or Gob, and translating it to an object that our code is able to work with.

Luckily, Go gives us a flexible `encoding/json` package for converting JSON data to Go structs. But before we define our struct for weather data, we gotta make the first stop of any web API client design: visiting the API documentation! Very often, the people maintaining a website's API will give you samples of the API request and response data formats, information on how to authenticate with the API, and explanations of the fundamentals to working with the API.

You can find ClimaCell's API reference [here](https://developer.climacell.co/v3/reference). Let's scroll through and look for information on how ClimaCell represents their weather data.

Starting at the [**Data Layers > Weather**](https://developer.climacell.co/v3/reference#data-layers-weather) section, there is a list of all the fields of data about the weather that the API can return. For example there's `temp` for what temperature it is, and `wind_speed` for how fast the wind is blowing. It is worth noting that for any given weather sample, if data for a field is unavailable, that field's value on the weather sample will be null.

Below that in the left bar is links to all the different URLs we can work with on ClimaCell's API, or as they're called in API terminology, **API endpoints**. Check out the [/weather/forecast/hourly endpoint's documentation](https://developer.climacell.co/v3/reference#get-hourly), which is the endpoint we want.

On the right, you can see a sample of a 200 (OK) response's JSON. Write down a few notes about the data format, and then compare what you noticed to my list below.

Some points of interest I spotted about this data format are:
* The response data is formatted as an array of JSON objects; one object for each weather sample in the forecast.
* The fields' names are in `snake_case`.
* For each kind of field you can request, the data is inside a JavaScript object saying the field's `value` in a weather sample, and what `units` of measure that value is in.

Knowing these details will help us design the Go struct we'll be converting our JSON data to. On the observation in the last bullet point, although for this tutorial we only really care about deserializing the `temp` field, we know that every numeric piece of data except the `lat` and `lon` coordinates is inside a "units, and nullable numeric value" object. So if we make a struct to deserialize those objects, like this:

```go
type FloatValue struct {
    Value *float64
    Units string
}
```

Then we could use that struct as the type to fields on our weather sample struct not just for the temperature, but for _any_ numeric fields in a weather sample!

## 📦 Making a struct to deserialize our data to

We have a good understanding of the format of the data we're working with, so let's make a `Weather` struct we can deserialize our data. If you remember from our last tutorial, our forecast response, with only temperature requested, was an array of objects like this one:

```json
  {
     "lon":              -71.146,
     "lat":              42.3826,
     "temp":             { "value": 5, "units": "C" },
     "observation_time": { "value": "2020-04-24T15:00:00.000Z" },
  }
```

`lat` and `lon` are float64's, and as we talked about in the last section, `temp` would be that FloatValue struct we made. And `observation_time` is a timestamp inside a JSON object, so we'd want a similar TimeValue struct. Since every weather sample has to have a timestamp, we'll assume that TimeValue is not nullable.

So in addition to an overall Weather struct, let's make our FloatValue and TimeValue structs. Make a folder under the `tea-temperature` folder titled `climacell`, and add this Go code to `weather.go`:

```go
package climacell

import (
    "time"
)

type FloatValue struct {
    Value *float64
    Units string
}

type NonNullableTimeValue struct { Value time.Time }
```

Sweet, now with FloatValue and TimeValue defined in the climacell package, and the float64 type defined by Go itself, we have types representing every field we requested. So let's go ahead and put them together on a Weather struct in `climacell/weather.go`:

```go
type Weather struct {
    Lat             float64
    Lon             float64
    Temp            *FloatValue
    ObservationTime NonNullableTimeValue
}
```

One thing to note is that we made `Temp` a `*FloatValue`. As we saw with our cURL request, only get back a weather field from the ClimaCell API if you specifically ask for it, so if a user didn't request the temperature in a forecast, for example, temp should not be on the JSON response, and therefore should be nil on the Weather structs for that response.

Let's try out our struct. In `main.go` we already had some bytes we can work with from that HTTP response, so let's try deserializing them! At the top of the file, import encoding/json and `"github.com/{your github name}/tea-temperature/climacell"`, and then in the part right after where we run `ioutil.ReadAll`, add this code to deserialize and work with our data:

```go
    var weatherSamples []climacell.Weather
    if err := json.Unmarshal(responseBytes, &weatherSamples); err != nil {
        log.Fatalf("error deserializing weather data")
    }
    
    for _, w := range weatherSamples {
       if w.Temp != nil && w.Temp.Value != nil {
         log.Printf("The temperature at %s is %f degrees %s\n",
           w.ObservationTime.Value, *w.Temp.Value, w.Temp.Units)
       } else {
         log.Printf("No temperature data available at %s\n",
           w.ObservationTime.Value)
       }
    }
```

Run `go install` and then `tea-temperature`, and the output will look like:

```go
2020/04/19 23:22:47 The temperature at 0001-01-01 00:00:00 +0000 UTC is 6.000000 degrees C
2020/04/19 23:22:47 The temperature at 0001-01-01 00:00:00 +0000 UTC is 6.000000 degrees C
2020/04/19 23:22:47 The temperature at 0001-01-01 00:00:00 +0000 UTC is 5.000000 degrees C
```

Strange, we have the temperature data, but the time field is the zero value for `time.Time`. In the actual cURL response, we did see a timestamp, so it's definitely there. We need just a small tweak on our `Weather` struct to get `observation_time` deserialized correctly. Our current progress is in [commit 2](https://github.com/andyhaskell/tea-temperature/commit/5f527cdb40d1681b3b6965fef85c9cc788a50403).

## 🏷 Adding struct tags for JSON deserialization

The reason why the JSON response's `observation_time` didn't get added to our Weather struct was because `observation_time`, and all other data fields on the response, are in snake_case. This presents a problem because Go's JSON deserialization by default checks for camelCase JSON object fields.

That means if our weather data instead was `{"observationTime": {"value": "2020-04-19T13:00:00.000Z"}}`, Go's JSON deserialization would detect the camelCase observationTime object field and correctly move its data over to our Weather struct's ObservationTime field.

Because ClimaCell uses a `snake_case` data format, though, we do need to tell Go's deserialization to look for an `observation_time` field. We can do that by adding **struct tags**, which are annotations you can add to Go structs to indicate things like which JSON field corresponds to the struct field, or how to represent a given struct field in a database.

Using a JSON struct tag looks like this:

```go
type Weather struct {
    Lat             float64
    Lon             float64
    Temp            *FloatValue
    ObservationTime TimeValue `json:"observation_time"`
}
```

The `json:"observation_time"` is our struct tag, and it tells the `encoding/json` package, "if you see a field titled observation_time when you're deserializing a JSON object to a Weather struct, put its data in the struct's ObservationTime field".

Though when I define an API client, even if we don't need struct tags on every field, I like adding them to all fields we deserialize from JSON so it's really explicit how a Go struct maps to a JSON object. So let's change the structs in `weather.go` to:

```go
type FloatValue struct {
    Value *float64 `json:"value"`
    Units string   `json:"units"`
}

type NonNullableTimeValue struct {
    Value time.Time `json:"value"`
}

type Weather struct {
    Lat             float64              `json:"lat"`
    Lon             float64              `json:"lon"`
    Temp            *FloatValue          `json:"temp"`
    ObservationTime NonNullableTimeValue `json:"observation_time"`
}
```

Now run `go install` again, and when you run `tea-temperature`, your output should look like this:

```go
2020/04/19 23:35:12 The temperature at 2020-04-24 13:00:00 +0000 UTC is 6.000000 degrees C
2020/04/19 23:35:12 The temperature at 2020-04-24 14:00:00 +0000 UTC is 6.000000 degrees C
2020/04/19 23:35:12 The temperature at 2020-04-24 15:00:00 +0000 UTC is 5.000000 degrees C
```

Great work! We deserialized our response data! Now, there's one quick refactor we can do to really streamline JSON deserialization, and that is use a `json.Decoder`! Our progress so far is in [commit 3](https://github.com/andyhaskell/tea-temperature/commit/f7f2fd1facb506f471ee8be1df228cc2b8662289).

## 💍 Using a json.Decoder

Previously, we've been converting our JSON responses to `Weather` structs by first using `ioutil.ReadAll` to convert the response body to a byte slice. Then we converted that byte slice into our slice of weather structs with `json.Unmarshal`.

It would be really convenient if we could do all the deserialization in one step, converting straight from the `io.ReadCloser` response body, to our slice of `Weather` structs, with no middleman.

That's where the `json.Decoder` type comes in! Instead of doing this:

```go
    responseBytes, err := ioutil.ReadAll(res.Body)
    if err != nil {
        log.Fatalf("error reading HTTP response body: %v", err)
    }

    var weatherSamples []climacell.Weather
    if err := json.Unmarshal(weatherBytes, &weatherSamples); err != nil {
        log.Fatalf("error deserializing weather data")
    }
```

You can pass any `io.Reader`, such as our response body, into `json.NewDecoder` to build a json.Decoder. Then, its Decode method handles all the deserialization. So converting our JSON to Weather structs would now look like this:

```go
    var weatherSamples []climacell.Weather
    d := json.NewDecoder(res.Body)
    if err := d.Decode(&weatherSamples); err != nil {
        log.Fatalf("error deserializing weather data")
    }
```

Not only is that shorter, but it's also more memory-efficient, so `json.Decoder.Decode` is a common idiom for deserializing HTTP responses in Go.

Our progress so far is in [commit 4](https://github.com/andyhaskell/tea-temperature/commit/a2cb6b5335a99c2f0941eb8dd060c33749da5c4c).

We looked through the ClimaCell API to learn about the hourly forecast endpoint and its response format, and then we made a Go type to represent ClimaCell weather data. As you can see, most of the work building this struct turned out to be studying API documentation. This is the case for any API you're building a client for. Building a client is all about doing research (both reading docs and sending requests to experiment with endpoints) so that your client's code can correctly handle all the API's details.

Up to this point, we've been sending requests with a Go `net/http` client, so in the next tutorial, we're gonna take our requests and data deserialization, and put it all together in our own `climacell.Client` type. That type will build on top of `http.Client` to make a Go type that's tailored for a streamlined experience for Go developers working with the ClimaCell API.

Until next time, 🦥 STAY SLOTHFUL! 🌺