Relaying a Microservice JSON Response to the Client by Unmarshalling Go Structs
Tags: go, api
Published: 2022-08-05 17:47:00

In this post we will transform a JSON response from a microservice by adding metadata like a name, date or a unique identifier and relaying it to the user client. This is part of my journey in building AI SaaS app on iOS for researchers, as an example we will pretend to send data to an AI object identifier for zoo animals.

The past week I have been working on connecting an image analysis microservice to the user frontend. [We can send an image via post request  and retrieve an array of x and y-coordinates as well as a label for the identified object and a confidence parameter](https://sebastianroy.de/sending-images-in-post-request-as-multipart-form-from-go-to-microservice/).

```go
{"success": true,
 "predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"y_min": 682,
			"x_min": 109,
			"y_max": 707,
			"x_max": 186
		},
		{
			"confidence": 0.7,
			"label": "Pinguin",
			"y_min": 782,
			"x_min": 209,
			"y_max": 717,
			"x_max": 196
		}]
}

```

In this post I will talk about transforming the JSON response and adding metadata like a name, date or a unique identifier. You could also convert the data, so instead of forwarding `x_min` and `x_max` coordinates, we could relay X-Min as coordinate and `width` instead of  `x_max`. In order to distinguish on the client if an object was detected by an algorithm or a human we add a new field `method`. This could also be a more complicated object to store the version of the AI algorithm. The relayed response should look like this

```go
{"name": "Zoo Image Analysis",
 "xid": "9m4e2mr0ui3e8a215n4g"
 "predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"x": 682,
			"y": 109,
			"width": 77, 
			"height": 25,
			"method": "AI"
		},
		{
			"confidence": 0.7,
			"label": "Penguin",
			"x": 782,
			"y": 209,
			"width":  -13,
			"height": -65,
			"method": "AI"
		}]
}
```

To do this we will create a `ResponseAdapter` struct that mimics the response structure of the AI microservice as well as a `PredictionAdapter`. Then we create a `Response` and a `Prediction` struct that contains the fields and methods we want to relay to our client.

1. Print unknown JSON data with an empty interface
2. Convert JSON data into a usable struct to access data fields
3. Rewrite into testable functions
4. Implementing custom UnmarshalJSON
5. Conclusion

When retrieving some JSON data from an API, we cannot call those properties like with `data.name`. Converting this data into structs takes some unmarshalling, mapping and interfacing. So let’s play with some JSON handling examples to step by step arrive at our goal, which is making data from an API call usable. 

You can follow these steps using your local tooling in VS Code and go-tools or just use [Go Playground](https://play.golang.com/) without any installation.

## Printing unknown JSON data with an empty interface

To have some sample data, we read some JSON data from an AI microservice as a multiline string into a byte array.

```go
var data = []byte(`{"success": true,
	"predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"y_min": 682,
			"x_min": 109,
			"y_max": 707,
			"x_max": 186
		},
		{
			"confidence": 0.7,
			"label": "Penguin",
			"y_min": 782,
			"x_min": 209,
			"y_max": 717,
			"x_max": 196
		}]
	}`)

```


Next we create a temporary empty interface
```go
var temp map[string]interface{}
```

Then we unmarshall the data into this variable temp. This requires an import of `„encoding/json“`.

```go
json.Unmarshal(data, &temp)
```

Catch any unmarshalling errors

```go
if err != nil {
		fmt.Println(err)
}
```

Then we can print out temp.

```go
fmt.Println(temp)
```

The result should be

```go
map[predictions:[map[confidence:0.69146144 label:Lion x_max:186 x_min:109 y_max:707 y_min:682] map[confidence:0.7 label:Penguin x_max:196 x_min:209 y_max:717 y_min:782]] success:true]
```

Great. If we want to access just the title 

```go
fmt.Println(temp.predictions[0])
```

we get an error.

```go
./prog.go:40:19: temp.predictions undefined (type map[string]interface{} has no field or method predictions)
```

Ok, let’s take care about this next. The listing for this section.

```go
package main

import (
	"encoding/json"
	"fmt"
)

func main() {

	var data = []byte(`{"success": true,
	"predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"y_min": 682,
			"x_min": 109,
			"y_max": 707,
			"x_max": 186
		},
		{
			"confidence": 0.7,
			"label": "Penguin",
			"y_min": 782,
			"x_min": 209,
			"y_max": 717,
			"x_max": 196
		}]
	}`)

	var temp map[string]interface{}

	//var prediction PredictionsAdapter

	err := json.Unmarshal(data, &temp)

	if err != nil {
		fmt.Println(err)
	}

	//fmt.Println(temp.predictions[0])
}
```


## Accessing data fields 

Assuming we know the expected data structure in advance, lets create a struct. 

```go
type ResponseAdapter struct {
	Success     bool                 `json:"success"`
	Predictions []PredictionsAdapter `json:"predictions"`
}
```

Scroll up to the first JSON response to see where this is coming from. This is mirroring the response we get from the AI microservice. To be exported, the names must start with a capital letter. `[]PredictionsAdapter` means that we expect multiple objects or an array of type PredictionsAdapter.

```go
type PredictionsAdapter struct {
	Confidence float64 `json:"confidence"`
	Label      string  `json:"label"`
	YMin       int     `json:"y_min"`
	XMin       int     `json:"x_min"`
	YMax       int     `json:"y_max"`
	XMax       int     `json:"x_max"`
}
```

We extend the `PredictionsAdapter` with two functions `Width()` and `Height()`, since we need this information later for our client response with `Predictions`.

```go
func (p PredictionsAdapter) Width() int {

	return p.XMax - p.XMin
}

func (p PredictionsAdapter) Height() int {

	return p.YMax - p.YMin
}
```

Instead of unmarshalling it into an empty interface, we use a variable of type `ResponseAdapter`.

```go
var responseAdapter ResponseAdapter

json.Unmarshall(data, &responseAdapter)

fmt.Println(responseAdapter.Predictions[0])
```

The `&` implies that we are referring to the address in memory. And thats how we can access JSON data from go as structs :).

```go
{0.69146144 Lion 682 109 707 186}
```

The full listing

```go
package main

import (
	"encoding/json"
	"fmt"
)

type ResponseAdapter struct {
	Success     bool                 `json:"success"`
	Predictions []PredictionsAdapter `json:"predictions"`
}

type PredictionsAdapter struct {
	Confidence float64 `json:"confidence"`
	Label      string  `json:"label"`
	YMin       int     `json:"y_min"`
	XMin       int     `json:"x_min"`
	YMax       int     `json:"y_max"`
	XMax       int     `json:"x_max"`
}

func (p PredictionsAdapter) Width() int {

	return p.XMax - p.XMin
}

func (p PredictionsAdapter) Height() int {

	return p.YMax - p.YMin
}

func main() {

	var data = []byte(`{"success": true,
	"predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"y_min": 682,
			"x_min": 109,
			"y_max": 707,
			"x_max": 186
		},
		{
			"confidence": 0.7,
			"label": "Penguin",
			"y_min": 782,
			"x_min": 209,
			"y_max": 717,
			"x_max": 196
		}]
	}`)

	//var temp map[string]interface{}

	var responseAdapter ResponseAdapter

	err := json.Unmarshal(data, &responseAdapter)

	if err != nil {
		fmt.Println(err)
	}

	fmt.Println(responseAdapter.Predictions[0])
}
```
## Testing

I find it always motivating to write something that works and see some result first. To adapt good practices let’s now rewrite everything into a testable function and get rid of `main()`. If you use Go Playground, it will automatically run the test functions in absence of `main()`.  

First import `"testing"` and `"github.com/google/go-cmp/cmp"`. It is a library to compare expectations `want` with actual result `got` of a function. It checks if both are equal with	`if !cmp.Equal(want, got)` and returns an error with the differences otherwise.

The declaration of `data`we plug out of the functional context such that it is accessible from each testing function.

Most of what has been happening in `main`we will rename to `ParseImageAnalysisResponse` which will receive the JSON content of type `[]byte` and return a `ResponseAdapter` and an `error`. 

The test function has the same name with prefix `TestXxx` and a reference to `*testing.T` which enables automated testing in go with `go test`. The first line then enables parallel test execution with `t.Parallel()`.

Then we declare our expected result. This time we don’t use the JSON format, since we expect data as a go struct. Thats why we state the type before opening curly parenthesises `want := ResponseAdapter{... .}`

```go
want := ResponseAdapter{
		Success: true,
		Predictions: []PredictionsAdapter{
			{
				Confidence: 0.69146144,
				Label:      "Lion",
				YMin:       682,
				XMin:       109,
				YMax:       707,
				XMax:       186,
			},
			{
				Confidence: 0.7,
				Label:      "Penguin",
				YMin:       782,
				XMin:       209,
				YMax:       717,
				XMax:       196,
			},
		},
}
```


Then we have to call the function with our `data` and catch possible errors. `got` will be of type `ResponseAdapter`, since this is what `ParseImageAnalysisResponseAdapter(…)` returns.

```go
got, err := ParseImageAnalysisResponseAdapter(data)
if err != nil {
		t.Fatal(err)
}
```

And last we compare expectation and result.

```go
if !cmp.Equal(want, got) {
	t.Error(cmp.Diff(want, got))
}
```

If everything is correct we can use Run on Go Playground or `go test` in the command line and the test should pass or return an error.

```go
=== RUN   TestParseImageAnalysisResponse
=== PAUSE TestParseImageAnalysisResponse
=== CONT  TestParseImageAnalysisResponse
{0.69146144 Lion 682 109 707 186}
--- PASS: TestParseImageAnalysisResponse (0.00s)
PASS
```

Full listing

```go
package main

import (
	"encoding/json"
	"fmt"
	"github.com/google/go-cmp/cmp"
	"testing"
)

type ResponseAdapter struct {
	Success     bool                 `json:"success"`
	Predictions []PredictionsAdapter `json:"predictions"`
}

type PredictionsAdapter struct {
	Confidence float64 `json:"confidence"`
	Label      string  `json:"label"`
	YMin       int     `json:"y_min"`
	XMin       int     `json:"x_min"`
	YMax       int     `json:"y_max"`
	XMax       int     `json:"x_max"`
}

var data = []byte(`{"success": true,
	"predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"y_min": 682,
			"x_min": 109,
			"y_max": 707,
			"x_max": 186
		},
		{
			"confidence": 0.7,
			"label": "Penguin",
			"y_min": 782,
			"x_min": 209,
			"y_max": 717,
			"x_max": 196
		}]
	}`)

func ParseImageAnalysisResponseAdapter(data []byte) (ResponseAdapter, error) {

	var responseAdapter ResponseAdapter

	err := json.Unmarshal(data, &responseAdapter)

	if err != nil {
		fmt.Println(err)
	}

	fmt.Println(responseAdapter.Predictions[0])
	
	return responseAdapter, nil
}

func TestParseImageAnalysisResponseAdapter(t *testing.T) {
	t.Parallel()

	want := ResponseAdapter{
		Success: true,
		Predictions: []PredictionsAdapter{
			{
				Confidence: 0.69146144,
				Label:      "Lion",
				YMin:       682,
				XMin:       109,
				YMax:       707,
				XMax:       186,
			},
			{
				Confidence: 0.7,
				Label:      "Penguin",
				YMin:       782,
				XMin:       209,
				YMax:       717,
				XMax:       196,
			},
		},
	}

	got, err := ParseImageAnalysisResponseAdapter(data)
	if err != nil {
		t.Fatal(err)
	}

	if !cmp.Equal(want, got) {
		t.Error(cmp.Diff(want, got))
	}

}
```

Now we have the data available in such a format that we can actually use it. We could even convert it to JSON and pass it to our user client. But sometimes the implementation logic requires that we add or remove some fields and provide the data in a different format. If that is what makes most sense, we should do it to maintain a well designed data workflow, even if it takes some additional work.

In the next section we will define the types `Response` and `Predictions` in a way we actually need them and transform them from our Adapter types using custom `UnmarshallJSON`. First we will add some custom metadata. Adding or changing fields in `[]PredictionsAdapter` however requires looping through each element in the array.

## Implementing custom UnmarshallJSON 

Let’s create the Test for `ParseImageAnalysisResponse` that we are going to implement next (ignore any errors). Out want will be a Response that contains some metadata like name and Xid and the array of predictions. We use the data to pass it to `ParseImageAnalysisResponse` and than see if our expectation was met.

```go
func TestParseImageAnalysisResponse(t *testing.T) {
	t.Parallel()

	want := Response{
		Name: "Test Image",
		Xid:  "9m4e2mr0ui3e8a215n4g",
		Predictions: []Prediction{
			{
				Confidence: 0.69146144,
				Label:      	"Lion",
				X:       		109,
				Y:       		682,
				Width:      	77,
				Height:     	25,
				Method:     	"AI",
			},
			{
				Confidence:	 	0.7,
				Label:      	"Penguin",
				X:       		209,
				Y:       		782,
				Width:          -13,
				Height:       	-65,
				Method:     	"AI",
			},		
		},
	}

	got, err := ParseImageAnalysisResponse(data)
	if err != nil {
		t.Fatal(err)
	}

	if !cmp.Equal(want, got) {
		t.Error(cmp.Diff(want, got))
	}

}
```


Next we define types that match the JSON API response we want to relay to our client, that could be a web app or a mobile app for Android or iOS. 

```go
type Prediction struct {
	Confidence  float64 `json:"confidence"`
	Label       string  `json:"label"`
	X       	int     `json:"x"`
	Y	        int     `json:"y"`
	Width       int     `json:"width"`
	Height      int     `json:"height"`
	Method      string  `json:"method"` // AI, Manual, Predicted(?)
}
```

The Response will contain many of those predictions, since many objects can be detected in an image. But further more it will include meta data like a `Name` and an `Xid`.  A Xid is a unique identifier that is a related but shorter version of UUID. We might need to change the type later, but for now let’s use `string`.

```go
type Response struct {
	Name        string       `json:"name"`
	Xid         string       `json:"xid"`
	Predictions []Prediction `json:"predictions"`
}
```

Now we will adapt the `ParseImageAnalysisResponse` function. It will still use the `data` of type `[]byte` as an argument, but instead of a `ResponseAdapter` it will now return our new `Response`. Therefore we use a variable `response` of type `Response` and use this when unmarshalling. We then check for errors and return the `response`.

```go
func ParseImageAnalysisResponse(data []byte) (Response, error) {

	var response Response
	// Use custom UnmarshalJSON method to obtain Response
	err := json.Unmarshal(data, &response)

	if err != nil {
		fmt.Println(err)
	}

	return response, nil

}
```

Since there is no easy 1-to-1 mapping from the `ResponseAdapter` to the `Response`, we need to do some manual work. Instead of the default `UnmarshalJSON` which we don’t have to specify, we will now use a customized version of it that explicitly states how we can derive our `Response` to the client from the data in the `ResponseAdapter`.
The `data` we send to `ParseImageAnalysisResponse` is the one we want to unmarshall. We know that we can unmarshall it into a `ResponseAdapter`, so thats what we will do first, but not into the pointer of the response object `r`, but into a temporary variable `tmp`. 

```go
var tmp ResponseAdapter
err := json.Unmarshal(data, &tmp)

if err != nil {
	fmt.Println(err)
}
```

This variable also has the Predictions in form of a `PredictionsAdapter`. We now cast them into a new Predictions-Array. 

```go
var p []Prediction
```

We do this by looping though each element `PredictionsAdapter`

```go
for _, element := range tmp.Predictions {
```

and create a temporary corresponding predictions element. 
This predictions element also defines the data for any added field or it could also leave out fields from `PredictionsAdapter`. Then we append the Predictions element to the Predictions Array `p`, that was defined before the loop. I did variable creation and appending kind of in one step here and closed the loop.

```go
p = append(p, Prediction{
			Confidence: element.Confidence,
			Method:     "AI",
			Label:      element.Label,
			XMin:       element.XMin,
			XMax:       element.XMax,
			YMin:       element.YMin,
			YMax:       element.YMax,
	})
}
```

Then we set the `Response` fields. That includes metadata like the name and Xid and setting the predictions array p.

```go
r.Name = "Test Image"
r.Xid = "9m4e2mr0ui3e8a215n4g"
r.Predictions = p
```

And last since UnmarshalJSON requires us to return an error, we do that as well.

```go
return err
```

Now lets see how this function looks in its full glory. 

```go
func (r *Response) UnmarshalJSON(data []byte) error {


	var tmp ResponseAdapter
	err := json.Unmarshal(data, &tmp)

	if err != nil {
		fmt.Println(err)
	}

	var p []Prediction
	for _, element := range tmp.Predictions {

		p = append(p, Prediction{
			Confidence: element.Confidence,
			Method:     "AI",
			Label:      element.Label,
			X:	        element.XMin,
			Y:     	    element.YMin,
			Width:      element.Width(),
			Height:     element.Height(),
		})

	}

	r.Name = "Test Image"
	r.Xid = "9m4e2mr0ui3e8a215n4g"
	r.Predictions = p

	return err
}
```



## Conclusion

Great. We can now take an image that was annotated with an AI microservice and make this analysis useful for relaying it the user client by enhancing it with more relevant data for a JSON response. On the networking side how to exactly send JSON responses from a HTTP request to a client was explained . I talked about [how to send POST requests to the AI micrososervice in a previous post](Sending%20Images%20in%20POST%20request%20as%20MultiPart%20Form%20from%20Go%20to%20Microservice).

Last, let’s look at the full listing, which you can find also on [Go Playground](https://play.golang.com/p/vBJPQtDN7pE). The tests could also be written into a separate file of course. 

> Thanks for reading, I hope you found this useful. This has been one of the first five posts on my [new blog](https://sebastianroy.de/), so I appreciate any comments and suggestions.

```go
package main

import (
	"encoding/json"
	"fmt"
	"github.com/google/go-cmp/cmp"
	"testing"
)

type ResponseAdapter struct {
	Success     bool                 `json:"success"`
	Predictions []PredictionsAdapter `json:"predictions"`
}

type PredictionsAdapter struct {
	Confidence float64 `json:"confidence"`
	Label      string  `json:"label"`
	YMin       int     `json:"y_min"`
	XMin       int     `json:"x_min"`
	YMax       int     `json:"y_max"`
	XMax       int     `json:"x_max"`
}


func (p PredictionsAdapter) Width() int {

	return p.XMax - p.XMin
}

func (p PredictionsAdapter) Height() int {

	return p.YMax - p.YMin
}


type Prediction struct {
	Confidence float64 `json:"confidence"`
	Label      string  `json:"label"`
	X          int     `json:"x"`
	Y          int     `json:"y"`
	Width      int     `json:"width"`
	Height     int     `json:"height"`
	Method     string  `json:"method"` // AI, Manual, Predicted(?)
}


type Response struct {
	Name        string       `json:"name"`
	Xid         string       `json:"xid"`
	Predictions []Prediction `json:"predictions"`
}


func (r *Response) UnmarshalJSON(data []byte) error {


	var tmp ResponseAdapter
	err := json.Unmarshal(data, &tmp)

	if err != nil {
		fmt.Println(err)
	}

	var p []Prediction
	for _, element := range tmp.Predictions {

		// fmt.Println("element", element, "at index", index)

		p = append(p, Prediction{
			Confidence: element.Confidence,
			Method:     "AI",
			Label:      element.Label,
			X:          element.XMin,
			Y:          element.YMin,
			Width:      element.Width(),
			Height:     element.Height(),
		})

	}

	r.Name = "Test Image"
	r.Xid = "9m4e2mr0ui3e8a215n4g"
	r.Predictions = p

	return err
}

var data = []byte(`{"success": true,
	"predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"y_min": 682,
			"x_min": 109,
			"y_max": 707,
			"x_max": 186
		},
		{
			"confidence": 0.7,
			"label": "Penguin",
			"y_min": 782,
			"x_min": 209,
			"y_max": 717,
			"x_max": 196
		}]
	}`)

func ParseImageAnalysisResponseAdapter(data []byte) (ResponseAdapter, error) {

	var responseAdapter ResponseAdapter

	err := json.Unmarshal(data, &responseAdapter)

	if err != nil {
		fmt.Println(err)
	}

	fmt.Println(responseAdapter.Predictions[0])
	
	return responseAdapter, nil
}

func ParseImageAnalysisResponse(data []byte) (Response, error) {

	var response Response
	// Use custom UnmarshalJSON method to obtain Response
	err := json.Unmarshal(data, &response)

	if err != nil {
		fmt.Println(err)
	}
	// fmt.Println("Response")
	// fmt.Println(response)

	return response, nil

}


func TestParseImageAnalysisResponseAdapter(t *testing.T) {
	t.Parallel()

	want := ResponseAdapter{
		Success: true,
		Predictions: []PredictionsAdapter{
			{
				Confidence: 0.69146144,
				Label:      "Lion",
				YMin:       682,
				XMin:       109,
				YMax:       707,
				XMax:       186,
			},
			{
				Confidence: 0.7,
				Label:      "Penguin",
				YMin:       782,
				XMin:       209,
				YMax:       717,
				XMax:       196,
			},
		},
	}

	got, err := ParseImageAnalysisResponseAdapter(data)
	if err != nil {
		t.Fatal(err)
	}

	if !cmp.Equal(want, got) {
		t.Error(cmp.Diff(want, got))
	}

}

func TestParseImageAnalysisResponse(t *testing.T) {
	t.Parallel()

	want := Response{
		Name: "Test Image",
		Xid:  "9m4e2mr0ui3e8a215n4g",
		Predictions: []Prediction{
			{
				Confidence: 0.69146144,
				Label:      	"Lion",
				X:       		109,
				Y:       		682,
				Width:      	77,
				Height:     	25,
				Method:     	"AI",
			},
			{
				Confidence:	 0.7,
				Label:      	"Penguin",
				X:       		209,
				Y:       		782,
				Width:          -13,
				Height:       	-65,
				Method:     	"AI",
			},
		},
	}

	got, err := ParseImageAnalysisResponse(data)
	if err != nil {
		t.Fatal(err)
	}

	if !cmp.Equal(want, got) {
		t.Error(cmp.Diff(want, got))
	}

}
```

## Links
[https://bitfieldconsulting.com/golang/map-string-interface](https://bitfieldconsulting.com/golang/map-string-interface)
[https://jhall.io/posts/go-json-tricks-array-as-structs/](https://jhall.io/posts/go-json-tricks-array-as-structs/)
[https://stackoverflow.com/questions/42377989/unmarshal-json-array-of-arrays-in-go](https://stackoverflow.com/questions/42377989/unmarshal-json-array-of-arrays-in-go)