How to handle type conversions with the DynamoDB Go SDK
Tags: programming, go, database, tutorial
Published: 2022-12-13 03:48:34

*Learn with practical code samples*

`DynamoDB` provides a [rich set of data types](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html) including `String`s, `Number`s, `Set`s, `List`s, `Map`s etc. In the [Go SDK for DynamoDB](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/dynamodb), the
[types](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/dynamodb/types) package contains Go representations of these data types and the [attributevalue](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue) module provides functions to work with Go and `DynamoDB` types. 

This blog post will demonstrate how to handle conversions between Go types in your application and `DynamoDB`. We will start off with simple code snippets to introduce some of the API constructs and wrap up with a example of how to use these Go SDK features in the context of a complete application (including a code walk though).

> You can refer to the complete code on [GitHub](https://github.com/abhirockzz/dynamodb-go-sdk-type-conversion)

To begin with, go through a few examples. 

> Please note that error handling has been purposely omitted in the below code snippets to keep them concise.

### Converting from Go to DynamoDB types

The [Marshal](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue#Marshal) family of functions takes care of this. It works with basic scalars (`int`, `uint`, `float`, `bool`, `string`), `map`s, `slice`s, and `struct`s.

To work with scalar types, just use the (generic) `Marshal` function:

```go
func marshalScalars() {
	av, err := attributevalue.Marshal("foo")
	log.Println(av.(*types.AttributeValueMemberS).Value)

	av, err = attributevalue.Marshal(true)
	log.Println(av.(*types.AttributeValueMemberBOOL).Value)

	av, err = attributevalue.Marshal(42)
	log.Println(av.(*types.AttributeValueMemberN).Value)

	av, err = attributevalue.Marshal(42.42)
	log.Println(av.(*types.AttributeValueMemberN).Value)
}
```

`Marshal` converts a Go data type into a [AttributeValue](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/dynamodb/types#AttributeValue). But `AttributeValue` itself is just an `interface` and requires you to cast it to a concrete type such as [AttributeValueMemberS](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/dynamodb/types#AttributeValueMemberS) (for `string`), [AttributeValueMemberBOOL](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/dynamodb/types#AttributeValueMemberBOOL) (for `boolean`) etc. 

> If you try to cast incompatible types, the SDK responds with a helpful error message. For example, `panic: interface conversion: types.AttributeValue is *types.AttributeValueMemberN, not *types.AttributeValueMemberS`

When working with `slice`s and` map`s, you are better off using specific functions such as [MarshalList](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue#MarshalList) and [MarshalMap](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue#MarshalMap):

```go
func marshalSlicesAndMaps() {
	avl, err := attributevalue.MarshalList([]string{"foo", "bar"})

	for _, v := range avl {
		log.Println(v.(*types.AttributeValueMemberS).Value)
	}

	avm, err := attributevalue.MarshalMap(map[string]interface{}{"foo": "bar", "boo": "42"})

	for k, v := range avm {
		log.Println(k, "=", v.(*types.AttributeValueMemberS).Value)
	}
}
```

The above examples gave you a sense of how to work with simple data types in isolation. In a real world application, you will make use of *composite* data types to represent your *domain* model - most likely they will be in the form of Go `struct`s. So let's look at a few examples of that.

### Working with Go structs

Here is a simple one:

```go
type User struct {
	Name string
	Age  string
}

func marshalStruct() {
	user := User{Name: "foo", Age: "42"}

	av, err := attributevalue.Marshal(user)

	avm := av.(*types.AttributeValueMemberM).Value
	log.Println("name", avm["Name"].(*types.AttributeValueMemberS).Value)
	log.Println("age", avm["Age"].(*types.AttributeValueMemberS).Value)

	avMap, err := attributevalue.MarshalMap(user)

	for name, value := range avMap {
		log.Println(name, "=", value.(*types.AttributeValueMemberS).Value)
	}
}
```

> Notice how convenient it is to use `MarshalMap` (instead of `Marshal`) when dealing Go structs especially if your application does not know all the attribute names.

So far, it seems like we can handle simple use-cases. But we can do better. This example had a homogenous data type i.e. the `struct` had only `string` type making it easy to iterate over the result `map` and cast the value to a `*types.AttributeValueMemberS` - if that were not the case, you would have to iterate over each and every attribute value type and type cast it to the appropriate Go type. This will be evident when working with rest of the `DynamoDB` APIs. For example, the result of a [GetItem](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/dynamodb#Client.GetItem) invocation ([GetItemOutput](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/dynamodb#GetItemOutput)) contains a `map[string]types.AttributeValue`. 

The SDK provides a way for us to make this much easier!

### Converting from DynamoDB to Go types

The `Unmarshal` family of functions takes care of this. Here is another example:

```go
type AdvancedUser struct {
	Name         string
	Age          int
	IsOnline     bool
	Favourites   []string
	Contact      map[string]string
	RegisteredOn time.Time
}

func marshalUnmarshal() {
	user := AdvancedUser{
		Name:         "abhishek",
		Age:          35,
		IsOnline:     false,
		Favourites:   []string{"Lost In Translation, The Walking Dead"},
		Contact:      map[string]string{"mobile": "+919718861200", "email": "abhirockzz@gmail.com"},
		RegisteredOn: time.Now(),
	}

	avMap, err := attributevalue.MarshalMap(user)

	var result AdvancedUser
	err = attributevalue.UnmarshalMap(avMap, &result)

	log.Println("\nname", result.Name, "\nage", result.Age, "\nfavs", result.Favourites)
}
```

With `MarshalMap`, we converted an instance of `AdvancedUser` struct into a `map[string]types.AttributeValue` (imagine you get this as a response to a `GetItem` API call). Now, instead of iterating over individual `AttributeValue`s, we simply use [UnmarshalMap](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue#UnmarshalMap) to convert it back a Go `struct`.


There is more! Utility functions like `UnmarshalListOfMaps` make it convenient to work with multiple `slice`s of Go `struct`s.

```go
func marshalUnmarshalMultiple() {
	user1 := User{Name: "user1", Age: "42"}

	user1Map, err := attributevalue.MarshalMap(user1)
	if err != nil {
		log.Fatal(err)
	}

	user2 := User{Name: "user2", Age: "24"}

	user2Map, err := attributevalue.MarshalMap(user2)
	if err != nil {
		log.Fatal(err)
	}

	var users []User

	err = attributevalue.UnmarshalListOfMaps([]map[string]types.AttributeValue{user1Map, user2Map}, &users)
	if err != nil {
		log.Fatal(err)
	}

	for _, user := range users {
		log.Println("name", user.Name, "age", user.Age)
	}
}
```

### Using struct tags for customization

`Marshal` and `Unmarshal` functions support the `dynamodbav` struct tag to control conversion between Go types and `DynamoDB` `AttributeValue`. Consider the following `struct`:

```go
type User struct {
	Email string `dynamodbav:"email" json:"user_email"`
	Age   int    `dynamodbav:"age,omitempty" json:"age,omitempty"`
	City  string `dynamodbav:"city" json:"city"`
}
```

Couple of common scenarios where the `dynamodbav` comes in handy.

**Customize attribute name**

Say, we have a table with **email** as the partition key. Without the `dynamodbav:"email"` tag, when we marshal the `User` struct and try to save in table, it will use `Email` (upper case) as the attribute name - `DynamoDB` will not accept this since [attribute names are case sensitive](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.NamingRules) - *"All names must be encoded using UTF-8, and are case-sensitive."*

> Notice that we have combined [json tags](https://cs.opensource.google/go/go/+/go1.19:src/encoding/json/encode.go;l=157) as well (this is perfectly valid) - it's not used by `DynamoDB` but the json library while encoding and decoding data

**Handle missing attributes**

`DynamoDB` is a NoSQL database and tables don't have a fixed schema (except for `partition` key and an optional `sort` key). For example, a user item may *not* include the age attribute.By using `dynamodbav:"age,omitempty"`, if the `Age` field is missing, it won't be sent to `DynamoDB` (it will be ignored)

> In the absence of this tag, our `DynamoDB` record will have `Age` attribute set to `0` - depending on your use case this may or may not

To look at all the usage patterns of this struct tag, refer to the [Marshal API documentation](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue#Marshal).

As promised before, let's explore how to put all these APIs to use within the scope of an...

### ... End-to-end example

We will look at a Go application that exposes a REST API with a few endpoints. It combines the CRUD APIs (`PutItem`, `GetItem` etc.) together with all the functions/APIs mentioned above.

**Test the application**

Before we see the code, let's quickly review and test the endpoints exposed by the application. You will need to have Go installed, clone the application and change to the right directory.

```bash
git clone https://github.com/abhirockzz/dynamodb-go-sdk-type-conversion
cd dynamodb-go-sdk-type-conversion
```

First, create a `DynamoDB` table (you can name it **users**). Use **city** as the `Partition` key, **email** as the `Sort` key.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cn7deegqv37tesorixk4.png)

You need some test data. You can do so manually, but I have included a simply utility to seed some test data during application startup. To use it, simply set the `SEED_TEST_DATA` variable at application startup:

```bash
export SEED_TEST_DATA=true

go run main.go

# output
started http server...
```

This will create `100` items. Check `DynamoDB` table to confirm:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/p2v37zrutocef4m8u9h6.png)

Your application should be available at port `8080`. You can use `curl` or any other `HTTP` client to invoke the endpoints:

```bash
# to get all users
curl -i http://localhost:8080/users/

# to get all users in a particular city
curl -i http://localhost:8080/users/London

# to get a specific user
curl -i "http://localhost:8080/user?city=London&email=user11@foo.com"
```

To better understand how the above APIs are used, let's briefly review key parts of the code:

### Code walk through

**Add new item to a DynamoDB table**

Starting with the HTTP handler for adding a `User`:

```go
func (h Handler) CreateUser(rw http.ResponseWriter, req *http.Request) {
	var user model.User

	err := json.NewDecoder(req.Body).Decode(&user)
	if err != nil {// handle error}

	err = h.d.Save(user)
	if err != nil {// handle error}

	err = json.NewEncoder(rw).Encode(user.Email)
	if err != nil {// handle error}
}
```

First, we convert the `JSON` payload into a `User` struct which we then pass to the `Save` function.

```go
func (d DB) Save(user model.User) error {

	item, err := attributevalue.MarshalMap(user)

	if err != nil {// handle error}

	_, err = d.client.PutItem(context.Background(), &dynamodb.PutItemInput{
		TableName: aws.String(d.table),
		Item:      item})

	if err != nil {// handle error}

	return nil
}
```

Notice how `MarshalMap` is used to convert the `User` struct to a `map[string]types.AttributeValue` that the `PutItem` API can accept:

**Get a single item from DynamoDB**

Since our table has a **composite** primary key (`city` is the `partition` key and `email` is the `sort` key), we will need to provide **both** of them to locate a specific user item:

```go
func (h Handler) FetchUser(rw http.ResponseWriter, req *http.Request) {

	email := req.URL.Query().Get("email")
	city := req.URL.Query().Get("city")

	log.Println("getting user with email", email, "in city", city)

	user, err := h.d.GetOne(email, city)
	if err != nil {// handle error}


	err = json.NewEncoder(rw).Encode(user)
	if err != nil {// handle error}
}
```

We extract the `email` and `city` from the query parameters in the `HTTP` request and pass it on to the database layer (`GetOne` function).

```go
func (d DB) GetOne(email, city string) (model.User, error) {

	result, err := d.client.GetItem(context.Background(),
		&dynamodb.GetItemInput{
			TableName: aws.String(d.table),
			Key: map[string]types.AttributeValue{
				"email": &types.AttributeValueMemberS{Value: email},
				"city":  &types.AttributeValueMemberS{Value: city}},
		})

	if err != nil {// handle error}

	if result.Item == nil {
		return model.User{}, ErrNotFound
	}

	var user model.User

	err = attributevalue.UnmarshalMap(result.Item, &user)
	if err != nil {// handle error}

	return user, nil
}
```

We invoke `GetItem` API and get back the result in form of a `map[string]types.AttributeValue` (via the `Item` attribute in `GetItemOutput`).
This is converted back into the Go (`User`) struct using `UnmarshalMap`.

> Notice that the `Key` attribute in `GetItemInput` also accepts a `map[string]types.AttributeValue`, but we don't use `MarshalMap` to create it

**Fetch multiple items**

We can choose to query for all users in a *specific* city - this is a perfectly valid access pattern since `city` is the `partition` key.

The HTTP handler function accepts the city as a path parameter, which is passed on to the database layer.

```go
func (h Handler) FetchUsers(rw http.ResponseWriter, req *http.Request) {
	city := mux.Vars(req)["city"]
	log.Println("city", city)

	log.Println("getting users in city", city)

	users, err := h.d.GetMany(city)

	if err != nil {
		http.Error(rw, err.Error(), http.StatusInternalServerError)
		return
	}

	err = json.NewEncoder(rw).Encode(users)
	if err != nil {
		http.Error(rw, err.Error(), http.StatusInternalServerError)
		return
	}
}
```

From there on, `GetMany` function does all the work:

```go
func (d DB) GetMany(city string) ([]model.User, error) {

	kcb := expression.Key("city").Equal(expression.Value(city))
	kce, _ := expression.NewBuilder().WithKeyCondition(kcb).Build()

	result, err := d.client.Query(context.Background(), &dynamodb.QueryInput{
		TableName:                 aws.String(d.table),
		KeyConditionExpression:    kce.KeyCondition(),
		ExpressionAttributeNames:  kce.Names(),
		ExpressionAttributeValues: kce.Values(),
	})

	if err != nil {
		log.Println("Query failed with error", err)
		return []model.User{}, err
	}

	users := []model.User{}

	if len(result.Items) == 0 {
		return users, nil
	}

	err = attributevalue.UnmarshalListOfMaps(result.Items, &users)
	if err != nil {
		log.Println("UnmarshalMap failed with error", err)
		return []model.User{}, err
	}

	return users, nil
}
```

Pay attention to two things:

- How a `KeyConditionExpression` is being used (this is from the `expressions` package)
- And more interestingly, the usage of `UnmarshalListOfMaps` function to directly convert a `[]map[string]types.AttributeValue` (`slice` of items from DynamoDB) into a `slice` of `User` struct. If not for this function, we would need to extract each item from the result i.e. a `map[string]types.AttributeValue` and call `UnmarshalMap` for each of them. So this is pretty handy!

**Finally - just get everything!**

The `GetAll` function uses `Scan` operation to retrieve **all** the records in the `DynamoDB` table.

> A `Scan` operation goes over the entire table (or secondary index) and it's highly likely that it will end up consuming a large chunk of the provisioned throughput, especially if it's a large table. It should be your last resort - check whether [Query API](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html) (or [BatchGetItem](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchGetItem.html)) works for your use case. 

```go
func (d DB) GetAll() ([]model.User, error) {

	result, err := d.client.Scan(context.Background(), &dynamodb.ScanInput{
		TableName: aws.String(d.table),
	})

	if err != nil {
		log.Println("Scan failed with error", err)
		return []model.User{}, err
	}

	users := []model.User{}

	err = attributevalue.UnmarshalListOfMaps(result.Items, &users)

	if err != nil {
		log.Println("UnmarshalMap failed with error", err)
		return []model.User{}, err
	}

	return users, nil
}
```

## Wrap up

I hope you found this useful and now you are aware of the APIs in `DynamoDB` Go SDK to work with simple Go types as well as `struct`s, `map`s, `slice`s etc. I would encourage you to explore some of the other nuances such as how to customize the `Marshal` and `Unmarshal` features using [MarshalWithOptions](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue#MarshalWithOptions) and [UnmarshalWithOptions](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue#UnmarshalWithOptions) respectively.