Learning Rust #4: Parsing JSON with strong types
Tags: rust, learning
Published: 2021-05-12 18:30:44

---
title: Learning Rust #4: Parsing JSON with strong types
published: true
description: I recently did a refactoring by defining strong types for the API responses my client uses using serde_json. Here's what I learned
tags: rust, learning
series: Learning Rust
canonical_url: https://hamatti.org/posts/learning-rust-4-parsing-json-with-strong-types/
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vxwphznjl3b2bxfvozov.png
---

_Last December I finally started learning Rust and last month I built and published my first app with Rust: [235](https://github.com/Hamatti/nhl-235). Learning Rust is my new monthly blog series that is definitely not a tutorial but rather a place for me to keep track of my learning and write about things I've learned along the way._

One big difference with Rust compared to Javascript and Python that I've been writing for most of my career is strong static typing. With JS and Python, I'd just call an API, get some data in JSON, parse that into a Python dict or Javascript object and then access different values directly.

With Rust, I'm working with differently typed structures and while I understand the value of it, sometimes it feels very cumbersome to write.

## I started with serde_json::Value

When I started working with Rust to build 235, I decided to use [Serde](https://serde.rs/) that seemed to be a popular library for serializing and deserializing data. While I was still learning the basics, it felt overwhelming to figure out the correct way of defining types for the entire API response. I decided to leave that as a refactoring exercise for a later day and started by using [`serde_json::Value`](https://docs.serde.rs/serde_json/value/enum.Value.html), which is kinda a catch-all solution. It is an Enum that represents any valid JSON value.

```rust
#[tokio::main]
async fn fetch_games() -> Result<serde_json::Value, Error> {
    let request_url = String::from("https://nhl-score-api.herokuapp.com/api/scores/latest");
    let response = reqwest::get(&request_url).await?;
    let scores: serde_json::Value = response.json().await?;

    Ok(scores)
}
```

Here's my original `fetch_games` function that called the API and parsed the data into a `serde_json::Value`. Nice and clean, easy to use.

But it led to my code being full of code like this

```rust
let games = scores["games"].as_array().unwrap();

let home_team = &game_json["teams"]["home"]["abbreviation"].as_str().unwrap();

let all_goals = game_json["goals"].as_array().unwrap_or(&empty);
```

I had to code very defensively because any type conversion with `as_` methods could have fail and thus needed to be unwrapped and in some cases, made sure the keys even existed in the beginning:

```rust
if (&game_json["teams"]).is_null() {
  return None;
}

let home_team = &game_json["teams"]["home"]["abbreviation"].as_str().unwrap();
let away_team = &game_json["teams"]["away"]["abbreviation"].as_str().unwrap();
```

Using `serde_json::Value` did help me build the first functioning versions of my application but in the back of my head I had the knowledge that defining the structures as Rust structs would make my code better.

One key thing that kept me from doing that in the beginning was I couldn't figure out how to deal with dynamic keys. The API I use uses team abbreviations as keys on certain places and I had no idea how to do it so I left the issue for months.

## How to define structs for JSON

Using custom structs to define expected types with JSON is rather straight-forward if you're familiar with structs.

I'll start with the example [from the documentation](https://docs.serde.rs/serde_json/):

```rust
#[derive(Serialize, Deserialize)]
struct Person {
    name: String,
    age: u8,
    phones: Vec<String>,
}
```

Here we define a `Person` that has name as `String`, an age as `u8` and a list of phone numbers as `Vec<String>`. Once we have this definition, all we have to do is to use that as a type of the result and `serde_json` and Rust will validate that the structure and types of data matches what we defined:

```rust
// Parse the string of data into a Person object.
let p: Person = serde_json::from_str(data)?;

// Do things just like with any other Rust data structure.
println!("Please call {} at the number {}", p.name, p.phones[0]);
```

## Then we draw the rest of the owl

I found the base example very good and nice to follow. However, that wasn't all that I needed so next I needed to figure out how to [draw the rest of the owl](https://knowyourmeme.com/memes/how-to-draw-an-owl).

### snake_case of Rust with camelCase of JSON

First, Rust likes to use [snake_case](https://en.wikipedia.org/wiki/Snake_case) and gives you a warning by default if you don't. JSON keys are usually in [camelCase](https://en.wikipedia.org/wiki/Camel_case) so my editor and my shell were full of warnings and that was annoying. I learned that I can silence them by defining an `allow` attribute for each struct that needed it:

```rust
#[derive(Debug, Serialize, Deserialize)]
#[allow(non_snake_case)]
pub struct TeamResponse {
    pub abbreviation: String,
    pub id: u64,
    pub locationName: String,
    pub shortName: String,
    pub teamName: String,
}
```

I haven't yet found out if there's a way to use snake case'd names and automatically map them to corresponding JSON values. (Raymond Hettinger from the Python community has [a nice talk for why it's valuable to have your code in idiomatic Python](https://www.youtube.com/watch?v=wf-BqAjZb8M); much of it is applicable to any language).

**edit**: Thanks to [PatatasDelPapa's comment in pull request](https://github.com/Hamatti/nhl-235/pull/27#discussion_r626688198), I was able to fix this:

```rust
#[derive(Debug, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct TeamResponse {
    pub abbreviation: String,
    pub id: u64,
    pub location_name: String,
    pub short_name: String,
    pub team_name: String,
}
```

### Code organizing API response types into a separate file

Second, you'll see that in the snippet above, definitions start with pub. That's because I wanted to move my API response type definitions to its own file so they don't pollute my main.rs and it's easier to find and change them. So I made `api_types.rs` file, moved them all there, added plenty of `pub`s and imported them with

```rust
mod api_types;
use api_types::{APIResponse, GameResponse, GoalResponse};
```

in main.rs.

### Dynamic keys

Third, I needed some support for dynamic keys. I was expecting to have to jump through some hoops but this ended up being one of the easiest steps in the process. Thanks to [this GitHub issue](https://github.com/serde-rs/serde/issues/1387), I found the solution: use `HashMap<String, serde_json::Value>` for any object that has dynamic keys.

```rust
#[derive(Debug, Serialize, Deserialize)]
#[allow(non_snake_case)]
pub struct GameResponse {
    pub status: StatusResponse,
    pub startTime: String,
    pub goals: Vec<GoalResponse>,
    pub scores: HashMap<String, serde_json::Value>,
    pub teams: TeamsResponse,
    pub preGameStats: PreGameStatsResponse,
    pub currentStats: CurrentStatsResponse,
}
```

In the documentation of the API, `scores` field is defined as

> `scores` object: each team’s goal count, plus one of these possible fields:
>- `overtime`: set to `true` if the game ended in overtime, absent if it didn’t
>- `shootout`: set to `true` if the game ended in shootout, absent if it didn’t

and looks like

```json
"scores": {
  "BOS": 4,
  "CHI": 3,
  "overtime": true
}
```

The keys for each team are gonna be dynamic so by using `HashMap`. The downside of this that I wasn't able to figure out, is that I'd like to put `overtime` and `shootout` keys into the type definitions.

### Optional keys

And finally, fourth, some keys are optional – or to be specific in this case, they are dependent on the situation.

I'm happy I already learned about `Option` types earlier (check out [Learning Rust #2](https://hamatti.org/posts/learning-rust-2-option-result/) if you haven't heard about them). And that's all you need with Rust and `serde_json`:

```rust
#[derive(Debug, Serialize, Deserialize)]
pub struct APIResponse {
    pub date: DateResponse,
    pub games: Vec<GameResponse>,
    pub errors: Option<HashMap<String, serde_json::Value>>,
}
```

In this example, which is the top level definition of my API response type, I've defined `errors` to be optional – it's only present if the API I use found some errors in the data it's using.

By saying that the type of the data is `Option`, we'll either get a `Some(value)` if the key exists and `None` if it doesn't. For most cases in this application, it's not random which keys are there and which are not. So if I'm already in a branch of logic where the prerequisite is being dealt with, I can trust that the value exists and can unwrap without dealing with potential `None`s.

## Result? Much cleaner code

Refactoring the code with these definitions is not only an exercise of writing code for the sake of writing code. The result of this work is that we push the problems at the boundaries of parsing the data and we don't have to deal with them at the logic level anymore.

I already showed some examples in the beginning about the code I wasn't happy about. Let's see how they look like now:

### Before

```rust
if (&game_json["teams"]).is_null() {
  return None;
}

let home_team = &game_json["teams"]["home"]["abbreviation"].as_str().unwrap();
let away_team = &game_json["teams"]["away"]["abbreviation"].as_str().unwrap();
```

### After

```rust
let home_team = &game_json.teams.home.abbreviation;
let away_team = &game_json.teams.away.abbreviation;
```

### Before

```rust
let empty: Vec<serde_json::Value> = Vec::new();
let all_goals = game_json["goals"].as_array().unwrap_or(&empty);
```

### After

```rust
let all_goals = &game_json.goals;
```

I don't know about you but I'm very happy to see my code being much more simpler and easier to read. I can also trust that once the data comes through from the API, it won't have any surprises.

## Anything I don't like?

Writing type definitions for these kind of nested structures feels sometimes overly complicated. There are some definitions that I don't need anywhere outside one value in the structure and having to write a named struct to that feels bit annoying.

One thing I did enjoy in Typescript is a way to write in-line nested types and interfaces like this:

```typescript
interface IEndpoints {
 auth: {
  login: string;
 }
}
```

In Rust, I would have needed to make two structs.

## Pull requests for all these changes

If you wanna check out all the changes and how the codebase cleaned up during this process, you can find it in GitHub: [hamatti/nhl-235/pull/27](https://github.com/Hamatti/nhl-235/pull/27/).