Dealing with multi-level nested API data and creating a seeding file with it
Tags: ruby, api, database
Published: 2022-10-03 01:35:23

<p>
Dealing with multi-level nested data from an API call can sometimes be overwhelming. We are going to go through an example from a project that I am currently working on and how it was broken down into smaller, more manageable pieces. The end goal is to create a seeding file to populate a data table.
</p>
<h1>Evaluating the API Response</h1>
<p>
The first step we need to do is look at the response that our <code>GET</code> request will receive from the API. We will look at what data we get, and what data we want to extract to use in our app. For this example we are using the ESPN API, and our goal is to get a list of all the current NFL players. We also want to get some information about each player. In the NFL there are 32 teams, each team having 53 active players. That alone adds up to 1,696 players. Then add in the players on each teams practice squad, and players on injured reserve and the number climbs well over 2,000 players. Now consider that we want to collect info on each player such as: their team name, team location, position they play, their playing status, and a couple other things and its easy to see how this can become overwhelming very quickly.
</p>
<p>
First things first, lets look at the API return and see what we are dealing with. The end point we are going to be using is:
</p>

```
/apis/site/v2/sports/football/nfl/teams/team_id/roster
```
<p>
When using this endpoint, the <code>team_id</code> will be replaced with the specific team that you want the roster info for. Here is the return when you use the <code>team_id</code> of <code>23</code>.
</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rhm4ltnsoquk895pp6md.png)

<p>
We see that we have received an object with 6 keys: athletes, coach, season, status, team, and timestamp. So we see the team key with the info of the team and location of the team that the players on this roster play for. But the players are in another key that is split up into 3 array objects, one for offense, one for defense, and one for special teams. Lets look at the array items in the athlete key.
</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6ttl96jeddwsrajd9qdg.png)

<p>
Now we see that each item in the under the athletes key contains an array of players. Lets go further and look in the first <code>items</code> array.
</p>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hqeycqj1tcu5pb66w64k.png)

<p>
And now inside one of the players.
</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b8r3t76yiplp2wtcik9j.png)
<p>
Lets not forget about the team info for this group of players.
</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sg81ehykfrdse3sq7y2y.png)

<p>
So now we can see all the info we are looking for, but this is for just one team, there are still 31 other teams!
</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a1rmx25v292hb1u7aesy.gif)
<p>
At this point its easy to feel overwhelmed but if we start to break it down into smaller steps it begins to feel much more manageable.
</p>
<h1>Breaking it Down</h1>
<p>There are many ways to approach a task like this. I find the easiest is to start at top level and work down through. As we move down through each level, take note if there is any data on that level that we want to capture. If there is, then capture it and then continue moving down through the levels. 
</p>
<p>
The API endpoint returns all the info for a single team so we will have to repeat the call for each team. This tells us we will need some kind of loop to make a call for each one. Because we know that there are a total of 32 teams, we can simply use the <code>#times</code> method to make a <code>GET</code> request for each team. Looking into the <code>team_id</code> we see that they start at 1 and increment up by one. We can declare a variable and set it to 1, then interpolate that variable into the API endpoint for the <code>team_id</code>, then just increment the variable by one at the end of each loop.
</p>

```
i = 0
32.times do
    res = RestClient.get "https://site.api.espn.com/apis/site/v2/sports/football/nfl/teams/#{i}/roster"
    player_hash = JSON.parse(res)

    end
    i = i + 1
end
```
<p>
We now have a loop that will return an object for each team and store that object in <code>player_hash</code>. We can pull the team name, team location, and even the link to the team logo image from the <code>player_hash</code>. We can now check those data points off the list of what we want to collect and look at the rest of the keys on this level to see if there is anything else to collect.
</p>
<p>
We are not going to be using any of the data from the timestamp key, the status key, the season key, or the coach key. The only other key of interest is the athletes key. Looking at the athletes key, we can see that it is an array of 3 objects. We can use the<code>#each</code> method to iterate through them and continue down through the levels looking for data to collect.
</p>

```
player_hash["athletes"].each do |group|
    group["items"]

end
```
<p>
This takes us through the athletes array, which contains an <code>items</code> array, and <code>position</code> key: value pair. We are not going to be using the general <code>position</code> information, so we know that there is no data to collect yet and we can continue on. We now iterate through the <code>items</code> array which is an array containing the player objects. These objects contain all the player information that we want to collect and we can check all the data points we are looking for off our list. We now have access to everything we need to use the <code>#create</code> method and populate the <code>players</code> table in our database.
</p>

```
group["items"].each do |p|
    Player.create(
        name: p["fullName"],
        position: p["position"]["name"],
        number: p["jersey"],
        status: p["status"]["name"],
        is_drafted: false,
        team_name: player_hash["team"]["name"],
        team_location: player_hash["team"]["location"],
        team_logo: player_hash["team"]["logo"],
        image: p["headshot"]
    )
end      
```
<p>
As a last little touch we can give some feed back while the seeding process is taking place with some <code>puts</code>. Below is the completed seeds.rb file.
</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4psyr2jop9bmdapb2i7h.png)

<p>
If you look at the code you will see the added line:
</p>

```
puts "Seeding #{player_hash["team"]["name"]} players ..."
```
<p> 
If you are still unclear with what exactly is happening, this line might help to visualize what is going on. Lets see the seeding in action.
</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pqdqs0qs1ggjd388n09h.gif)
<p>
We have now sucessfully seeded our <code>players</code> table with all 2,393 NFL players! 
</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ngunr732wj6ao52g8m1j.png)
<p>
A note for those that were looking closely, if you noticed that the <code>.times</code> was set to 34 instead of 32, good eye. This was done to compensate for the <code>team_id</code> that the API uses. In the id list, it skips numbers 31 and 32 so the <code>team_id</code> go from 1 to 30, then skips to 33 and 34.
</p>