Sort, Filter, and Remap API Data in Python
Tags: python, beginners, webdev, api
Published: 2020-09-30 13:30:17

Are you taking data from an API in the format the web services gives it to you? You should not dictate the structure of data inside your application based on how an API provider structures their data. Instead, you can take advantage of the power of Python's list manipulation techniques to sort, filter, and reorganize data in ways that best suit your needs.

In this article, we'll explore a few methods of using native Python features to sort, filter, and remap data from an external REST API that you can then use within your application, or pass down to a client as part of your own API. Looking for techniques to do this in Javascript? [We wrote about that in a previous article](https://blog.bearer.sh/javascript-api-array-data-manipulation/).

To follow along, you'll need to have Python3 installed and be familiar with running Python code.

## Retrieving data from an API

To get started, we'll need some data to manipulate. For the examples in this article, we can use GitHub's v3 REST API and its `search/repositories` endpoint. The endpoint takes a few parameters, but we'll use `q` to pass in the `Bearer` search query, and `per_page` to limit the results to 10. This makes it easier to follow along if you are printing results to the console, but feel free to increase the returned results per page if you want a larger dataset to work with. You can also change the search query to anything you like, as it won't affect the manipulations that we'll perform.

You can use any HTTP client, but our examples will be in Requests. Begin by installing `requests` and importing it into your project.

To install it, use `pip` (or pip3, depending on your local setup):

```shell
pip install requests
```

Next, import requests and set up the API call.

```python
import requests

def get_repos():
  res = requests.get('https://api.github.com/search/repositories?q=bearer&per_page=10')
  return res.json()['items'] # Returns the value of the 'items' key from the JSON response body
```

The code above defines a function, `get_repos` that will make the API call and return only the part of the response that we are interested in. In this case, the `items` list on the JSON response. From this point forward, whenever we need the items data, we can assign it to a variable with `x = get_repos()`. This will leave us with a `list` of `dict`, or dictionary, data types.

Now with the setup out of the way, we can begin manipulating the data from the API.

## Sorting results in Python

While many APIs allow you to sort the results in some way, rarely do you have access to sort by any property. GitHub, for example, allows sorting by stars, forks, help-wanted-issues, and how recently a repo has been updated. If you want a different sorting method, you'll need to handle this on your own. One thing to keep in mind is that we are only going to sort the results we retrieve. This means that if you capture the first ten results, sort them, then capture the next 10 results, they will not all be in the right order. If working with large datasets, it's best to capture all the data before sorting to avoid repeating the work on the same set of data multiple times.

Python offers the [`sorted` built-in function](https://docs.python.org/3/library/functions.html#sorted) to make sorting lists easier. `sorted` takes the list to you want to sort, and a `key` representing the value to use when sorting the results. Let's take the following example: Sort the repositories by open issue count, from fewest issues to most issues.

```python
# This line can be used in each example, but you only need it once in your file
repos = get_repos()

fewest_issues_sort = sorted(repos, key=lambda repo: repo['open_issues_count'])
```

The code above passes `repos` in to the `sorted` function, and then uses a lambda function for the `key`. Lambda functions in Python are anonymous functions that you can use in-line without defining them elsewhere. They are great for use-cases like ours. This lambda is saying: "pass each iteration of repos to this function as `repo`, then return its `open_issues_count`." The sorted function uses the open issues count as the key. The result is a new list of dictionaries sorted by fewest issues to most issues.

What if you want to reverse the order to show repos with the highest issue count first? You can achieve this with the `reverse` argument in `sorted`. By setting `reverse` to `True` the same logic will apply, but in reverse.

```python
most_issues_sort = sorted(repos, key=lambda repo: repo['open_issues_count'], reverse=True)
```

You can use the same technique to sort by any value.

## Filtering list data with Python

Filters pair well with sorting. They allow you to reduce a list down to only the entries that matter for your needs. There are a variety of ways to filter a list, but they all use the same concept of building a new list from a subset of the original list's entries.

One of the most effective ways to do this is using [list comprehension](https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions). List comprehension is a method for creating lists, but it can also simplify complex loops into shorter expressions. Let's take an example where we only want repositories in our list that have a description filled out.

```python
# Fetch the repos if you haven't already
repos = get_repos()

repos_with_descriptions = [repo for repo in repos if repo['description'] is not None]
```

The code above does a lot with a single line, so let's break it down. It assigns `repos_with_descriptions` to a new list, `[ ]`. The first `repo` in `repo for` represents what we want to add to the new list. The next part `repo in repos` should look familiar from any for-in loop. Finally, the end is our condition: `if repo['description'] is not None`. When a description is not set in the GitHub API, it returns `null` to the JSON response. Python interprets this as `None`, so we check to see if the description is _not_ set to `None` since we only want repositories with a description.

What if we want two conditions? For example, only repositories that have a description and a homepage URL set.

```python
repos_with_desc_and_home = [repo for repo in repos if repo['description'] is not None and repo['homepage'] is not None]
```

As with our previous example, this code says: "Make a new list made up of each `repo` in `repos`, but only include those where the description is not `None` and the homepage is not `None`."

In both examples we've used the `is not` operator. Let's look at a more conventional condition and only include repositories with 100 or more stars.

```python
popular_repos = [repo for repo in repos if repo['stargazers_count'] >= 100]
```

The code again uses list comprehension, but this time includes a more traditional greater-than-or-equal condition to compare values. You could also implement any of these examples with your preferred looping and list-generation methods.

## Remapping or normalizing data

Sometimes an API returns more information than you need. GraphQL solves this by allowing the client to specify exactly the data it needs, but for now most APIs are still REST-like. If you need to reduce the data down to only a few properties before you interact with it or send it on to other parts of your app, you can do so by iterating over the list and building a new one with only the parts you want.

Loops or list comprehension can do this, but we will use Python's `map` function. `map` applies a function to every item in an iterable. Let's look at an example where we take the original `repos` list and simplify it into fewer key/value pairs with map.

```python
def simplify(repo):
	return {
		'url': repo['html_url'],
		'name': repo['name'],
		'owner': repo['owner']['login'],
		'description': repo['description'],
		'stars': repo['stargazers_count']
	}

new_repos = map(simplify, repos)
```

Here, `map` receives the `simplify` function which will run over each item in `repos`. You can use this same method to manipulate the data further, normalize values, and even strip sensitive information.

## Beyond the basics

The techniques mentioned in this article are a great starting point to handle the majority of your use cases. API's mostly return JSON, which Python can convert into dictionaries. In our examples, we captured the `items` list from the response dictionary and used it alongside many built-in Python functions to iterate over the data.

Whether the data coming from an API is the core of your product offering, or only a part of the value you provide to your users, you need to be able to manipulate it to best suit your needs. You also need to know that the source of this data is reliable.

How vital is this API data to the success of your app? What happens when the APIs experience performance problems or outages? There are safeguards you can put in place to avoid and manage downtimes and [protect your integrations](https://blog.bearer.sh/api-integration-best-practices/) from failure. At Bearer we're building a solution that automatically monitors API performance, notifies you of problems, and can even intelligently react when a problem happens. [Learn more about Bearer](https://www.bearer.sh) and give it a try today.

<!--kg-card-end: markdown-->