Querying paginated API endpoints the Rails way
Tags: ruby, rails, api
Published: 2020-08-19 03:29:19

---
published: true
author: jesalg
layout: post
canonical_url: https://jes.al/2020/08/querying-paginated-api-endpoints-the-rails-way/
tags:
  - ruby
  - rails
  - api
title: "Querying paginated API endpoints the Rails way"
description: How to leverage the key features of the Enumerator and Hash objects in Ruby to build elegant API wrappers that follow Rails conventions
---
Recently I was working on querying an API that didn't have a Ruby SDK. So I had the opportunity to write a thin service wrapper for it. This API allowed requesting a specific page and the total number of records to include per page. 

So say for example, the API returned a list of people. Here's what that would look like:

```json
{
  "meta": {
    "page": { "total": 200 }
  },
  "items": [
    {
      "type": "people",
      // ...
    },
    {
      "type": "people",
      // ...
    }
  ]
}
```

**First pass**

I made a first attempt at writing a `PeopleService` PORO that is responsible for querying those records:

```ruby
class PeopleService	
  def where(params = {})
    default_params = { page: 1, count: 25 }
    params = default_params.merge(params)
    
    make_request(params)
  end

  private

  def make_request(params)
    # Make external API call using the params
  end
end
```

That would do the job if the caller writes their our own iteration logic. For instance, if we want to retrieve all the People from the API:

```ruby
page_count = 0
people = []
loop do
  page_count += 1
  result = PeopleService.new.where(page: page_count)
  people << result[:items]
  break if page_count = result.dig(:meta, :page, :total).to_i
end
```

Although that feels a bit messy. The API contract is leaking out of the `PeopleService` abstraction layer that we just created.

**Let's make it Rails-y**

I want my service object to follow more Rails like conventions. So in other words, I'd like to be able to iterate over the results from the `PeopleService` in an `ActiveRecord` like syntax. For example: `PeopleService.new.all.each{ |person| #some operation }`

**Enumeration**

So to achieve that, we will have to make use Ruby's [Enumerator](https://ruby-doc.org/core-2.6/Enumerator.html) object:

```ruby
class PeopleService	
  def initialize
    # Setup API auth params
  end

  def where(params = {})
    default_params = { page: 1, count: 25 }
    params = default_params.merge(params)

    make_request(params)
  end
  
  def all(params = {})
    Enumerator.new do |yielder|
      page = 1
      loop do
        result = where(params.merge(page: key))
        result[:items].map { |item| yielder << item }

        # This loop knows how to rescues from this exception and will treat it as a break		
        raise StopIteration if page >= result.dig(:meta, :page, :total).to_i
    
        page += 1
      end
    end.lazy
  end

  private

  def make_request(params)
    # Make external API call using the params
  end
end
```

That will get us closer to what we are looking for. Using the Enumerator object above is what will give us the ability to iterate over the results returned from the `all` method. It will instantiate and return an [Enumerable](https://ruby-doc.org/core-2.6.5/Enumerable.html) object. 

That will unlock a powerful ability to chain a number of enumerators together and perform block operations on them which will make our service highly [composable](https://en.wikipedia.org/wiki/Composability).

So for example, if we wanted to group the people by their location we could chain the results with a `group_by` function: 

```ruby
PeopleService.new.all.group_by{ |person| person.location }
```

Lastly, you might have noticed we tacked in a `.lazy` at the end of the enumerable instance. What that does is makes it into an instance of `Enumerator::Lazy` and only returns the results that we specifically enumerate over. 

So say if this API had 1,000 pages of results. Without the **lazy** enumerator, `PeopleService.new.all` would actually query all of those 1,000 pages as soon as you call it. That would be extremely slow and resource-intensive. In most cases, we might even hit a rate limit set by the API provider. What we rather want is that it *only* query the pages that we actually enumerate over.

So for example, if we are trying to find the person object with a specific email, it will stop querying the API as soon as it finds a page that contains Jon Doe:

```ruby
PeopleService.new.all.find { |person| person.email = 'jon.doe@example.com' }  
```

**Caching**

Right now calling the `all` method again, despite it being a lazy enumerator will query the pages it has already queried. So for example:

```ruby
ps = PeopleService.new

# This will iterate through the pages until we find Jon Doe
person = ps.all.find { |person| person.email = 'jon.doe@example.com' }

# Calling this again **should not** query the same pages again. We should already store the results.
person = ps.all.find { |person| person.email = 'jon.doe@example.com' }
```

Similar to `ActiveRecord`'s query cache, we also want to cache the results from our query for performance. This is where one of the most underrated features of the [Hash](https://ruby-doc.org/core-2.6.6/Hash.html#method-c-new) class comes into play.

If you instantiate a Hash with a block, it will use that block to calculate the value of the key. In our case, we can tell the hash to call the API to fetch the results of the page we are looking for. 

The beauty of this feature is that it will only call the block once per key. So if the key has already been assigned a value, it will not call it again:

```ruby
h = Hash.new do |h, key|
  h[key] = where(page: key)
end

h[1]
# Fetches results for page 1 fromt he API
# => (500.0ms) [{...},{...},{...}]

# Next call to the same key is already assigned, the block isn't executed
h[1]
# => (Cached 0.0ms) [{...},{...},{...}]
```

When using the Hash approach in our class, we will also want to make sure that we use [Memoization](https://en.wikipedia.org/wiki/Memoization) (using the `||=` operator) to ensure that the Hash itself is cached in an instance variable called `all_pages`. 

This will allow us to call the `all` method multiple times after the class has instantiated and ensure the results don't get overwritten:

```ruby
class PeopleService	
...
def all(params = {})
  Enumerator.new do |yielder|
    page = 1
    loop do
      @all_pages ||= Hash.new do |h, key|
        h[key] = where(params.merge(page: key))
      end
      result = @all_pages[page]
      result[:items].map { |item| yielder << item }
  
      raise StopIteration if page >= result.dig(:meta, :page, :total).to_i
  
      page += 1
    end
  end.lazy
end
...
end
```

**Final form**

Here's what our finished product looks like after leveraging the key features of the Enumerator and Hash objects. Now our `all` method's interface will be very similar to the one provided by `ActiveRecord`

```ruby
class PeopleService	
  def where(params = {})
    default_params = { page: 1, count: 25 }
    params = default_params.merge(params)

    make_request(params)
  end
  
  def all(params = {})
    Enumerator.new do |yielder|
      page = 1
      loop do
        @all_pages ||= Hash.new do |h, key|
          h[key] = where(params.merge(page: key))
        end
        result = @all_pages[page]
        result[:items].map { |item| yielder << item }
    
        raise StopIteration if page >= result.dig(:meta, :page, :total).to_i
    
        page += 1
      end
    end.lazy
  end

  private

  def make_request(params)
    # Make external API call using the params
  end
end
```
Usage:
```ruby
  ps = PeopleService.new
  ps.all.each do |person|
     # some operation on the person object
  end
```
Let me know if that was useful. Would love to hear about any other techniques that you've found particularly interesting when querying external APIs.

_This post was originally published on <a href="https://jes.al/2020/08/querying-paginated-api-endpoints-the-rails-way/">my blog</a>. If you liked this post, please share it on social media and <a href="https://twitter.com/jesalg">follow me on Twitter</a>!_