TIL: query the NPM paginated Rest API with Elixir
Tags: npm, elixir, aws
Published: 2023-03-13 01:22:12

These are notes on how to use `Elixir` to find all the NPM packages published by AWS with their download count. 

The total count is 1676. We start and query the [npms-api](https://api-docs.npms.io/) search endpoint: we give a string and it returns a list of matching Javascript packages. The endpoint is **paginated**. With this list, we query the NPM registry to get statistics for each package.

We send a total of approx 1750 requests. The longer one is the paginated API with a response time of approx 200-250ms per query.  Then we send 1700 queries to get the stats on each package with an average response of 10-15ms. We get the result on average in around 20s (compared to a sequential 1700/25*0.200=13s + 1700*.0.010=17s = 30s). 

We used `Elixir` to achieve this with the HTTP client `Finch` to stream and the Rust based JSON parser `jsonrs`.

We used an easy path: build the full list of found packages, and then query the details for each element of the list.

We used [Stream.resource](https://hexdocs.pm/elixir/1.14.3/Stream.html#resource/3) to build the list of packages. We used the total count returned by the endpoint to handle the pagination and increment a counter on each iteration.

We then use [async_stream](https://hexdocs.pm/elixir/1.14/Task.html#async_stream/3) to query the second endpoint. It returns statistics on a given package. We retrieve the downloaded count during a given period.

We then [Enum.sort_by](https://hexdocs.pm/elixir/1.14/Enum.html#sort_by/3) to order the list of maps on a given key.

With the result, we run a side-effect as a `Task`: it saves the data in a file. Since we need the data unchanged for the final step, use the function [tap](https://hexdocs.pm/elixir/1.14/Kernel.html#tap/2) and the data passes by. This allows us to keep the flow and run this side -effect.


We eventually prettify the data into a list of maps `%{"package_name" => downloaded_count}`.

`Elixir` can nicely chain all these streams. In a Livebook, the following code will default to the AWS packages.

```elixir
Mix.install([
  {:finch, "~> 0.16.0"},
  {:jsonrs, "~> 0.3.1"}
])
Supervisor.start_link(
  [
    {Finch, name: MyFinch},
    {Task.Supervisor, name: MyTaskSup}
  ], 
  strategy: :one_for_one,
  name: MySup
)
```


```elixir
defmodule Npm do
  require Logger

  @registry "https://api.npmjs.org"
  @search_point "https://api.npms.io/v2/search"

  @starting "2022-01-01"
  @ending "2023-01-01"
  @search "@aws-sdk/client"
  @aws_npm_packages "aws-npm-packages.json"

  def find(save? \\ false, string \\ @search, starting \\ @starting, ending \\ @ending) do

    check_response = fn response ->
      case response do
        {:ok, result} ->
          result

        {:error, reason} ->
          {:error, reason}
      end
    end

    #  the optional "save to file"
    save_to_file = fn list ->
      Logger.info(%{length: length(list)})

      Task.Supervisor.async_nolink(MyTaskSup, fn ->
        case Jsonrs.encode(list, lean: true, pretty: true) do
          {:ok, result} -> File.write!(@aws_npm_packages, result)
          {:error, reason} -> reason
        end
      end)
    end

    # the iterating function in Stream.resource
    next = fn {data, page} ->
      {response, total} = search(string, 25 * page)

      case page * 25 >= total do
        true ->
          {:halt, data}

        false ->
          {response, {data, page + 1}}
      end
    end

    try do
      Stream.resource(
        fn -> {[], 0} end,
        &next.(&1),
        fn _ -> nil end
      )
      |> Task.async_stream(&downloaded(&1, starting, ending))
      |> Stream.map(&check_response.(&1))
      |> Enum.sort_by(&Map.get(&1, "downloads"), :desc)
      |> tap(fn data -> if save?, do: save_to_file.(data) end)
      |> Enum.map(fn %{"downloads" => d, "package" => name} ->
        Map.put(%{}, name, d)
      end)
  
    rescue
      e ->
        Logger.warn(e)
    end
  end

  # we send a tuple {stream, total}
  def search(string, from \\ 0) do
    url = 
      URI.new!(@search_point)
      |> URI.append_query(
        URI.encode_query(%{q: string, size: 25, from: from})
        )
      |> URI.to_string()

    with {:ok, %{body: body}} <-
           Finch.build(:get, url)
           |> Finch.request(MyFinch),
         {:ok, %{"results" => results, "total" => total}} <- Jsonrs.decode(body) do
      {
        Stream.filter(results, fn package ->
          Map.has_key?(package, "flags") === false &&
            get_in(package, ["package", "name"]) |> String.contains?(string)
        end)
        |> Stream.map(&get_in(&1, ["package", "name"])),
        total
      }
    else
      {:error, reason} ->
        reason
    end

  # the second endpoint
  def downloaded(package_name, start, ending) do
 
    path = @registry <> "/downloads/point/" <> "#{start}" <> ":" <> "#{ending}" <> "/" <> "#{package_name}"

    with {:ok, %{body: result}} <-
           Finch.build(:get, path) |> Finch.request(MyFinch),
         {:ok, response} <- Jsonrs.decode(result) do
      response
    else
      {:error, reason} -> reason
    end
  end
```

The usage is:

```elixir
iex> Npm.find(false, "@aws-sdk/client", "2022-01-01", "2022-03-01")
```


![aws-sdk-graphic](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ptz39izbjxk1tglbig1n.png)



The same result with `@google-cloud/`:


![google-cloud](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/thurc4y5hl7fanzvg0wb.png)



If you have Livebook, installed, you can run a session with the button:


[![Run in Livebook](https://livebook.dev/badge/v1/blue.svg)](https://livebook.dev/run?url=https://github.com/ndrean/gruland/blob/main/livebook.livemd)


If not, you can easily run a Livebook from Docker. Run the image:

```
docker run -p 8080:8080 -p 8081:8081 --pull always -e LIVEBOOK_PASSWORD="securesecret" livebook/livebook
```

and then from another terminal, execute:

```
open http://localhost:8080/import?url=https://github.com/ndrean/gruland/blob/main/livebook.livemd
```