Advanced Discord.js: Cache APIs Requests with Redis
Tags: redis, discordjs, node, javascript
Published: 2021-04-23 09:00:00

## Table Of Contents
 * [Getting started](#getting-started)
   * [Installation](#installation)
   * [Usage](#usage)
 * [Cache data from APIs](#cache-data-from-apis)
   * [Preparation](#preparation)
   * [Solving the problem](#solving-the-problem)
     * [Implementation](#s implementation)
 * [Wrapping up](#wrapping-up)
   * [Real-world usecases](#realworld-usecases)

> This article is part of a series that can be found [here](https://dev.to/en3sis/series/10809). We're reusing some code from the [previous article](https://dev.to/en3sis/advanced-discord-js-custom-embeds-using-attachments-2bpn)

---
Looking for a Discord.JS bot? [Check out Hans](https://github.com/en3sis/hans)! Now it's open-sourced 🥳!
 
[<img src="https://res.cloudinary.com/vac/image/upload/v1683578541/Hans/hans-banner-pattern.png"/>](https://github.com/en3sis/hans)

---

Nowadays we depend on many APIs that we interact with (weather, games stats, etc...)

Many times we don't have to worry about rate limits but in some cases, we do.
If the API has a low rate limit (e.g. `x` amount of requests per minute) and if we want to deliver the maximum amount of data to our users at some point caching could be the best way to do it.

> **NOTE**: The other option would be to store it in a persistent database such as MongoDB or SQLite, but this would be slower and we add extra network requests if we use an external service

## Getting started <a name="getting-started"></a>

First of all, what is Redis?

> Redis is an in-memory data structure store, used as a distributed, in-memory key–value database, cache and message broker, with optional durability. - [redis.com](https://redis.io/)

This looks promising!

- It's storing data in memory so it will be amazingly fast to read/write.
- We can temporally store data (it can also be persistent).
  For us we're interested in temporary caching, we don't want to show outdated data.

**NOTE**: You can check out the guide for [MacOS](https://medium.com/@petehouston/install-and-config-redis-on-mac-os-x-via-homebrew-eb8df9a4f298) or [Windows](https://redislabs.com/blog/redis-on-windows-10/). More information is available at the [official website](https://redis.io/)

### Installation <a name="installation"></a>

Now that we have Redis running in our system, we can now grab the node package:

`npm i redis`

It's time to test it out!
We can write a new command that will set a temporary key with the data for us.

> **NOTE**: Redis values can store strings, hashes, lists, and so on. We're just interested in strings so for this we'll have to stringify our data before storing it.

### Usage <a name="usage"></a>

We could write two simple functions that will take care of writing and reading data from Redis:

```js
const { promisify } = require('util');
const redis = require('redis');
const client = redis.createClient();

/* Promisfy so we can have promise base functionality */
const getAsync = promisify(client.get).bind(client);
const setAsync = promisify(client.set).bind(client);
const setexAsync = promisify(client.setex).bind(client);
const ttlAsync = promisify(client.ttl).bind(client);

client.on('error', function (error) {
  console.error(error);
});

/**
 * Writes strigify data to cache
 * @param {string} key key for the cache entry
 * @param {*} value any object/string/number */
const cacheSet = async (key, value) => {
  return await setAsync(key, JSON.stringify(value));
};

/** Retrieves data for a given key
 * @param {string} key key of the cached entry */
const cacheGet = async (key) => {
  const data = await getAsync(key);

  return JSON.parse(data);
};

module.exports = async (msg, key, value) => {
  await cacheSet(key, value);

  return msg.channel.send(`We just stored: key: **${key}** | value: **${value}**`);
};
```

Now we can tell Redis to store some data under a specific **key**.

> NOTE: this is not just with discord.js, you can apply the same thing to any Node.js APP or even your own API

Let's try it out by storing a new value from our command, a name for example:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pkieo5zcpkwvt03y7118.png)

We can check our Redis instance to be sure that we actually store it. We'll use the built-in [redis-cli](https://redis.io/topics/rediscli):

- Run `redis-cli` and we'll get something like this:
  ```bash
  /data> redis-cli
  127.0.0.1:6379>
  ```
- Use `KEYS *` to recive all our stored keys
  ```bash
  127.0.0.1:6379> KEYS *
  1) "username"
  127.0.0.1:6379>
  ```
- Use `GET username` to retrieve our stored value
  ```bash
  127.0.0.1:6379> GET username
  "\"en3sis\""
  127.0.0.1:6379>
  ```

This is what we expected to happen. Now we can get to the fun part and unlock all the potential.

## Cache data from APIs <a name="cache-data"></a>

For this demo, we'll use a free [Weather API](https://goweather.herokuapp.com/weather/nuremberg). Later in the article, we'll explore some real-world examples where this approach shines.

### Preparation <a name="preparation"></a>

We'll install [Axios HTTP](https://github.com/axios/axios) client to fetch the API (you can use whatever else. `npm install axios` ) and create a function that will allow us to fetch the API.

```js
/**
 * Fetch from the Weather API endpoint
 * @param {string} city - City to be fetched
 */
const fetchData = async (city) => {
  const { data } = await axios.get(`https://goweather.herokuapp.com/weather/${city}`);

  return data;
};
```

We'll change our command to grab the data from the API and send some of the stats to the chat.

```js
// New code addition
const axios = require('axios');

// ...

// New code addition
module.exports = async (msg, key) => {
  const currentWeather = await fetchData(key);

  return msg.channel.send({
    embed: {
      title: `Weather in ${key}`,
      fields: [
        {
          name: ' 🌡 Temp:',
          value: `**${currentWeather.temperature}**`,
          inline: true,
        },
        {
          name: '🍃  Wind:',
          value: `**${currentWeather.wind}**`,
          inline: true,
        },
      ],
      color: 0x03a9f4,
    },
  });
};
```

If we run the command we'll get the following result:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dmpo77cone44r1kjcckg.gif)

> Sometimes I miss going back to Spain, you can tell why based on the weather :P

### Solving the problem <a name="solving-the-problem"></a>

Let's imagine that our API has a rate limit of 1000 requests per month. With the current implementation, we could only serve 1k requests and not a single extra one.

Now imagine our Bot is part of multiple Guilds and multiple users are using our new command. If **user 1** fetches the data for `Almería`, a beautiful city located in the southeast of Spain on the Mediterranean Sea, we could store this data for 1h for example. We don't really need fresh data (**every 10 min**, and few paid APIs allow you to do so).   
Now, when **user 2** in another server also wants to see the weather in Almería, we'll fetch the data from our local, in memory and blazing fast (~1ms response time) Redis cache.
For the following hour we could show the weather in Almería for **1 billion users** and we only spent **one single HTPP request**!

#### Implementation <a name="implementation"></a>

You notice that I mentioned the persistence of the data, another great build-in function that Redis has is TTL (time to live) where you can specify for how long you want some data to be cached, without having to worry about cronjobs, re-validation, and so on.
We'll add a new function that will cache some data for the amount of time we indicated:

```js
/**
 * Writes strigify data to cache
 * @param {string} key key for the cache entry
 * @param {*} value any object/string/number
 * @param {number} ttl cache duration in seconds, default 3600 (1h) */
const cacheSetTTL = async (key, value, ttl = 3600) => {
  return await setexAsync(key, ttl, JSON.stringify(value));
};
```

Now we can refactor our code so that every time we want to retrieve the weather from a given City, we first check the cache. If the city is in the cache, we use that data. If it's not in the cache, we'll fetch the data from the API and save the copy in our Redis instance. We can implement this directly in our `fetchData()` function.

```js
/**
 * Fetch for the Weather API endpoint
 * @param {string} city - City to be fetched
 */
const fetchData = async (city) => {
  const isCached = await cacheGet(city);

  if (isCached) {
    console.log('⚡️  From cache');

    return isCached;
  } else {
    // Fetch data
    const { data } = await axios.get(`https://goweather.herokuapp.com/weather/${city}`);

    // Save data to cache
    await cacheSetTTL(city, data);

    return data;
  }
};
```

> The whole code can be found [here](https://github.com/en3sis/discord-guides/blob/main/examples/cache.js)

And we're done! We can now run our command, check for the weather in a given city and return the already cached data or fetch and store it.

When we run our command, it will:

1. Check for the KEY in Redis
2. It won't find it so it will make the HTTP request to the API
3. Saves the data in Redis using the city as KEY
4. Return the data from our `fetchData()` function and send the embed

![Diagram of workflow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tcz2gvi9tfnf8vhdlyoj.png)

For the second time we (or another user) use the command, it will grab the data directly from the cache.

```bash
# In Discord
> cache nuremberg
> cache nuremberg
> cache nuremberg

# We should see in our application a console log saying:
Logged in as Hans!
⚡️  From cache
⚡️  From cache

```

For the first command, we fetch and store the data, for the following commands we serve the data from the cache.

> **NOTE**: You can always check the remaining time left by using the redis-cli or the `ttlAsync` function:

```bash
127.0.0.1:6379> KEYS *
1) "nuremberg"
127.0.0.1:6379> TTL nuremberg
(integer) 3370 # remining time in seconds
127.0.0.1:6379>
```

## Wrapping up <a name="wrapping-ip"></a>

I hope that this walkthrough helped you get a better understanding and given you some ideas on how to handle the sometimes annoying rate limits.

### Real-world usecases <a name="realworld"></a>

As promised before, here some examples of when this is really useful.

1. When dealing as in our example with APIs like the weather where we want to reuse the most amount of data with spending a single request.
2. Games APIs: I used it to fetch data from games like Battlefield and reuse the data for things like players' comparison. If I want to see `user A` stats, then `user B` used the command to see him and I decide to compare our profiles, see how's doing better I can run the command with something like `!bf userA userB` and instead of doing two requests to the API to get each player stats, I used the data already available in my cache.
3. Same as before, one of the commands is the COVID-19 stats. I also cache the data for a given country (since it's updated once per day) so I can reuse the cache data when another user from a different server fetches the data from the same country.
4. Dashboard and Discord API: Discord only allows you to fetch the API by sending an `x ammount of requests per second. While working with a Dashboard where you need to fetch the Guild channels, users, roles... you don't want to do it every time you load a Guild dashboard. For it, I only do it once and set a TTL of ~2mins for some parameters.


### Finally
As always, you can find the code with all examples at https://github.com/en3sis/discord-guides

Any feedback, questions, or suggestions are welcome!
Thanks for reading! ~ https://twitter.com/en3sis