Lighthouse Scanner: Stateless NodeJS Microservices with Redis DB
Tags: node, microservices, redis
Published: 2020-06-22 16:03:39

Lighthouse is a service to scan webpages and see how good they score on terms of SEO, performance and best practices. You can use the scanner here: **[https://lighthouse.admantium.com/](https://lighthouse.admantium.com/)**.

Microservices execute functions. They operate with data, and the produce data. This data should not be inherent to the microservice, but it should be persisted in a database. In a cloud environment, when a microservice becomes unavailable, it is replaced by a new one. Newly deployed microservices simple pick up the state from the database. With this approach, there is no downtime at all.

In my lighthouse project, the scanner microservice produces two types of data: Jobs, which represent scan requests that need to be executed. Reports, which are the results of the jobs. This data needs to be persisted and queried. So, what is the best database for this? How can I scale the database with increasing traffic?

In this article, I detail how to decouple the microservice from its data.

_This article originally appeared at [my blog](https://admantium.com/blog/light04_stateless_microservice_with_redis/)_.

## Which Database to Choose?

How do you choose your database? It’s obvious to stick to those that you have already worked with: MySQL, PostgreSQL, SQLite, MongoDB or CouchDB. In lighthouse, I was tempted to use MongoDB because all the moving data is easily represented with JSON objects. But the primary data that is produced in the app is not something that needs to be stored forever: A job is just a temporary entry, and even scan results are freshly produced on demand.

Which database has this implicit "time to live" for data as its key features? I remembered [Redis](http://redis.io), the number one key value store according to this [report](https://db-engines.com/en/ranking/key-value+store). Redis works with simple command on the console: Set a value with `set msg "Hello"`, and receive a value with `get msg`. It’s that simple. Redis supports different data structures like lists, sets, ordered list, hashes. It’s also blazingly fast because it runs in-memory. Its schema-less nature means you can structure data in any way you want, its evolvable to your requirements. The final point to convince me was that within a tutorial of mere hours you were ready to go. In total, I spend a day of learning all essential commands, data structures and configuration/management aspects, and then half a day to add it to my lighthouse app.

In the remainder of this article, I will show you Redis by example when using it to make the lighthouse scanner truly stateless.

## Redis Basics

Redis provides two main commands: `redis-server` and `redis-cli`. The server starts on 5678 and accepts connections from all clients. The `redis-cli` starts an interactive terminal session. Here, you execute [redis commands](https://redis.io/commands) to create, read, update or delete data, as well as system maintenance commands. Date that is exchanged between Client and Server is just serialized text. In its default configuration, there is no authorization, TLS or access control lists, but this can be configured in a [very readable config file](https://raw.githubusercontent.com/antirez/redis/6.0/redis.conf).

Redis nature as a pure key value store becomes visible by considering the basic data structures:

- Strings
- Hashmaps
- Lists, Sets, Ordered Sets

To define a string value, the command is simply `set KEY value`. For a hashmap, it’s a key followed by field-value pairs: `hmset KEY  field1 value field2 value`. And a list is modified with `lpush LIST value1 value2`. Reading these values is a simple `get KEY` for strings and `hgetall KEY` for hashes.

Let’s see an example working with a list. We create the list `jobs:list` and push the values `job1, job2, job3` into it. Then with `lrange` we print the list content, starting at the index `0` until its end. We extract a value with `lpop` and print the list content again.

```bash
client@redis> lpush jobs:list job1 job2 job3
(integer) 3
client@redis> lrange jobs:list 0 -1
1) "job3"
2) "job2"
3) "job1"
client@redis> lpop jobs:list
"job3"
client@redis> lrange jobs:list 0 -1
1) "job2"
2) "job1"
client@redis>
```

Redis features other [specialized date structures](https://redis.io/topics/data-types-intro): Geo for defining and working with geo coordinates, or Stream for timeseries-like data. And Redis features several modules that extends the core feature and data structures. For example: [RedisReJson](https://oss.redislabs.com/redisjson/) for manipulating JSON, [RedisGraph](https://oss.redislabs.com/redisgraph/) for implementing graphs, and modules like [Redis Gears](https://oss.redislabs.com/redisgears/) which enable in-memory event-base data transformations.

I know, for experienced programmer this looks very simple, simple to the point that you might raise the question "So what’s so great about it"? For me, this simplicity is refreshing! You pick the data structure that is most suitable for you, learn and apply the commands, and can just use it immediately. Your data is stored very space and functional efficiently, manipulated with simple commands. Redis has a very flat learning curve, within mere hours you can get a good understanding and experience with all major data structures. And also, installing and running Redis just works without any additional configuration.

## Building a Stateless Microservice

Now let’s see how to apply Redis when building a stateless microservices. The primary imperatives are:

- All data must be persisted immediately
- Persisting and Reading must be fast and efficient
- Working data must be deleted easily

In lighthouse, data is created or modified in these use cases:

- Create and update a job
- Create a report

Let’s discuss each case, and see which Redis data structure to apply best.

### Jobs

A job object captures information and state of a scan job. It is pure working data and does not carry significance after it has been completed.

Concretely:

- A job object consists of `uuid`, `domain` and `status`.
- The `uuid` is the identifier
- The `status` changes from `created` to `finish` or `error`
- The `domain` is used to retrieve the scan report once it is finished

Lets see how to create, read, update and delete these values.

To store this key-values pairs, a Redis hash map is the best choice: Indexed by its `uuid`, with fields for `status` and `domain`. To create a hash map for a request to scan the domain `http://example.com`, we just execute the following command:

```shell
hset "0b25ab16-6efd-485c-b260-1766505a3811" domain "http://example.com" status "started"
```

After its creation, the data value can be retrieved with the following command:

```shell
hgetall "0b25ab16-6efd-485c-b260-1766505a3811"
1) "domain"
2) "http://example.com"
3) "status"
4) "started"
```

To update the value, we use the same command again, and just list the changed values.

```shell
hset "0b25ab16-6efd-485c-b260-1766505a3811" status "finished"
```

Finally, to delete this data, you either use the explicit `del` command or you set a time in seconds how long the data will be kept. In lighthouse, I decide to keep jobs exactly 24 hours.

```shell
expire "0b25ab16-6efd-485c-b260-1766505a3811" 86400
```

### Reports

Once a scan job is done, a report will be generated. This report is a single, self-contained HTML page. As of yet, there is no need to further structure this data, so I just store it completely as text with the key being the domain name.

```
set "example.com" "<!doctype html><html lang=\"en\"><head><meta charset=\"utf-8\"> ..."
```

To read this report:

```shell
get "example.com"
```

### NodeJS Wrapper

While you can execute Shell commands from within a Node.js application, I like to use the [ioredis library](https://www.npmjs.com/package/ioredis) as a small wrapper. The wrapper provides you with an object called `redis`. This object contains method for all the Redis commands, and its arguments are just string values. With this library, you can keep the simplicity of the Redis commands. Let’s see some examples.

The command to create a job hash map becomes the following:

```js
redis.hset("0b25ab16-6efd-485c-b260-1766505a3811", "domain", "http://example.com", "status", "started")
```

We can abstract the creation, updating and setting the expiration date of a job into the following function and use it throughout the project:

```js
function updateJob(uuid, details) {
  redis.hset(uuid, 'domain', details.domain, 'status', details.status);
  redis.expire(uuid, 86400);
}
```

Reading a job cannot get simpler than this:

```js
redis.hgetall(uuid);
```

## Conclusion

Redis is a fast and efficient in-memory database that support a wide variety of data formats. With easy to learn and to apply commands you structure data as it best fits your use case. All commands are executed immediately, thus making a microservice working with Redis absolutely stateless. Redis offers advanced modules to support graph data structure or timeseries and event streams. Take a look at Redis for your next project.