Benchmarking Redis with k6
Tags: k6, redis, javascript, testing
Published: 2021-08-18 01:47:15

Previously, I have covered an article on [Load Testing SQL Databases with k6](https://k6.io/blog/load-testing-sql-databases-with-k6/). For your information, from k6 version 0.29.0 onwards, you can [write a k6 Go extension](https://github.com/grafana/xk6) and build your own k6 binaries. This comes in handy as you can use a single framework for load testing different protocols, such as ZMTQ, SQL, Avro, MLLP, etc.

In this series of k6 extensions, let’s benchmark Redis now. According to [redis.io](https://redis.io/), Redis is a type of in-memory data structure store that can be used as database, cache and message broker.

You might want to evaluate the performance or scalability of Redis instances in given hardware, giving you better insights into the throughput the Redis service can handle. 

This tutorial covers Redis performance testing via two different approaches on a Linux machine:
- redis-benchmark
- xk6-redis


## redis-benchmark

By default, Redis comes with its own benchmark utility called [redis-benchmark](https://redis.io/topics/benchmarks). It is similar to Apache's ab utility and can simulate a number of clients sending a total number of queries simultaneously.

### Options

Make sure that you have Redis installed in your system. If you have not done so, kindly head over to the [official Redis download page](https://redis.io/download) and install it based on the instructions given.

Once you are done with it, you should be able to run the following command:

```bash
redis-benchmark --help
```

You should see the following output:

```bash
Usage: redis-benchmark [-h <host>] [-p <port>] [-c <clients>] [-n <requests>] [-k <boolean>]

 -h <hostname>  	Server hostname (default 127.0.0.1)
 -p <port>      	Server port (default 6379)
 -s <socket>    	Server socket (overrides host and port)
 -a <password>  	Password for Redis Auth
 -c <clients>   	Number of parallel connections (default 50)
 -n <requests>  	Total number of requests (default 100000)
 -d <size>      	Data size of SET/GET value in bytes (default 3)
 --dbnum <db>   	SELECT the specified db number (default 0)
 -k <boolean>   	1=keep alive 0=reconnect (default 1)
 -r <keyspacelen>   Use random keys for SET/GET/INCR, random values for SADD
  Using this option the benchmark will expand the string __rand_int__
  inside an argument with a 12 digits number in the specified range
  from 0 to keyspacelen-1. The substitution changes every time a command
  is executed. Default tests use this to hit random keys in the
  specified range.
 -P <numreq>    	Pipeline <numreq> requests. Default 1 (no pipeline).
 -e             	If server replies with errors, show them on stdout.
                	(no more than 1 error per second is displayed)
 -q             	Quiet. Just show query/sec values
 --csv          	Output in CSV format
 -l             	Loop. Run the tests forever
 -t <tests>     	Only run the comma separated list of tests. The test
                	names are the same as the ones produced as output.
 -I             	Idle mode. Just open N idle connections and wait.
```

### Examples

Depending on your needs, a typical example is to just run the benchmark with the default configuration:

```bash
redis-benchmark
```

It is a good idea to use the `-q` option. Here is an example for running 100k of requests in quiet mode:

```bash
redis-benchmark -q -n 100000
```

In addition, you can run parallel clients via the `-c` option. The following example use 20 parallel clients for a total of 100k requests:

```bash
redis-benchmark -q -n 100000 -c 20
```

You can restrict the test to run only a subset of the commands. For example, you can use the following command to test only set and get commands:

```bash
redis-benchmark -q -t set,get -n 100000
```

In fact, you can run test on specific commands for benchmarking like the following example: 

```bash
redis-benchmark -q -n 100000 script load "redis.call('set','key','value')"
```

If your Redis server is running on a different hostname and port, you can benchmark the server as follows:

```bash
redis-benchmark -h 192.168.1.1 -p 6379 -n 100000 -c 20
```

You should get the following output indicating the requests per second for each of the test conducted:

```bash
PING_INLINE: 43478.26 requests per second
PING_BULK: 41666.67 requests per second
SET: 43478.26 requests per second
GET: 43478.26 requests per second
INCR: 40000.00 requests per second
LPUSH: 43478.26 requests per second
RPUSH: 37037.04 requests per second
LPOP: 45454.55 requests per second
RPOP: 34482.76 requests per second
SADD: 43478.26 requests per second
HSET: 45454.55 requests per second
SPOP: 45454.55 requests per second
LPUSH (needed to benchmark LRANGE): 40000.00 requests per second
LRANGE_100 (first 100 elements): 45454.55 requests per second
LRANGE_300 (first 300 elements): 43478.26 requests per second
LRANGE_500 (first 450 elements): 47619.05 requests per second
LRANGE_600 (first 600 elements): 38461.54 requests per second
MSET (10 keys): 41666.67 requests per second
```


### Latency

Sometimes, you might prefer to analyze the latency instead. There are two types of [latency measurement](https://redis.io/topics/latency) provided by redis-cli:
- latency
- intrinsic latency

In this case, we measure latency as the time between sending a request to Redis and receiving a response. On the other hand, intrinsic latency refers to the system latency that is highly dependent on external factors such as operating system kernel or virtualization. Since Redis 2.8.7, you can measure the intrinsic latency independently.

Please note that you can only run redis-cli in the machine which hosts the Redis server unlike redis-benchmark which is runnable on the client machine. Besides that, this mode is not connected to a Redis server at all and the measurement is based on the largest time in which the kernel does not provide CPU time to run to the redis-cli process itself. As a result, it is not an actual measurement of the latency between client and Redis server.

Having said that, it does provide a quick analysis if there is something wrong with the machine that hosts the Redis server.

Run the following command to get the overall latency of your Redis server:

```bash
redis-cli --latency
```

You should see an increase in the sample as time goes by and the average latency:

```bash
min: 0, max: 5, avg: 0.22 (2406 samples)
```

Use `Ctrl+C` to stop it as the process will run indefinitely without stopping.

For intrinsic latency, you should use the following command instead:

```bash
redis-cli --intrinsic-latency 10
```

You can pass an integer representing the duration of the test. In this case, the test will run for 10 seconds. The output is as follows:

```bash
Max latency so far: 1 microseconds.
Max latency so far: 15 microseconds.
Max latency so far: 16 microseconds.
Max latency so far: 17 microseconds.
Max latency so far: 18 microseconds.
Max latency so far: 20 microseconds.
Max latency so far: 21 microseconds.
Max latency so far: 24 microseconds.
Max latency so far: 25 microseconds.
Max latency so far: 50 microseconds.
Max latency so far: 74 microseconds.
Max latency so far: 87 microseconds.
Max latency so far: 150 microseconds.
Max latency so far: 1089 microseconds.
Max latency so far: 1715 microseconds.
Max latency so far: 2344 microseconds.
Max latency so far: 7438 microseconds.
Max latency so far: 8002 microseconds.

158645097 total runs (avg latency: 0.0630 microseconds / 63.03 nanoseconds per run).
Worst run took 126948x longer than the average latency.
```

The average latency is about 0.22 milliseconds while the intrinsic latency is 0.063 microseconds.

Let’s proceed to the next section and start exploring another testing approach using k6.

## xk6-redis

k6 provides the capabilities to do performance testing with scripting language. This is a big plus to developers and Q&A testers as you will have a better control of the entire workflow of the test. For example, you can ramp up or ramp down the requests at specific intervals of the test which is not achievable when using redis-benchmark.

Fortunately, k6 provides the [xk6-redis extension](https://github.com/grafana/xk6-redis) as part of their ecosystem. You can use it directly to build your own custom k6 binaries for testing Redis server.

This extension comes with the following API:

| Output                                                        | Usage                   |
| ------------------------------------------------------------- | ----------------------- |
| Client(options) | Represent the Client construtor. Returns a new Redis client object.   |
| client.set(key, value, expiration time) | Set the given key with the given value and expiration time.   |
| client.get(key) | Get returns the value for the given key.   |

### Building k6 with the redis extension 

Before that, make sure you have the following installed in your machine:
- Go
- Git

Once you have completed the installation, run the following to install xk6 module:

```bash
go install github.com/k6io/xk6/cmd/xk6@latest
```

If you have installed xk6 directory to Go module, you can make your Redis k6 build by running:

```bash
xk6 build --with github.com/k6io/xk6-redis
```

You should get a k6 executable in your current working directory.

Alternatively, you can download the pre-compiled binaries at the [following Github repository](https://github.com/grafana/xk6/releases/). The latest version at the time of this writing is v0.4.1. If you have trouble identifying the architecture of your Linux machine, simply run the following command:

```bash
dpkg --print-architecture
```

Let’s say that the command returns the following:

```bash
amd64
```

You should download the xk6_0.4.1_linux_amd64.tar.gz asset and extract it as follows:

```bash
tar -xvf xk6_0.4.1_linux_amd64.tar.gz
```

You should get the following files in your working directory:
- README.md
- LICENSE
- xk6

Then, run the following command to build k6 for Redis: 

```bash
./xk6 build --with github.com/k6io/xk6-redis
```

You should have now a new k6 binary in your working directory.

### k6 Script

Next, let’s create a new JavaScript file called test_script.js in the same directory as your k6 executable. Append the following import statement at the top of the file:

```javascript
import redis from 'k6/x/redis';
```

Continue by adding the following code which connect to your Redis server:

```javascript
const client = new redis.Client({
  addr: 'localhost:6379',
  password: '',
  db: 0,
});
```

It accepts the following an object with the following fields:
- addr: hostname and port of your Redis server denoted as hostname:port.
- password: password of your Redis server.
- db: the db number ranging from 0 to 15.

To keep it simple and short, the test case is going to be as follows:
- Set a new key:value on start of test.
- Running parallel VUs to get the same key repeatedly.

The k6 setup function runs only once at the test start, independently of the test load and duration. Let’s set the key: value as follows: 

```javascript
export function setup() {
  client.set('key', 'value', 0);
}
```

The set function accepts three input parameters:
- key
- value
- expiration time

Then, define the default function which will be called repeatedly by each VU during the entire test:

```javascript
export default function () {
  client.get('key');
}
```

The complete code is as follows:

```javascript
import redis from 'k6/x/redis';
import { check } from 'k6';

const client = new redis.Client({
  addr: 'localhost:6379',
  password: '',
  db: 0,
});

export function setup() {
  client.set('key', 'value', 0);
}

export default function () {
  client.get('key');
}
```

### Running the test

Save the test script and run the following command to the test your Redis server for 5 seconds:

```bash
./k6 run test_script.js --duration 5s
```

By default, it is using one Virtual User (VU) but you can modify it with the `--vus` flag. You should see the following output:


```bash
          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
 	script: test_script.js
 	output: -

  scenarios: (100.00%) 1 scenario, 1 max VUs, 35s max duration (incl. graceful stop):
       	* default: 1 looping VUs for 5s (gracefulStop: 30s)


running (05.0s), 0/1 VUs, 42037 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs  5s

 	█ setup

 	data_received........: 0 B   0 B/s
 	data_sent............: 0 B   0 B/s
 	iteration_duration...: avg=104.45µs min=53.7µs med=88.6µs max=9.32ms p(90)=115.4µs p(95)=129.5µs
 	iterations...........: 42037 8401.691798/s
 	vus..................: 1 	min=1     	max=1
 	vus_max..............: 1 	min=1     	max=1

```

This test reports that the Redis server handles 8401 iterations per second.  Because each iteration refers to one execution of the default function and there is one request call in our default function, the server is handling 8401 GET requests per second in this test.

### Scale the load

Let’s increase the load gradually until it encounters an error. For a start, set the VUs to 100 as follows:

```bash
./k6 run test_script.js --duration 5s --vus 100
```

The output is as follows:

```bash
running (05.0s), 000/100 VUs, 111939 complete and 0 interrupted iterations
default ↓ [======================================] 100 VUs  5s

 	█ setup

 	data_received........: 0 B	0 B/s
 	data_sent............: 0 B	0 B/s
 	iteration_duration...: avg=4.39ms min=46.8µs med=3.32ms max=87.24ms p(90)=9.5ms p(95)=12.51ms
 	iterations...........: 111939 22304.954101/s
 	vus..................: 100	min=100    	max=100
 	vus_max..............: 100	min=100    	max=100
```

It indicates that your Redis server can sustain about 22304 iterations per second for 100 users at the same time.

Continue the test and set the VUs to 1000 this time:

```bash
./k6 run test_script.js --duration 5s --vus 1000
```

Depending on the configuration of your Redis, you might encounter the following error:

```bash
ERRO[0003] ERR max number of clients reached
running at go.k6.io/k6/js/common.Bind.func1 (native)
default at file:///home/wfng/test_script.js:14:14(4)  executor=constant-vus scenario=default source=stacktrace
```

It indicates that you have reached the max number of clients allowed. You can check the number of active connection by running the following command inside redis-cli:

```bash
info clients
```


It will returns the following output:

```bash
# Clients
connected_clients:7
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
```


To get the max limit, use the following instead:

```bash
config get maxclients
```


The output is as follows:

```bash
1) "maxclients"
2) "500"
```


### Latency

Now, let’s have a look at how to get the latency via k6. At the time of this writing, the xk6-redis extension does not report latency as part of its metrics. However, you can easily extend the code in your script and implement your own [custom metrics](https://k6.io/docs/using-k6/metrics/#custom-metrics).

Have a look at the following workaround to measure latency. First, let’s add the following import statement at the top of your k6 script:

```javascript
import { Trend } from 'k6/metrics';
```

Then, initialize a Trend instance as follows:

```javascript
let RedisLatencyMetric = new Trend('redis_latency', true);
```

It accepts two input arguments:
  - `name`: the name of the custom metric.
  - `isTime`: a boolean indicating whether the values added to the metric are time values or just untyped values.


Add the final touch by modifying the default function as follows:

```javascript
export default function () {
  const start = Date.now();
  client.get('key');
  const latency = Date.now() - start;
  RedisLatencyMetric.add(latency);
}
```

Have a look at the following complete code which initialize the options directly inside the script:

```javascript
import { Trend } from 'k6/metrics';
import redis from 'k6/x/redis';

let RedisLatencyMetric = new Trend('redis_latency', true);

export let options = {
  vus: 40,
  duration: '10s',
}

const client = new redis.Client({
  addr: 'localhost:6379',
  password: '',
  db: 0,
});

export function setup() {
  client.set('key', 'value', 0);
}

export default function () {
  const start = Date.now();
  client.get('key');
  const latency = Date.now() - start;
  RedisLatencyMetric.add(latency);
}
```

You should be able to see redis_latency metrics once the test has completed.

```bash
iteration_duration...: avg=782.57µs min=67.35µs med=732.92µs max=15.86ms p(90)=1.1ms p(95)=1.3ms
iterations...........: 506755 50660.636169/s
redis_latency........: avg=764.8µs  min=0s  	med=1ms  	max=16ms	p(90)=1ms   p(95)=1ms
```

> ⚠️  Please note that this workaround of measuring the latency is only indicative, as the JavaScript implementation adds an overhead that might skew the reported latency, especially when the latency is in the sub-microsecond range. 
> 
> It would be great if the xk6-redis extension provided its own built-in Redis latency metrics similar to the HTTP request metrics. Measuring Redis latency in Go directly would be much more accurate and avoid the unnecessary RedisLatencyMetric script code.

## Conclusion

All in all, redis-benchmark is a good tool that provides you with a quick glimpse of the performance of your Redis server. On the other hand, k6 is scriptable in JavaScript and can provide you with better control over the execution and workflow of your test. A scripting language is more flexible for testing various ways to connect and query your Redis server.

In fact, you can utilize both of the tools to get the best out of them. For example, you can run redis-benchmark when you install it on your machine for the first time, to get a rough idea of the performance. Subsequently, use k6 for more advanced cases like integrating your test with your existing toolbox or automating your testing.