Intro to Redis Cluster Sharding – Advantages, Limitations, Deploying & Client Connections
Tags: redis, rediscluster, sharding, database
Published: 2019-02-07 18:33:44

---
title: Intro to Redis Cluster Sharding – Advantages, Limitations, Deploying & Client Connections
published: true
description: In this post, you'll learn about Redis Cluster sharding opportunities, the advantages and limitations, when to deploy, and how to connect to your Redis Cluster.
tags: Redis, Redis Cluster, Sharding, Database,
canonical_url: https://scalegrid.io/blog/intro-to-redis-cluster-sharding-advantages-limitations-deploying-and-client-connections/
---
<link rel="canonical" href="https://scalegrid.io/blog/intro-to-redis-cluster-sharding-advantages-limitations-deploying-and-client-connections/" />

<a href="https://scalegrid.io/blog/intro-to-redis-cluster-sharding-advantages-limitations-deploying-and-client-connections/"><img class="aligncenter size-full wp-image-3885" src="https://scalegrid.io/blog/wp-content/uploads/2019/02/Intro-to-Redis-Cluster-Sharding-Advantages-Limitations-When-To-Deploy-and-Client-Connections-ScaleGrid.jpg" alt="Intro to Redis Cluster Sharding" width="100%" height="auto" /></a>

<p style="text-align: justify;"><a style="color: #2da964;" title="Highly Available Redis Clusters with Automated Sharding Launches at ScaleGrid" href="https://scalegrid.io/blog/highly-available-redis-clusters-with-automated-sharding-launches-at-scalegrid" target="_blank">Redis Cluster</a> is the native sharding implementation available within Redis that allows you to automatically distribute your data across multiple nodes without having to rely on external tools and utilities. At ScaleGrid, we recently added support for Redis Clusters on our platform through our fully managed <a style="color: #2da964;" title="Redis Hosting" href="https://scalegrid.io/redis.html" target="_blank">Redis hosting</a> plans. In this post, we're going to introduce you to the advanced <a href="https://scalegrid.io/blog/intro-to-redis-cluster-sharding-advantages-limitations-deploying-and-client-connections/">Redis Cluster sharding</a> opportunities, discuss its advantages and limitations, when you should deploy, and how to connect to your Redis Cluster.</p>

<h2>Sharding with Redis Cluster</h2>
<p style="text-align: justify;">The entire keyspace in Redis Clusters is divided in 16384 slots (called hash slots) and these slots are assigned to multiple Redis nodes. A given key is mapped to one of these slots, and the hash slot for a key is <a style="color: #2da964;" href="https://redis.io/topics/cluster-spec#keys-distribution-model" target="_blank">computed</a> as:</p>
<tt>HASH_SLOT = CRC16(key) mod 16384</tt>
<p style="text-align: justify;">Multi-key operations are supported on Redis Clusters as long as all the keys involved in a single command execution belong to the same hash slot. This can be ensured using the concept of <a style="color: #2da964;" href="https://redis.io/topics/cluster-spec#keys-hash-tags" target="_blank">hash tags</a>.</p>
<p style="text-align: justify;">The <a style="color: #2da964;" href="https://redis.io/topics/cluster-spec" target="_blank">Redis Cluster Specification</a> is the definitive guide to understanding the internals of the technology, while the <a style="color: #2da964;" href="https://redis.io/topics/cluster-tutorial" target="_blank">Redis Cluster Tutorial </a>provides deployment and administration guidelines.</p>

<h2>Advantages</h2>
<p style="text-align: justify;">Check out the top advantages of Redis Clusters to see how it can benefit your deployments:</p>

<ul>
	<li style="text-align: justify;">
<h3>High Performance</h3>
Redis Cluster promises the same level of performance as standalone Redis deployments.</li>
	<li style="text-align: justify;">
<h3>High Availability</h3>
Redis Cluster supports the standard Redis master-replica configuration to ensure high availability and durability. It also implements a <a style="color: #2da964;" href="https://en.wikipedia.org/wiki/Raft_(computer_science)" target="_blank">Raft</a>-like <a style="color: #2da964;" href="https://redis.io/topics/cluster-spec#configuration-handling-propagation-and-failovers" target="_blank">consensus approach</a> to ensure availability of the entire cluster.</li>
	<li style="text-align: justify;">
<h3>Horizontal &amp; Vertical Scalability</h3>
Adding or removing new Redis nodes from the Cluster can occur transparently and without any downtime. This makes adding and removing shards, retiring, or scaling individual nodes easy.</li>
	<li style="text-align: justify;">
<h3>Native Solution</h3>
Deploying Redis Clusters requires no external proxies or tools, so there are no new tools you need to learn or worry about. It also provides almost complete compatibility with standalone Redis deployments.</li>
</ul>
<h2>Limitations</h2>
Let's also be aware of the limitations to ensure it's right for your deployment:
<ul>
	<li style="text-align: justify;">
<h3>Requires Client Support<strong>
</strong></h3>
Clients need to make changes in order to support Redis Clusters. While Redis Clusters have been around for many years now, there are still clients that don't support it. Refer to the <a style="color: #2da964;" href="https://redis.io/clients" target="_blank">Redis client</a> documentation to ensure the client you use is supported before choosing to deploy Redis Clusters.</li>
	<li style="text-align: justify;">
<h3>Limited Multi-Key Operation Support<strong>
</strong></h3>
As mentioned in the previous section, multi-key operations are supported only when all the keys in a single operation belong to the same slot. This is something to be careful about when designing your data structures.</li>
	<li style="text-align: justify;">
<h3>Only Supports One Database<strong>
</strong></h3>
Unlike standalone databases, Redis Clusters support only one database (database 0), and the <a style="color: #2da964;" href="https://redis.io/topics/cluster-spec#implemented-subset" target="_blank">SELECT </a>command isn't allowed. Since most people don't use multiple databases, this too is not a major limitation.</li>
</ul>
<h2>When Should You Deploy a Redis Cluster?</h2>
<p style="text-align: justify;">The Redis Cluster solution can be a good fit for you if you need a sharded Redis solution. Redis Cluster is a native solution that is simple and offers great performance.</p>
<p style="text-align: justify;">Typically, people start looking at sharding their Redis deployments when they've started to saturate a standalone Redis node with writes and want to spread writes out to multiple nodes. Even though Redis is primarily <a style="color: #2da964;" href="https://redis.io/topics/faq#redis-is-single-threaded-how-can-i-exploit-multiple-cpu--cores" target="_blank">single threaded</a>, I/O typically becomes network or memory bound on a standalone before it can start saturating the CPU. Memory bound can be overcome to an extent by choosing to add more memory to a standalone system, but it starts to become prohibitive in terms of cost, backup, restart, warm up times, etc. beyond a certain point.</p>
<p style="text-align: justify;">On the other hand, if you're looking only to spread your read across multiple nodes, it's much easier to just add read replicas to the standalone.</p>
<p style="text-align: justify;">When compared to other sharding solutions for Redis, shard rebalancing in Redis Clusters is transparent to applications. This makes it very easy to add or remove shards without effecting the application.</p>


<h2>Connecting To a Redis Cluster</h2>
<p style="text-align: justify;">If you choose to deploy a Redis Cluster with ScaleGrid, you get a fully featured Redis Cluster deployment that's totally compatible with the standard release.</p>
<p style="text-align: justify;">If you're just starting out, sign up for a <a style="color: #2da964;" href="https://console.scalegrid.io/users/register" target="_blank">free 30-day trial</a> on the ScaleGrid console, and check out this documentation on <a style="color: #2da964;" title="Create a Redis Hosting Deployment" href="https://help.scalegrid.io/docs/redis-new-cluster" target="_blank">creating your first Redis deployment</a>.</p>
<p style="text-align: justify;">Here's what you need in order to connect to the Redis Cluster:</p>

<ul>
	<li>List of node names</li>
	<li>Ports</li>
	<li>Authentication string</li>
</ul>
<p style="text-align: justify;">The Overview tab of your Redis deployment details page has the list of masters of each shard, along with port numbers and authentication information:</p>
<a href="https://scalegrid.io/blog/wp-content/uploads/2019/01/RedisClusConnString.png"><img class="aligncenter size-full wp-image-3885" src="https://scalegrid.io/blog/wp-content/uploads/2019/01/RedisClusConnString.png" alt="RedisClusConnString" width="1495" height="323" /></a>
<p style="text-align: justify;">Alternatively, the list of all the nodes of the cluster is available on the Machines tab:</p>
<a href="https://scalegrid.io/blog/wp-content/uploads/2019/01/RedisMachinesList.png"><img class="aligncenter size-full wp-image-3886" src="https://scalegrid.io/blog/wp-content/uploads/2019/01/RedisMachinesList.png" alt="RedisMachinesList" width="1615" height="631" /></a>
<p style="text-align: justify;">As you may have noticed, not a lot of examples are available showing you how to connect to an authentication-enabled Redis Cluster available online, here are a few examples using some of the popular clients.</p>

<h3>Connecting With Java</h3>
<p style="text-align: justify;">Among the popular Redis <a style="color: #2da964;" href="https://redis.io/clients#java" target="_blank">Java clients</a>, <a style="color: #2da964;" href="https://github.com/xetorthio/jedis" target="_blank">Jedis </a>and <a style="color: #2da964;" href="https://github.com/lettuce-io/lettuce-core" target="_blank">Lettuce</a> support Redis Clusters. We will take up Jedis for our example.</p>

<h4>Jedis</h4>
<p style="text-align: justify;">Redis Cluster connections are abstracted by the <a style="color: #2da964;" href="https://github.com/xetorthio/jedis#jedis-cluster" target="_blank"><tt>JedisCluster</tt></a> class. The best examples of using this class to connect to Redis Clusters can be found in the Jedis tests, <a style="color: #2da964;" href="https://github.com/xetorthio/jedis/blob/master/src/test/java/redis/clients/jedis/tests/JedisClusterTest.java" target="_blank">Jedis source code</a>. Unfortunately at the this time, when authentication is specified, the <tt>JedisCluster</tt> constructor is not very clean. Here's an example which writes 100 keys to the Redis Cluster. Note that since the keys aren't tagged, they will end up in different slots on different nodes:</p>

<pre>...
import java.util.HashSet;
import java.util.Set;
import redis.clients.jedis.HostAndPort;
import redis.clients.jedis.JedisCluster;
import redis.clients.jedis.JedisPoolConfig;
...

public class RedisClusterModeTest {
    public static final int    DEFAULT_TIMEOUT      = 5000;
    public static final int    DEFAULT_REDIRECTIONS = 5;

    public static void main(String[] args) {
        Set jedisClusterNodes = new HashSet();
        jedisClusterNodes.add(new HostAndPort("SG-example-1.servers.scalegrid.io, 6379));
        jedisClusterNodes.add(new HostAndPort("SG-example-2.servers.scalegrid.io", 6379));
        jedisClusterNodes.add(new HostAndPort("SG-example-3.servers.scalegrid.io", 6379));
        jedisClusterNodes.add(new HostAndPort("SG-example-4.servers.scalegrid.io", 6379));
        jedisClusterNodes.add(new HostAndPort("SG-example-5.servers.scalegrid.io", 6379));
        jedisClusterNodes.add(new HostAndPort("SG-example-6.servers.scalegrid.io", 6379));

        JedisCluster jedis = new JedisCluster(jedisClusterNodes, DEFAULT_TIMEOUT, DEFAULT_TIMEOUT, DEFAULT_REDIRECTIONS, &lt;auth&gt;, new JedisPoolConfig());
        for (int i = 0; i &lt; 100; i++) {
            jedis.set("key" + i, "value" + i);
        }
        jedis.close();
    }
}
</pre>
<p style="text-align: justify;">The arguments to the constructor are documented in the <a style="color: #2da964;" href="https://www.javadoc.io/doc/redis.clients/jedis/3.0.1" target="_blank">Jedis API docs</a>. We recommend that you specify all the nodes of the cluster during cluster creation with Jedis.</p>

<h3>Connecting With Ruby</h3>
<p style="text-align: justify;">The most popular <a style="color: #2da964;" href="https://redis.io/clients#ruby" target="_blank">Redis client in Ruby</a> is <a style="color: #2da964;" href="https://github.com/redis/redis-rb" target="_blank">redis-rb</a>. It also supports Redis Clusters so we'll use it in our example.</p>

<h4>Redis-rb</h4>
<p style="text-align: justify;">redis-rb versions <a style="color: #2da964;" href="https://github.com/redis/redis-rb/wiki/Cluster-mode" target="_blank">4.1.0 and above</a> have support for Redis Clusters. The <tt>'cluster'</tt> option needs to be specified during connection initialization, and you can refer to this <a style="color: #2da964;" href="https://www.rubydoc.info/gems/redis/Redis:initialize" target="_blank">documentation</a> for exact semantics. Here's the same program as the Java example above in Ruby:</p>

<pre>require 'redis'
require 'pp'

NODES = ["redis://SG-example-1.servers.scalegrid.io:6379",
         "redis://SG-example-2.servers.scalegrid.io:6379",
         "redis://SG-example-3.servers.scalegrid.io:6379",
         "redis://SG-example-4.servers.scalegrid.io:6379",
         "redis://SG-example-5.servers.scalegrid.io:6379",
         "redis://SG-example-6.servers.scalegrid.io:6379"]
begin
    pp "Attempting connection..."
    redis = Redis.new(cluster: NODES, password: &lt;auth&gt;)
    100.times { |i| redis.set("key#{i}", "value#{i}") }
    pp "Done..."
    redis.close
rescue StandardError =&gt; e
    puts e.message
end
</pre>
<h3>Connecting With Node.js</h3>
<p style="text-align: justify;"><a style="color: #2da964;" href="https://github.com/NodeRedis/node_redis" target="_blank">Node_redis</a> is the most popular <a style="color: #2da964;" href="https://redis.io/clients#nodejs" target="_blank">Redis client in Node.js</a>. However, it doesn't <a style="color: #2da964;" href="https://github.com/NodeRedis/node_redis/issues/574" target="_blank">officially support</a> Redis Clusters yet. <a style="color: #2da964;" href="https://github.com/luin/ioredis" target="_blank">ioredis</a> is another popular Redis client which has Redis Clusters support so we'll use this for our Node.js example.</p>

<h4>ioredis</h4>
<p style="text-align: justify;">The ioredis <a style="color: #2da964;" href="https://github.com/luin/ioredis/blob/master/API.md#Cluster" target="_blank">documentation</a> describes the details of the additional parameters that must be passed for connecting to Redis Clusters, and a basic example is also provided in the <a style="color: #2da964;" href="https://github.com/luin/ioredis#cluster">README</a>. Here's an example program which prompts the user for a key and read its value from the Redis Cluster:</p>

<pre>const readline = require('readline');
const Redis = require('ioredis');

var cluster = new Redis.Cluster([{
    port: 6379,
    host: 'SG-example-1.servers.scalegrid.io'
},
{
    port: 6379,
    host: 'SG-example-2.servers.scalegrid.io'
},
{
    port: 6379,
    host: 'SG-example-3.servers.scalegrid.io'
},
{
    port: 6379,
    host: 'SG-example-4.servers.scalegrid.io'
},
{
    port: 6379,
    host: 'SG-example-5.servers.scalegrid.io'
},
{
    port: 6379,
    host: 'SG-example-6.servers.scalegrid.io'
}
], { redisOptions: { password: '&lt;auth&gt;' } });

const rl = readline.createInterface({
    input: process.stdin,
    output: process.stdout,
    prompt: 'enter key&gt; '
});

console.log('Welcome to the Redis Cluster reader. Enter the key which you want to read [Ctrl D to Exit]');
rl.prompt();
rl.on('line', (line) =&gt; {
    if (line.trim()) {
        cluster.get(line, function (err, result) {
            if (err) {
                console.error(err);
            } else {
                console.log("value: " + result);
            }
            rl.prompt();
        });
    } else {
        console.error("No input received");
        rl.prompt();
    }
}).on('close', () =&gt; {
    console.log('\nterminating');
    cluster.quit();
    process.exit(0);
});
</pre>
<p style="text-align: justify;">You should be able to execute any of these examples after installing the most recent versions of the Redis drivers on your client machines.</p>
<p style="text-align: justify; padding-bottom:20px;">If you're ready to upgrade your Redis deployments to fully managed <a style="color: #2da964;" title="Redis hosting" href="https://scalegrid.io/redis.html" target="_blank">Redis hosting</a>, check out the amazing features available on the ScaleGrid platform through a free 30-day trial. Our <a style="color: #2da964;" href="https://scalegrid.io/redis/aws.html" target="_blank">Redis on AWS</a> plans are available across <a style="color: #2da964;" href="https://help.scalegrid.io/docs/redis-data-centers" target="_blank">14 different datacenters worldwide</a>, and we're the only Redis service that allows you to manage your deployments within your own cloud account.</p>