Go and Redis, Better Together 🤝
Tags: redis, go, programming, database
Published: 2023-04-24 07:18:21


I recently presented this talk at the [Conf42 Golang 2023](https://www.conf42.com/golang2023) and I thought it might be a good idea to turn it into a blog post for folks who don't want to spend 40+ mins watching the talk (it's ok, I understand 😉) or just staring at slides trying to imagine what I was saying. 

So here you go....

> By the way, you are still welcome to [watch the talk](https://www.youtube.com/watch?v=IfL6J_6el7E&list=PLIuxSyKxlQrBDOCy9ThDZ0kJX7skLE2ks) or [download the slides](https://speakerdeck.com/abhirockzz/go-and-redis-better-together)! 🙌 There are a lot of great talks that you can get [from this playlist](https://www.youtube.com/playlist?list=PLIuxSyKxlQrBDOCy9ThDZ0kJX7skLE2ks).

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0xebphpxo4fkao1ger42.png)

This talk was geared towards folks who are looking to get started with Redis and Go. Or perhaps you are already experienced with both these topics – in that case, it might be a good refresher!

To that extent, I had a very simple agenda.

- I started off by setting the context about Redis and Go
- Provide an overview of Go and Redis ecosystem, including the client options you’ve got..
- Followed by some hand-on stuff and,
- Wrap with some tips/tricks and resources.

## I love Redis and Go!

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/p5esx75b9if6a6hq3wnc.png)

Since its release in 2009, it did not take Redis too long to win the hearts and minds of the developer community! As per [db-engines trends](https://db-engines.com/en/ranking_trend/key-value+store) statistics, Redis has been topping the charts since 2013. And on [Stack Overflow annual survey](https://insights.stackoverflow.com/survey/2021#technology-most-loved-dreaded-and-wanted), it’s been voted most loved database for 5 years in a row.


Go has become the language of the Cloud. It powers many cloud-native projects (apparently, [75% of CNCF projects are written in Go](https://go.dev/solutions/cloud#key-benefits)) including Docker, Kubernetes, Prometheus, Terraform etc. In fact there many databases written in Go - like Influxdb, etcd, Vitess, TiDB etc. 

Go also caters to a wide variety of general purpose use cases:

- Web apps and APIs, 
- Data processing pipelines, 
- Infrastructure as code, 
- SRE/DevOps solutions, 
- Command line apps (this is a really popular one!)
- and more...

No wonder Go has become such [a popular programming language](https://survey.stackoverflow.co/2022/#technology-most-loved-dreaded-and-wanted%20%20https://survey.stackoverflow.co/2022/)!

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fjndjz1szif0c1hku4ch.png)

Now you might be thinking - *"Hey Go is down at the bottom."* But if you notice carefully, it is the only statically-typed lang after Rust (of course there are C# and Kotlin down there as well) And this is from 2022 - if you look at data from 2021 to 2018, you will notice that Go has maintained its **top 5 spot**.

Go and Redis have a few things in common, but to me "Simplicity" is the one that really stands out to me.

## Simplicity

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pffoy7svglro6xj38qi1.png)

Redis is a key value store, but the values can be [any of these data structures](https://redis.io/commands/) that you see. These are all data structures that we as developers use every day - lists, sets, maps etc. Redis just feels like an extension to these core programming language constructs.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/06vlh7sg1i63a45zm280.png)

With Go, this comes in various forms…

- Excellent tooling, 
- Or a comprehensive standard library,
- Easy to use concurrency primitives...

And sometimes it's in the form of *not* bloating the language with unnecessary features. To cite an example, it took a while for generics to be added to the language. Now I am not trying to trick you into thinking that Go is simple.. Or for that matter any programming language is simple.

But with Go, the goal is to give you simple components to help build complex things, and **hide** complexity behind a simple facade. 

And there are folks who have explained this in great detail (and much better than I can!). I would really encourage you to [check out this talk](https://www.youtube.com/watch?v=rFejpH_tAHM) by Rob Pike (and [the slides](https://go.dev/talks/2015/simplicity-is-complicated.slide#1)), the co-creator of Go (its from 2015 but still very much applicable to the essence and spirit of Go).

## Redis 101

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/73c5lyu0gb3wt5t4nwoc.png)

Quick intro to Redis (some of the key points):

- Data structure server - At its core, Redis is nothing but a key value store, where the key can be `string` or even a `binary`. The important thing to note is that as far as the value is concerned, you can choose from a variety of data structures such as `string`s,` hash`es, `list`s, `set`s etc.
- Redis is **not** just a cache - It’s a really solid messaging platform as well
- HA - You can configure redis to be highly avaialbe by setting up primary-replica replication, or take it a step further with Redis Cluster.
- Persistence: Redis is primarily in-memory but you can configure it to persist to disk as well. But there are solutions like [Amazon MemoryDB](https://docs.aws.amazon.com/memorydb/latest/devguide/what-is-memorydb-for-redis.html) that can actually take it a notch further (thanks to it's [distributed transactional log](https://aws.amazon.com/memorydb/faqs/#Performance_and_durability))
- Since its open-source and wildy popular, you can get offerings from pretty much every cloud provider – big or small or even run it on Kubernetes, on cloud, on-prem or hybrid mode. If you want to put Redis in production, you can be rest assured that there are no dearth of options for you to run and operate it.

### Redis data types

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hfvb8zekckswkb3rm8e3.png)

What you see here is a list of core data structures:

- A `string` seems really simple, but are quite powerful. They can be used for something as simple as storing a key-value pair to implementing advanced patterns like distributed locking and rate-limiting
- A `hash` is similar to a `map` in Java, or `dictionary` in Python. It is used to store object like structures like user profile, customer info etc.
- A `set` behaves like its mathematical version - helps you maintain unique set of items along with ability to list them, count them, execute unions, intersection and so on.
- Think of `sorted set`s are like this big brother to a `set` - makes it possible to implement things like leaderboards which is very useful in areas like gaming. For e.g. you can store player scores in a sorted set and when you need to get the top 10 players, you can simply invoke specific sorted set commands and get that data. The beauty is that sorting happens on the database side, so there is no client side logic you need to apply.
- `List`s are a really versatile data structure as well. You can use them to store many things, but using it as a worker queue is very common. 
There are popular open source solution such as `sidekiq` and `celery` that already support Redis as a backend for job queuing solutions.
- [Redis streams](https://redis.io/docs/data-types/streams/) (added in Redis 5) is used for streaming use-cases
- And also ephemeral messaging with `pub/sub` - it's a publish-broadcast mechanism where you can send/receive messages to/from channels.
- There is also `Geospatial` data structure and a really cool one called `Hyperloglog` which is an alternative to a traditional `set`. It can store millions of items while optimizing for data storage and you can count the number of unique items with really high accuracy.

## Go clients for Redis

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9vhk6qhl36d98i249qu8.png)

👆🏼👆🏼 These are the most popular Go clients for Redis.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7wlndjib1roxamca3cao.png)

`go-redis` is by far the most popular client. It has what you’d expect.. Features, decent documentation, active community. Moving this under the [official Redis GitHub org](https://uptrace.dev/blog/go-redis-v9.html) is just icing on the cake!

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gcg204h5xqniqaqr636d.png)


`redigo` is a fairly stable and tenured client that supports all the standard Redis data types and features such as transactions, pipelining etc. 
It is also used to implement other Go client libraries such as `redisearch-go` and Redis `TimeSeries` Go client.
That being said, its API is a bit too flexible. While some may prefer that, but I feel that it’s not a good fit when I am using a type-safe language like Go (that’s just my personal opinion).

But the biggest drawback to me is that it does not support Redis Cluster!

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/93qx8e3kodml5z4kedcm.png)

`rueidis` is *relatively* new (at the time of writing) but quickly evolving client library. It supports [RESP3 protocol](https://redis.io/docs/reference/protocol-spec/), client-side caching and supports a variety of Redis Modules. As far as the API is concerned, this client adopts an interesting approach. It provides `a Do function` as well (like `redigo` client), but the way it creates the command is via a *builder* pattern - this retains strong type checking (unlike `redigo`).

To be honest with you, I haven't used this client a lot, and its looks like its packed with a lot of features. So, I cant complain much as of now!

> For those who are looking for deeper performance numbers – this [benchmark comparison](https://github.com/rueian/rueidis#benchmark-comparison-with-go-redis-v9) with the `go-redis` library might be interesting.


## Client ecosystem

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fhto27q0onjqhlr1yymp.png)

Now lets take a look from an ecosystem perspective. To clarify, the point here is not chest thumping based on GitHub stars - its just to give you a sense of things. 

For folks not using Go and Redis, the popularity of the Go client might come as a surprise. Java workloads form a huge chunk of the Redis workloads and **Jedis** is the bread and butter client when it comes to Java apps with Redis (and it's pretty old). But I was surprised to [redisson](https://github.com/redisson/redisson) topping the charts, followed by `go-redis` (yay!), (two) [Nodejs](https://www.npmjs.com/package/redis) [clients](https://www.npmjs.com/package/ioredis), [Python](https://github.com/redis/redis-py) and finally back to Java.

> Another thing to note is that I was looking for more than `10000` stars. At the time of writing, the [phpredis](https://github.com/phpredis/phpredis) client was close to `9600` stars.

Time for some practical stuff....

## Demos

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rx67sewcbnlcsqe6hbdw.png)

During the demo, I covered:

A walk though of the Go Redis client:

- Basics such as connecting to Redis
- Using common data types like string with `TTL`
- Use `hash` and `struct` support in `go-redis`
- How to use `set` as well as `pipelining` technique.

`Hyperloglog` and how does it differ from a `Set`:

```go
func hllandset() {
	pipe := client.Pipeline()

	ips := []string{}

	for i := 1; i <= 10_00_000; i++ {
		ips = append(ips, fake.IP(fake.WithIPv4()))
	}

	pipe.SAdd(context.Background(), "set_of_ips", ips)
	pipe.PFAdd(context.Background(), "hll_of_ips", ips)

	pipe.Exec(ctx)

	//redis-cli MEMORY usage set_of_ips
	fmt.Println("no. of unique views (SCARD) -", client.SCard(ctx, "set_of_ips").Val())

	//redis-cli MEMORY usage hll_of_ips
	fmt.Println("no. of unique views (PFCOUNT) -", client.PFCount(ctx, "hll_of_ips").Val())
```

Chat application using `pub/sub` (below is a trimmed down version of the code):

```go
package main

import (
    //omitted
)

var client *redis.Client
var Users map[string]*websocket.Conn
var sub *redis.PubSub
var upgrader = websocket.Upgrader{}

const chatChannel = "chats"

func init() {
	Users = map[string]*websocket.Conn{}
}

func main() {
    //...connect to redis (omitted)

	broadcast()

	http.HandleFunc("/chat/", chat)
	server := http.Server{Addr: ":8080", Handler: nil}
    
    //...start server (omitted)

	exit := make(chan os.Signal, 1)
	signal.Notify(exit, syscall.SIGTERM, syscall.SIGINT)
	<-exit

	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancel()

	//... clean up all connected connections, unsubscribe and shutdown server (omitted)
}

func chat(w http.ResponseWriter, r *http.Request) {
	user := strings.TrimPrefix(r.URL.Path, "/chat/")

	upgrader.CheckOrigin = func(r *http.Request) bool {
		return true
	}

	// 1. create websocket connection
	c, err := upgrader.Upgrade(w, r, nil)
	if err != nil {
		log.Print("upgrade:", err)
		return
	}

	// 2. associate the user (name) with the actual connection
	Users[user] = c
	fmt.Println(user, "in chat")

	for {
		_, message, err := c.ReadMessage()
		if err != nil {
            //error handling and disconnect (omitted)
		}

		// 3. when a messages comes in via the connection, publish messages to redis channel
		client.Publish(context.Background(), chatChannel, user+":"+string(message))
	}
}

func broadcast() {
	go func() {
		sub = client.Subscribe(context.Background(), chatChannel)
		messages := sub.Channel()
		for message := range messages {
			from := strings.Split(message.Payload, ":")[0]
			// 3. if a messages is received on the redis channel, broadcast it to all connected sessions (users)
			for user, peer := range Users {
				if from != user {
					peer.WriteMessage(websocket.TextMessage, []byte(message.Payload))
				}
			}
		}
	}()
}
```

## Tips and tricks

In this section, I covered some of the points from [Using Redis on Cloud? Here are ten things you should know](https://acloudguru.com/blog/engineering/how-to-use-redis-on-cloud). 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ou0zn6281c1d1fwj4j71.png)

These include:

- Connecting to Redis - common mistakes
- Scalability options - Vertical, Horizontal
- Using read-replicas
- Influence how your keys are distributed across a Redis cluster
- Execute bulk operations across Redis Cluster
- [Sharded Pub/Sub](https://redis.io/docs/manual/pubsub/#sharded-pubsub)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/czq7l46l8bg8qgm6hqtx.png)

## Resources

... and finally I wrapped up with some resources, including the [Discord channel for Go Redis community](https://discord.com/channels/752070105847955518/761147465251553280).

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kfs38sq777h9x2dct4nr.png)

Happy Building!