Redis is letting you down, but it doesn't have to
Tags: redis, heroku, rails, ruby
Published: 2018-02-18 00:17:03

---
title: "Redis is letting you down, but it doesn't have to"
published: true
tags: Redis, Heroku, Rails, Ruby
canonical_url: https://www.briankephart.com/redis-is-letting-you-down
description: "Are you seeing Redis::ConnectionError: Connection lost (ECONNRESET)? Is your cache hit rate suspiciously low? A story of the bug I didn't know I had, how I found it, and how I got rid of it. It's easy, I promise."
---

<p>For awhile now, I've had a <a href="http://www.rubyonrails.org" target="_blank" rel="noopener">Rails</a> app on <a href="https://www.heroku.com" target="_blank" rel="noopener">Heroku</a> that uses <a href="https://elements.heroku.com/addons/heroku-redis" target="_blank" rel="noopener">Heroku Redis</a> for caching. Certain endpoints on this app use page caching, so rendering doesn't require parsing the template files unless there's a cache miss. It's great, and it's fast.</p>
<p>Over time, though, I noticed in my performance profiler that template files were being rendered more often than I would expect, slowing down those page loads. In this context, it's no big deal if a page occasionally takes a few hundred milliseconds to render (instead of less than 150), so I didn't worry about it too much. There are better ways to spend my development time than fixing a few slow requests that still take less than a second.</p>
<p>Then, I updated to Rails 5.2. Even though I've been happy using the <a href="https://github.com/sorentwo/readthis" target="_blank" rel="noopener">Readthis gem</a>&nbsp;for Redis caching, I thought I'd give Rails' new built-in&nbsp;Redis Cache Store a try, just to eliminate a gem dependency. So far, so good, performance is no different than with Readthis.</p>
<p>Then, I noticed an error handler in the <a href="http://edgeguides.rubyonrails.org/caching_with_rails.html#activesupport-cache-rediscachestore" target="_blank" rel="noopener">documentation</a>. Like Readthis, Redis Cache Store is fault tolerant. If there's a Redis error, it's treated as a cache miss instead of an exception. This is great for failsafe operation, but I wondered if there were any errors not being reported. My error reporting service is <a href="https://rollbar.com" target="_blank" rel="noopener">Rollbar</a>, so I added this line to my cache config:</p>
```
error_handler: -&gt; (method:, returning:, exception:) {
Rollbar.warning(exception, method: method, returning: returning)}
```
<p>After an hour, I had a handful of warnings from Rollbar containing the exception&nbsp;<code>Redis::ConnectionError: Connection lost (ECONNRESET)</code>. What gives? My low-volume app was losing its connection to Redis several times an hour. This continued overnight.</p>
<p>I looked through <a href="https://github.com/rails/rails/blob/master/activesupport/lib/active_support/cache/redis_cache_store.rb#L57" target="_blank" rel="noopener">the code</a> for the Redis Cache Store and noticed something in the default settings:</p>
<p><code class="block">reconnect_attempts: 0</code></p>
<p>Could it be that easy? I set <code>reconnect_attempts: 1</code>&nbsp;in my config and sure enough, my errors disappeared, and so did my cache misses. This one line of code sped up my average response time on these endpoints significantly, all by speeding up the slowest requests.</p>
<p>I still don't know why Redis disconnects so often, or whether my experience is normal when using Redis as a service. It makes sense to me, though, that a service provided over a network could have occasional connection issues, especially when it's trying to operate fast enough for caching.</p>
<p>The lesson: Be sure to set up error monitoring on any fault-tolerant Redis service, and see if you can benefit from allowing reconnect attempts.</p>