Deploy Rails in Amazon ECS: Part 5 - Integrate Sidekiq and Redis for background processing 
Tags: aws, ruby, devops, docker
Published: 2019-09-15 16:14:50

---
title: Deploy Rails in Amazon ECS: Part 5 - Integrate Sidekiq and Redis for background processing 
published: true
description: 
tags: aws, ruby, devops, docker
---

This is the fifth part of the *Deploy  Rails in Amazon ECS* post. It's part of a broader series called *More than 'hello world' in Docker*. The series will get you from *hello world* in Docker to having your application deployed in AWS.

- [Build Rails + Sidekiq web apps in Docker](https://dev.to/jamby1100/more-than-hello-world-in-docker-run-rails-sidekiq-web-apps-in-docker-1b37)
- **Deploy Rails in Amazon ECS**
  - [Concepts](https://dev.to/jamby1100/deploy-rails-in-amazon-ecs-part-1-concepts-26nl)
  - [Push an image to ECR](https://dev.to/jamby1100/deploy-rails-in-amazon-ecs-part-2-push-an-image-to-ecr-43fl)
  - [Create the RDS database, Task Definition, and Load Balancer](https://dev.to/jamby1100/deploy-rails-in-amazon-ecs-part-3-create-the-rds-database-task-definition-and-load-balancer-1ffe)
  - [Create the ECS Cluster and connect it all together](https://dev.to/jamby1100/deploy-rails-in-amazon-ecs-part-4-create-an-ecs-cluster-338f)
  - **Configure Sidekiq - we are here**
- [Automate Deploys with AWS CodeDeploy](https://dev.to/raphael_jambalos/automate-deployments-with-aws-codepipeline-52f5)
  - [Deep Dive on AWS CodeBuild](https://dev.to/jamby1100/automate-docker-build-with-aws-codebuild-9om)

In this post, we will be extending the ECS Setup we've built by adding a Sidekiq functionality. We will do this by:
- creating a Redis server that will hold tasks for Sidekiq
- creating an ECS service hosting our Sidekiq server
- connecting the web and Sidekiq service to this Redis server

# 13 | Creating a Task Definition for Sidekiq

For this step, follow the instructions on [section 7](https://dev.to/jamby1100/deploy-rails-in-amazon-ecs-part-3-create-the-rds-database-task-definition-and-load-balancer-1ffe) but changing the following configurations:
- **(7.3)** Put *docker-rails-app-sidekiq* as the name of the task definition
- **(7.5)** Use "sidekiq" as the container's name instead of "web".
- **(7.6)** Put `sidekiq,-C,config/sidekiq.yml` for the command. This is the command to run Sidekiq. You can specify another yml configuration file if yours is named differently.

# 14 | Creating a Sidekiq ECS Service

For this step, follow the instructions on [section 10](https://dev.to/jamby1100/deploy-rails-in-amazon-ecs-part-4-create-an-ecs-cluster-338f) but changing the ones listed below. Since this Sidekiq service will never receive traffic, we don't need to attach it to a load balancer. This service will just periodically fetch jobs from the Redis database we are about to make.

- **(10.2)** Choose the task definition family of *docker-rails-app-sidekiq*. For the name, choose `simple-ruby-app-sidekiq`.
- **(10.3)** For the load balancer, click "None".

# 15 | Start an EC2 instance for Redis

Sidekiq services need a Redis database to fetch tasks from. These tasks are either scheduled to run periodically or are fed in by the web services we created in the [last post](https://dev.to/jamby1100/deploy-rails-in-amazon-ecs-part-4-create-an-ecs-cluster-338f).

To make things simple, we are going to create an EC2 instance to run a Redis database application server. We will then configure our Sidekiq ECS service to be able to connect to this Redis database.

**(15.1)** Start an EC2 instance

For our EC2 instance, choose an Ubuntu 18.04 t2.micro instance. For this example, a t2.micro is enough but do note this can differ base on how much your workload uses Redis and Sidekiq. If your application also uses Redis for caching, then it's a good idea to make this instance bigger.

Here are the networking options you should apply:

- Make sure to choose the same VPC you selected when you created the ECS Cluster in [section 9](https://dev.to/jamby1100/deploy-rails-in-amazon-ecs-part-4-create-an-ecs-cluster-338f). 
- Select a Public Subnet.
- Select *auto-assign public IP* to be true. 

We generally don't need the Redis database server to be accessible to the public. But to make things simple for this example, we want it to be publicly accessible so we can connect publicly to the EC2 instance and install Redis. On production environments (which generally must be more secured), you usually connect to a jumpbox instance first before being able to connect to this EC2 instance hosting Redis. The EC2 instance is then placed in a *private* subnet and is not assigned a public IP.

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/y4ba9n5dxbvk86htrc2r.png)

Stick to the defaults for the storage and tagging pages.

Next, we will create a security group named *my-docker-redis-sg*. Create a rule that allows traffic from port 22 (so you can SSH into the instance), and from port 6379 (so anyone can access our Redis instance). 

Setting port 6379 accessible to the public is not best practice. We don't want anyone to be able to connect to this Redis server. Even if you have a secured password, all it might take is a few minutes for a sophisticated brute force attack to crack it.
 
![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/prb5dygq3kwzjjoizh0g.png)

**(15.2)** Configure the EC2 instance

Now that we have our EC2 instance up and running, let's connect to it. Go to the Services tab, then find EC2. On the left-side menu, click on Instances. You should be able to see the instance you just created. Once its status becomes "running", click the connect button on the upper right. You should be able to see instructions on how to connect to the instance.

If you haven't used your public key, you need to do a `chmod 400 yourkey.pem` to change its permissions. This is a required step for you to be able to use the key. Next, connect to the instance via the ssh command shown on the image below. If that doesn't work, it usually asks you to change `root` to `ubuntu` or `ec2-user`, depending on the OS that you used.

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/hjvigm9grys4zuha9o3i.png)

**(15.3)** Install Redis

After connecting to the instance, install Redis using the instructions in this [Digital Ocean documentation](https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-redis-on-ubuntu-18-04).

**(15.4)** Configure Redis

There are some changes though that you have to make to your Redis configuration. These changes will allow us to connect to the Redis instance. You can do this by `vi /etc/redis/redis.conf`.

- Comment out `bind 127.0.0.1 ::1`
- Make `protected-mode yes` to `protected-mode no`

**(15.5)** Start the Redis server

Then, type `sudo systemctl restart redis.service` to make your changes effective. Do a `ps -ef | grep rails` to ensure your process is running:

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/a9ac3ciqwl5k2kh2akji.png)

# 16 | Revising Task Definition

Now that we have a Redis instance and a Sidekiq ECS server, the missing piece is how we will let the Web ECS service throw jobs to the Redis database instance and how we will let the Sidekiq ECS service fetch jobs from the Redis database instance. To do this, we would have to change the task definitions of the Web and Sidekiq ECS services to include a `REDIS_URL` environment variable.

**(16.1)** On the services tab, search for ECS and click it. Click on "Task Definition" on the left-hand side menu. Then, click on the web's task definition (`docker-rails-app`).

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/yi9sl7qneh8m5xufqpp8.png)

**(16.2)** Click "Create New Revision"
![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/g2snewmrzue48tjmj7p5.png)

**(16.3)** Add a `REDIS_URL` environment variable in the container definition for web.
![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/1bmf122xzwha2azg9stn.png)

You should be able to see something like this:
![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/rn2jvclgoo2rjx467moa.png)

**(16.4)** Repeat steps 16.1 to 16.3 for the Sidekiq task definition (`docker-rails-app-sidekiq`).

# 17 | Updating the service

Now that we have an updated task definition for both web and Sidekiq, it's time to deploy!

**(17.1)** Click the *simple-ruby-app* service. On that page, click "Update"

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/me6w11x7c2dv4yx7ukgv.png)

**(17.2)** Choose the latest revision of the task definition of this service. Then tick "Force New Deployment" to make sure this new version gets deployed right away.

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/uo8k2h7frhggrk5it1ai.png)

**(17.3)** Monitor your deployment here. It takes a few minutes for your changes to take effect. You should be able to see here if your new version has been deployed. Your app has been deployed if `PRIMARY` has a running count of 1+

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/jr0py7yt65as48ocp5hl.png)

**(17.4)** Repeat the same steps for the Sidekiq service (`simple-ruby-app-sidekiq`)

# 18 | Finished!

Using the URL from [section 12](https://dev.to/jamby1100/deploy-rails-in-amazon-ecs-part-4-create-an-ecs-cluster-338f), access the website once again. You should be able to click the like button. This like button will tell the web ECS service to throw a task in the Redis database server. The Sidekiq ECS service gets this task from Redis and adds 1 to the like_count, as instructed on the worker.

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/nvdbss0ifctwlfol1uda.png)

Deploying a dockerized Rails application in production takes a lot of heavy lifting _at the start_, but once the setup is done, you'd be able to enjoy the benefits of Docker and ECS: lower costs, faster deployments, faster rollbacks, easier dev onboarding and more.

Now that you've completed your own sample Rails app in Docker, you'd be able to extend this to your own production environments. 

*If you have any comments, suggestions or just want to let me know how this series has helped you, feel free to leave a comment below, or message me! I'd love to hear from you!*

*Special thanks to my editor, Allen, for making my posts more coherent*.