Moving to solid_queue from sidekiq
Tags: ruby, rails, watercooler, tutorial
Published: 2023-12-28 14:16:40


A little bit about me, I am an engineer turned indiehacker for the first time and working on my startup [devtree](https://devtree.app?utm_source=devto). I started working on it in July this year and growing it slowly.

Its a simple Ruby on Rails application running on Flyio backed by Postgres and background jobs running with Sidekiq and redis.


 
## The problem

I use Upstash serverless Redis in my app that is provided by Flyio with their tight integration with Upstash.

Upstash pricing is per request based, which was different for apps running on flyio as they were providing a free tier based on memory restrictions which worked for me.

But last month they decided to unify their pricing everywhere after new year on 1st Jan. Based on my setup with sidekiq, it sends gazillion request per hour to check for new jobs which eats up their free quota of 10k requests per day.


![upstash redis](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y3kgwa8f97av3c4k6bzl.png)

As much I like sidekiq, I can't bear the redis costs when I am not making any money.

## Solution

I was looking for alternative solutions specifically non-redis based. I had a few suggestions from my network which included `solid_queue` which surprisingly I forgot about.

Its a non-redis solution, works with postgres and I wouldn't spend weeks implementing it. So, I decided to give it a try.

Here's the process I followed:

1 - **Create a new branch**
2 - **Add the gem**

```
gem 'solid_queue'

```
3 - **Change the active job adapter**

```
config.active_job.queue_adapter = :solid_queue

```
4 - **Install solid queue**

```
rails generate solid_queue:install

```
5 - **Run migrations for job tables**

```
rails solid_queue:install:migrations
rails db:migrate

```
6 - **Run the worker**

```
bundle exec rake solid_queue:start
# or add it in Procfile.dev
# solid: bundle exec rake solid_queue:start

```

Now you can create your first job

```
rails g job my_job

# app/jobs/my_job.rb

class MyJob < ApplicationJob
  self.queue_adapter = :solid_queue
  # ...
end

```

## Concepts

Solid queue will create a few tables for you to store your jobs but focus on 2 for now. 
`solid_queue_scheduled_executions` and `solid_queue_ready_executions`

You can guess from their names, one is used to store jobs when they are to be run in the future.

The other is to store jobs which need immediate execution.


### Dispatchers

`Dispatchers` are in charge of selecting jobs scheduled to run in the future that are due and dispatching them, which is simply moving them from the `solid_queue_scheduled_executions` table over to the `solid_queue_ready_executions` table so that workers can pick them up. They also do some maintenance work related to concurrency controls.

### Workers

`Workers` are in charge of picking jobs ready to run from queues and processing them. They work off the `solid_queue_ready_executions` table

### Configuration


You can also configure the polling interval, number of workers based on your requirements by using a yaml configuration file.

add a configuration file in config/solid_queue.yml:

```
development:
  dispatchers:
    - polling_interval: 5 #in seconds
      batch_size: 100
  workers:
    - queues: '*'
      threads: 5
      processes: 1
      polling_interval: 5 #in seconds
```

With this configuration, it will spin 5 worker threads to process the jobs every 5 seconds.

There are tons of other job level and queue level configurations that you can add for better control over concurrency, uniqueness and queue management that you can find [here](https://github.com/basecamp/solid_queue?tab=readme-ov-file#configuration).


## Caveat

I have been monitoring my app for a few days and unfortunately it has crashed a few times due out of memory which never happened before, seems like 512mb ram is not enough 🤷.

I haven't debugged the issue for now but will look into it for the root cause.


This is it form side, let me know if you would like to know more about the technical side of my startup.
I share about my startup publicly on [twitter](https://twitter.com/zainnazirbutt) and [linkedin](https://linkedin.com/in/zainnazirbutt).