Digging into Solid Queue and Rails
Tags: rails, tutorial, ruby
Published: 2024-03-05 19:24:24

---
title: Digging into Solid Queue and Rails
published: true
date: 2024-01-07 18:21:01 UTC
tags: rails, tutorial, ruby
canonical_url: https://web-crunch.com/posts/digging-into-solid-queue-rails
---


{% youtube imU_9_C4UcQ %}

In the realm of Ruby on Rails and background job processing, [Solid Queue](https://github.com/basecamp/solid_queue) is the newest contender. Solid Queue is a robust alternative to well-established solutions like [good\_job](https://github.com/bensheldon/good_job) and [Sidekiq](https://github.com/sidekiq/sidekiq). This guide explores Solid Queue, delving into its distinctive features and comparing it with its counterparts.

## What is Solid Queue?

[Solid Queue](https://github.com/basecamp/solid_queue) is an open-source background processing library tailored for Ruby on Rails applications. Its primary objective is to efficiently manage asynchronous tasks, ensuring your application's performance remains rock-solid even during peak traffic and demanding workloads.

## Key Features of Solid Queue:

1. **Reliability:** Solid Queue places a premium on reliability and data integrity. It guarantees that jobs are processed precisely once, eliminating the risk of duplicates or data loss.
2. **Transaction Support:** Unlike some competing solutions, Solid Queue seamlessly integrates with database transactions, ensuring consistent data when both your application and background jobs interact.
3. **Concurrency Control:** Solid Queue offers granular control over concurrency, enabling you to specify the number of workers for each job type and optimize resource utilization.
4. **Active Job Compatibility:** If your Rails application relies on Active Job, you'll appreciate Solid Queue's effortless compatibility, simplifying migration and adoption.

## How Solid Queue Sets Itself Apart

- **Data Integrity:** Solid Queue is committed to data integrity, which is critical for applications handling sensitive information or intricate business logic. It guarantees that jobs are processed reliably, eliminating the possibility of duplicates.
- **Database Transactions:** Integrating Solid Queue with database transactions is a standout feature. In contrast, GoodJob and Sidekiq may necessitate additional configuration through something like Redis and management to maintain transactional consistency.
- **Concurrency Control:** While all three solutions offer concurrency control, Solid Queue's flexibility in specifying worker counts for different job types empowers you to fine-tune resource allocation for optimal performance.

### Examples

To show how to use SolidQueue in a Ruby on Rails application, you'll need to take the following steps on a new Rails application.

Install the gem. Add the following to your `Gemfile` and run `bundle install` .

```ruby
gem "solid_queue"
```

Run the installer

```bash
rails generate solid_queue:install
```

This adds a few migrations, replaces a few configurations in the `test,` `development,` and `production` environment, and adds a `solid_queue.yml` config file to the app. All the values are commented out by default.

```bash
gsub config/environments/development.rb
        gsub config/environments/test.rb
        gsub config/environments/production.rb
      create config/solid_queue.yml
       rails railties:install:migrations FROM=solid_queue
Copied migration 20240102191754_create_solid_queue_tables.solid_queue.rb from solid_queue
```

The configuration wasn’t added to my `development.rb` environment file but made it to the `production.rb` file. You may need to append the following to your preferred environment configurations if you aren't adding it to `config/application.rb`

```bash
config.active_job.queue_adapter = :solid_queue
```

Inside the `config/solid_queue.yml` configuration file, be sure to uncomment the defaults.

```yaml
default: &default
  dispatchers:
    - polling_interval: 1
      batch_size: 500
  workers:
    - queues: "*"
      threads: 5
      processes: 1
      polling_interval: 0.1

development:
 <<: *default

test:
 <<: *default

production:
 <<: *default
```

Be sure to configure Solid Queue as your Active Job queue adapter

```ruby
# config/application.rb
config.active_job.queue_adapter = :solid_queue
```

Run the migrations

```bash
rails db:migrate
```

This creates several tables related to different states of a given job:

```bash
== 20240102191754 CreateSolidQueueTables: migrating ===========================
-- create_table(:solid_queue_jobs)
   -> 0.0024s
-- create_table(:solid_queue_scheduled_executions)
   -> 0.0009s
-- create_table(:solid_queue_ready_executions)
   -> 0.0012s
-- create_table(:solid_queue_claimed_executions)
   -> 0.0009s
-- create_table(:solid_queue_blocked_executions)
   -> 0.0009s
-- create_table(:solid_queue_failed_executions)
   -> 0.0007s
-- create_table(:solid_queue_pauses)
   -> 0.0007s
-- create_table(:solid_queue_processes)
   -> 0.0009s
-- create_table(:solid_queue_semaphores)
   -> 0.0012s
-- add_foreign_key(:solid_queue_blocked_executions, :solid_queue_jobs, {:column=>:job_id, :on_delete=>:cascade})
   -> 0.0062s
-- add_foreign_key(:solid_queue_claimed_executions, :solid_queue_jobs, {:column=>:job_id, :on_delete=>:cascade})
   -> 0.0054s
-- add_foreign_key(:solid_queue_failed_executions, :solid_queue_jobs, {:column=>:job_id, :on_delete=>:cascade})
   -> 0.0050s
-- add_foreign_key(:solid_queue_ready_executions, :solid_queue_jobs, {:column=>:job_id, :on_delete=>:cascade})
   -> 0.0061s
-- add_foreign_key(:solid_queue_scheduled_executions, :solid_queue_jobs, {:column=>:job_id, :on_delete=>:cascade})
   -> 0.0053s
== 20240102191754 CreateSolidQueueTables: migrated (0.0381s) ==================
```

Start Solid Queue’s supervisor to be able to run jobs. This should be a new terminal instance.

```bash
bundle exec rake solid_queue:start
```

Define and enqueue jobs as you would with Active Job:

```ruby
# app/jobs/sample_job.rb
class SampleJob < ApplicationJob
  def perform
    # Your job logic here
  end
end

# Enqueue the job
SampleJob.perform_later


rails console
> SampleJob.perform_later
```

Pretty simple, right? I love that it lives within the framework and doesn’t make me learn a complicated new standard. I come from Sidekiq, and the workflow is highly similar, minus a few bits.

## Handling Failed jobs and retries

Solid Queue does not have an automatic retry feature; instead, it depends on [Active Job for this functionality](https://edgeguides.rubyonrails.org/active_job_basics.html#retrying-or-discarding-failed-jobs). Any failed jobs will be stored in the system, and a record of the failed execution will be created in the `solid_queue_failed_executions` table. These failed jobs will remain in the system until manually discarded or re-enqueued.

The team plans to introduce a UI component called _Mission Control_ that allows you to examine and diagnose any specific job or group of jobs efficiently.

For now, you can manually intervene with these methods:

```bash
failed_execution = SolidQueue::FailedExecution.find(...) # Find the failed execution related to your job
failed_execution.error # inspect the error

failed_execution.retry # This will re-enqueue the job as if it was enqueued for the first time
failed_execution.discard # This will delete the job from the system
```

This seems nice for the ability to manually retry if necessary in your app’s logic rather than having it be arbitrary. Some might argue the latter.

## Adding some UI in the meantime

I noticed on Twitter a library called [Panoptic](https://github.com/virolea/panoptic). The simple UI was built with Bootstrap CSS to provide a GUI for Solid Queue.

You can add it much like Sidekiq if you’re familiar with that configuration style. It’s an engine that bolts on the app.

```ruby
# add to your Gemfile
gem "panoptic"
```

Then in your `config/routes.rb` file, add the following one-liner

```ruby
# config/routes.rb
Rails.application.routes.draw do
  mount Panoptic::Engine => "/panoptic"
  # more code
end
```

You can visit `localhost:3000:/panoptic` to see it in action.

Again, the team has stated that something called _Mission Control_ is on the near horizon, so I wouldn't depend heavily on this library. It is something, though!

### In pursuit of increased developer happiness with Puma

While running `bundle exec rake solid_queue:start` each time you need background job processing, it works fine; a handy Puma plugin integration in Solid Queue allows for some nice automations. When Puma is running (rails server), the processing of your job will kick in alongside it.

I need to add a one-liner to my `puma.rb` file.

```ruby
# config/puma.rb
plugin :solid_queue
```

## Conclusion

Solid Queue is a formidable choice for background job processing in Ruby on Rails applications. It prioritizes reliability, data integrity, and seamless database transaction integration.

While GoodJob and Sidekiq remain excellent alternatives, Solid Queue's unique attributes make it a compelling candidate for your next project without additional dependencies or resources. I think you will need to consider the extra database space it might reserve, but hopefully, it shouldn't be enough to cause too much concern.