Run Sidekiq 6 as daemon in Production environment on Ubuntu 20.04
Tags: sidekiq, rails, systemd
Published: 2021-08-20 12:52:29

> I also have Traditional Chinese version {% link https://dev.to/kevinluo201/systemd-ubuntu-sidekiq-6-d9c %}

Sidekiq is a very convenient queuing tool for developers to handle the asynchronous tasks and response clients sooner.

Deploying Sidekiq on production server is not hard but there are many details which should be reminded.

I'd like to share my experience of executing Sidekiq 6 by `systemd` as daemon service on Ubuntu 20.04

## What is our goal?
* Execute Sidekiq
* Sidekiq should be start automatically after reboot or crash
* Sidekiq should be restarted after we deploy Rails application via Capistrano

## How do we execute Sidekiq?
To know how to execute Sidekiq in the production server. We need to know how do we Execute Sidekiq in the development environment.
```bash
bundle exec sidekiq
```
It is in fact very simple. You can append more arguments such as `-C xxxx.yml` for specific config file, but we can keep it simple now.

So, actually, you can do the same thing on the production server.
If you want it to be more like a daemon. You can add a `&` to make the command run in the background:
```bash
RAILS_ENV=production bundle exec sidekiq &
```
If you just want to execute Sidekiq in the production environment, that’s it.

But it is not very practical and clearly doesn’t fit our needs:
1. It won’t be restarted after the server reboot
2. It won’t restart automatically after we deploy the application or it crashes due to exceptions
3. It does not feel “Pro”. You want a more “Pro” way to do this even you’re not using Sidekiq Pro….(just kidding😂)

To make it more ideal and practical, we need to use `systemd`

## What is systemd?
`systemd` is a service manager program to manage the daemon program and it is used by Linux system. 
With `systemd`, we can… 
* use the command `systemctl` to start/stop any service
* **enable** services, and the enabled services will be started automatically after the system boot.
* You can specify what to do if the service fails, for example, restarting it.

Those are what we need to know about `systemd` for now, feel free to google more information about it.
It sounds like it can fit our needs, so let’s use `systemd` to control Sidekiq!

## Add Sidekiq as a service unit
Every service is treat as an `unit` and it has a unit file to describe each service.
We can make a file under `/lib/systemd/system/sidekiq.service`

By the way the user to execute Sidekiq is named `deployer`
```bash
# /lib/systemd/system/sidekiq.service
[Unit]
Description=sidekiq
After=syslog.target network.target

[Service]
Type=simple
WorkingDirectory=/path/to/your/app

# If you use rbenv:
# ExecStart=/bin/bash -lc 'exec /home/deploy/.rbenv/shims/bundle exec sidekiq -e production'
# If you use the system's ruby:
# ExecStart=/usr/local/bin/bundle exec sidekiq -e production
# If you use rvm in production without gemset and your ruby version is 2.6.5
# ExecStart=/home/deploy/.rvm/gems/ruby-2.6.5/wrappers/bundle exec sidekiq -e production
# If you use rvm in production with gemset and your ruby version is 2.6.5
# ExecStart=/home/deploy/.rvm/gems/ruby-2.6.5@gemset-name/wrappers/bundle exec sidekiq -e production
# If you use rvm in production with gemset and ruby version/gemset is specified in .ruby-version,
# .ruby-gemsetor or .rvmrc file in the working directory
ExecStart=/home/deployer/.rvm/bin/rvm in /path/to/your/app/current do bundle exec sidekiq -e production

User=deployer
Group=deployer
UMask=0002

# Greatly reduce Ruby memory fragmentation and heap usage
# https://www.mikeperham.com/2018/04/25/taming-rails-memory-bloat/
Environment=MALLOC_ARENA_MAX=2

# if we crash, restart
RestartSec=1
Restart=on-failure

# output goes to /var/log/syslog
StandardOutput=syslog
StandardError=syslog

# This will default to "bundler" if we don't specify it
SyslogIdentifier=sidekiq

[Install]
WantedBy=multi-user.target
```
Because `systemd` scans all services under `/lib/systemd/system/`, so your `sidekiq.service` can be found by `systemd` now.

After creating sidekiq.service, you can execute the following commands
```bash
# reload serivces
sudo systemctl daemon-reload
# enable the sidekiq.service so it will start automatically after rebooting
sudo systemctl enable sidekiq.service
# start the sidekiq
sudo service sidekiq start
# we can check the log in /var/log/syslog
sudo cat /var/log/syslog
# we can check if Sidekiq is started
sudo ps aux | grep sidekiq
# or
sudo systemctl status
```

## Restart Sidekiq by Capistrano 
I use Capistrano to deploy rails because it saves my time and prevent human mistakes.
After I deploy the latest code to production server, I also want Sidekiq to be restarted so it can load the latest code.
To add this behavior into Capistrano workflow, we can use the gem `capistrano-sidekiq`
Adding this gem into Gemfile then do bundle install
```ruby
# Gemfile
gem 'capistrano-sidekiq', group: :development
```
In the Capfile, add:
```ruby
# Capfile
require 'capistrano/sidekiq'
# Default sidekiq tasks
install_plugin Capistrano::Sidekiq
# We specify that we want to use systemd to control sidekiq
install_plugin Capistrano::Sidekiq::Systemd
```
Then during the `cap production deploy`, it will do the following tasks in order:
* stop get new task from redis
* stop Sidekiq service
* start Sidekiq service

## Make an user-wide sidekiq.service
There is one thing I didn’t mention on purpose.
The service unit we just created is a system-wide and always need `sudo` to use systemctl.
If we want any unprivileged user to utilize systemd, we need a user-wide service unit.
Moreover, `capistrano/sidekiq` default setting expects the user-wide service unit to control the Sidekiq.

And Yes, we can just modify the default setting of `capistrano/sidekiq` to control to use the system-wide systemd, but I just want to explain there are many options we can choose from. (And I was also confused why my `sidekiq.service` not work…when I was learning how to execute Sidekiq on production server)

If you have already enabled the system-wise `sidekiq.service`, you need to disable it and delete the service-unit:
```bash
sudo systemctl stop sidekiq
sudo systemctl disable sidekiq.service
sudo rm /lib/systemd/system/sidekiq.service
```

The user-wide services should be put under `~/.config/systemd/user/`, where is the location `systemd` scans for the user-wide services. Add a file named `sidekiq.service` again:

```bash
[Unit]
Description=sidekiq
After=syslog.target network.target

[Service]
# notify can be used only after Sidekiq 6.0.6
# if the version is under 6.0.6, use Type=simple
Type=notify
WatchdogSec=10

WorkingDirectory=/path/to/your/app/current

# If you use rbenv:
# ExecStart=/bin/bash -lc 'exec /home/deploy/.rbenv/shims/bundle exec sidekiq -e production'
# If you use the system's ruby:
# ExecStart=/usr/local/bin/bundle exec sidekiq -e production
# If you use rvm in production without gemset and your ruby version is 2.6.5
# ExecStart=/home/deploy/.rvm/gems/ruby-2.6.5/wrappers/bundle exec sidekiq -e production
# If you use rvm in production with gemset and your ruby version is 2.6.5
# ExecStart=/home/deploy/.rvm/gems/ruby-2.6.5@gemset-name/wrappers/bundle exec sidekiq -e production
# If you use rvm in production with gemset and ruby version/gemset is specified in .ruby-version,
# .ruby-gemsetor or .rvmrc file in the working directory
ExecStart=/home/deployer/.rvm/bin/rvm in /path/to/your/app/current do bundle exec sidekiq -e production
ExecReload=/usr/bin/kill -TSTP $MAINPID

# Greatly reduce Ruby memory fragmentation and heap usage
# https://www.mikeperham.com/2018/04/25/taming-rails-memory-bloat/
Environment=MALLOC_ARENA_MAX=2

# if we crash, restart
RestartSec=1
Restart=on-failure

# output goes to /var/log/syslog
StandardOutput=syslog
StandardError=syslog

# This will default to "bundler" if we don't specify it
SyslogIdentifier=sidekiq

[Install]
WantedBy=default.target
```

Then we can use following commands to control sidekiq.service:

```bash
systemctl --user daemon-reload 
systemctl --user enable sidekiq.service

# you can use systemctl to control sidekiq
systemctl --user {start,stop,restart} sidekiq
```

With this `sidekiq.service` file and enabled it by systemd, the `capistrano/sidekiq` should work properly : )

## Common problems

### Sidekiq always turns down while start it via user-wise systemd
User-wise systemd actually only allows logged-in user to execute the services.
Therefore, all the services will be shut down while the last user session is ended.
To keep the users session to execute the user-wise systemd service, we can use the command below:
```bash
loginctl enable-linger [user_name]
```
Then the user's session will be created once the system starts, and will be kept. And the user-wise systemd services can be preserved to run.

### When running Sidekiq:quiet task during deploying, it has error below

```
sidekiq:quiet
    01 systemctl --user reload sidekiq
    01 Failed to reload sidekiq.service: Job type reload is not applicable for unit sidekiq.service.
  ✘ 01 deployer@xxxxxxxx 0.067s
```

#### Solution

* add `ExecReload=/bin/kill -TSTP $MAINPID` into
* Although sidekiq readme has said that it is recommended to use this command to stop receiving tasks from redis. Capistrano/sidekiq does not have the corresponding explanation.

### The target is not found

* The target is a group of service unit
* You can use `systemctl list-units —type=target` to find the available targets

### Redis is not started yet

* Actually this is the situation I imagine…I haven’t had this situation yet.
* I think add redis.service in After may solve this… `After=syslog.target network.target redis.service`

### The -L log/sidekiq.log is not work

* Log redirection is not supported in Sidekiq 6 anymore, please read [Logging · mperham/sidekiq Wiki · GitHub](https://github.com/mperham/sidekiq/wiki/Logging)
* we can Add Sidekiq tag to the logs in the syslog by `bundle exec sidekiq 2>&1 | logger -t sidekiq`

If you think this article is helpful, you can buy me a coffee to encourage me 😉
[![Buy Me A Coffee](https://cdn.buymeacoffee.com/buttons/default-black.png)](https://www.buymeacoffee.com/kevinluo)

## Reference

* An example of `sidekiq.service` provided by sidekiq repo: [sidekiq/sidekiq.service at master · mperham/sidekiq · GitHub](https://github.com/mperham/sidekiq/blob/master/examples/systemd/sidekiq.service)
* [GitHub - seuros/capistrano-sidekiq: Sidekiq integration for Capistrano](https://github.com/seuros/capistrano-sidekiq)