Background job processing with Marten and Mosquito
Tags: crystal, marten
Published: 2023-04-08 21:19:33

[Marten](https://github.com/martenframework/marten) is a web framework written in Crystal that makes building web applications easy and enjoyable. [Mosquito](https://github.com/mosquito-cr/mosquito) is a background task runner for Crystal that uses Redis and that makes it easy to schedule and run tasks asynchronously.

Combining Marten with Mosquito can be a powerful way to build web applications with background tasks that run asynchronously. In this blog post, we'll show you how to get started with using Marten and Mosquito together.

## Installation

Assuming that you already have a working Marten project that was initialized with the [built-in authentication](https://martenframework.com/docs/authentication/introduction) at hand, you can start by adding Mosquito to your `shard.yml` file:

```yaml
dependencies:
  mosquito:
    github: mosquito-cr/mosquito
```

Then run `shards install` to install the new dependencies.

We can now ensure that Mosquito is properly required by our project. To do so, let's update the content of the `src/project.cr` file as follows:

```crystal
# Third party requirements.
require "marten"
require "mosquito"
# other requirements...

# Project requirements.
require "./jobs/**"

# other requirements...
```

You'll notice that we require both `mosquito` and the files of a `jobs` folder, where we'll define the actual jobs.

Now we can create a new `config/initializers/mosquito.cr` initializer file where the actual Mosquito configuration will live. We won't cover all the [Mosquito configuration options](https://mosquito-cr.github.io/manual/index.html#configuration) in this article, but this initializer could simply set the right Redis URL to use for now and could look something like this:

```crystal
Mosquito.configure do |settings|
  settings.redis_url = (ENV["REDIS_URL"]? || "redis://localhost:6379")
end
```

Finally, we need to create a new file where we will (i) setup Marten itself and (ii) start the Mosquito runner. In this light, let's create a new `src/worker.cr` file with the following content:

```crystal
require "./project"

Marten.setup

Mosquito::Runner.start
```

## Defining jobs

For the purpose of this article, we will be defining jobs under a `src/jobs` folder. If your Marten project is split into  multiple [applications](https://martenframework.com/docs/development/applications), you can also decide to create a `jobs` folder inside each of these applications.

So let's create a `src/jobs` folder for now and let's define a `send_welcome_email_job.cr` file in it with the following content:

```crystal
class SendWelcomeEmailJob < Mosquito::QueuedJob
  params(user_id : Int64)

  def perform
    user = Auth::User.get!(id: user_id)
    user.send_welcome_email
    user.update!(welcome_email_sent: true)
  end
end
```

This example job simply retrieves a `User` model record using a `user_id` job parameter, sends an email by calling a hypothetical `#send_welcome_email` method, and updates the considered record by setting a hypothetical  `welcome_email_sent` field to `true`.

## Enqueuing jobs

Enqueuing a job is as simple as initializing an instance of a job class with the specified parameters and calling the `#enqueue` method. The job above was all about sending a welcome email to newly created users. As such, this job could be enqueued from a [callback](https://martenframework.com/docs/models-and-databases/callbacks) that we could define in the `Auth::User` model. For example:

```crystal
module Auth
  class User < MartenAuth::User
    after_commit :trigger_welcome_email_sending, on: :create

    private def trigger_welcome_email_sending
      SendWelcomeEmailJob.new(user_id: id!.to_i64).enqueue
    end
  end
end
```

In this example, we are triggering the job from a model callback but it should be noted that you can enqueue jobs from pretty much anywhere in your Marten project codebase (eg. [handlers](https://martenframework.com/docs/handlers-and-http/introduction), [schemas](https://martenframework.com/docs/schemas/introduction), etc).

## Running the Mosquito worker

As you might have guessed, the Mosquito worker runs in a dedicated process. This means that we have to explicitly start the runner we defined earlier by compiling and executing the `src/worker.cr` file. While in development, we can simply leverage the `crystal run` command to do so:

```bash
crystal run src/worker.cr
```

## A few tips regarding deployments

At deployment time, it's likely that you already compile two binaries: your project's server (`src/server.cr`) and your project's management CLI (`manage.cr`). In addition to these, you will now need to ensure that your worker is compiled as well. This can be accomplished by relying on the `crystal build` command:

```bash
crystal build src/worker.cr -o bin/worker --release
```

You could also add a target to your `shard.yml` file and even trigger the execution of your compiled worker from a `Procfile` file depending on your deployment strategy. You can learn more about deploying Marten in the [dedicated documentation](https://martenframework.com/docs/deployment/introduction).

## Conclusion

In this blog post, we've gone through the steps of setting up Mosquito with Marten, including installation, configuration, and usage. Using Mosquito with the Marten web framework can help you build fast and scalable web applications that leverage background job processing capabilities.