Configuring Incoming Webhook Queues in Bullet Train
Tags: rails
Published: 2024-02-16 17:11:55

---
canonical_url: https://useattr.actor/articles/bullet-train-incoming-webhook-queues
---

[Attractor](https://useattr.actor) is a pretty background-job intensive app. If you think about it, every commit's changes have to be analyzed, and the results stored, to keep track of code quality and emerging tech debt.

Since it's based on [Bullet Train](https://bullettrain.co/), it makes sense to handle most of these workloads in **incoming webhooks**. Broadly speaking, there are three groups of incoming webhooks at work:

- webhook handlers for _processing code analytics data_ being reported back from sandbox containers,
- webhook handlers for _webhooks sent from GitHub_ (e.g. when a new pull request is created),
- everything else, e.g. _webhooks coming from [Paddle](https://www.paddle.com/)_, my payment provider.

## Webhook Processing

Each incoming webhook in Bullet Train is processed by a controller that is created by [super scaffolding](https://bullettrain.co/docs/super-scaffolding).

It might look like this:

```rb
class Webhooks::Incoming::SandboxWebhooksController < Webhooks::Incoming::WebhooksController
  before_action :authenticate_token!

  def create
    Webhooks::Incoming::SandboxWebhook.create(data: JSON.parse(request.body.read))
      .process_async
    render json: {status: "OK"}, status: :created
  end
  
  # ...
end
```

The created model file contains the processing logic:

```rb
class Webhooks::Incoming::SandboxWebhook < ApplicationRecord
  include Webhooks::Incoming::Webhook

  def process
    # logic goes here
  end
end
```

The actual processing is done by a generic `Webhooks::Incoming::WebhookProcessingJob` that's part of Bullet Train and calls back to this `process` method. In other words, in a vanilla Bullet Train app, Sidekiq is responsible for working off your incoming webhooks.

So, let's look at the respective `sidekiq.yml` file:

```yml
:concurrency: 5
staging:
  :concurrency: 5
production:
  :concurrency: 5
:queues:
  - critical
  - default
  - mailers
  - low
  - action_mailbox_routing
```

There are 5 Sidekiq queues configured. A pretty nasty issue is hidden here in plain sight, though: All incoming webhooks are processed in the `default` queue.

## Dynamically Configuring the Webhook Queue

Why is this a problem? Let's walk through a scenario where a new customer signs up.

1. she authenticates the Attractor GitHub app, and connects two pretty large repositories,
2. she enters her credit card information and wants to conclude the sign-up,
3. nothing happens, she is stuck.

What's happening here? It took me a while to find out.

As pointed out above, all incoming webhooks are processed in the `default` queue, which in this case means:

1. all analysis jobs (which can be hundreds or even thousands) are queued up there,
2. the one incoming webhook responsible for processing the payment, too - right at the bottom.

In other words, the payment would be processed only after all analysis jobs (and not only hers, maybe a couple of other customers did sign up simultaneously) have concluded. An unbearable situation, to have a customer hanging in a state of uncertainty whether her payment did successfully activate her subscription of [Attractor](https://useattr.actor).

To remedy this, I could have changed the sign up flow to first require potential customers to check out, then allow them to connect the GitHub app. That would have been a pretty large effort, though, so I opted to try something else.

We can reopen the `Webhooks::Incoming::WebhookProcessingJob` and have the `queue_as` method decide dynamically where to queue the webhook it's meant to process:

```rb
Webhooks::Incoming::WebhookProcessingJob.queue_as do
  webhook = arguments.first

  case webhook
  when Webhooks::Incoming::PaddleWebhook
    :critical
  when Webhooks::Incoming::SandboxWebhook
    :low
  else
    :default
  end
end
```

The only remaining question was where to put this small monkey patch. Placing it into an initializer didn't work, because your app's constants (such as the `Webhooks::Incoming::WebhookProcessingJob` class) haven't loaded at this time.

So the only option is to run it after the app has finished initializing, i.e. in an `after_initialize` block in your `config/application.rb`:

```rb
require_relative "boot"

require "rails/all"

# ...

module MyApp
  class Application < Rails::Application
    # more config
    
    config.after_initialize do
      Webhooks::Incoming::WebhookProcessingJob.queue_as do
        webhook = arguments.first

        case webhook
        when Webhooks::Incoming::PaddleWebhook
          :critical
        when Webhooks::Incoming::SandboxWebhook
          :low
        else
          :default
        end
      end
    end
  end
end
```

With this tweak, all webhooks coming in from the code analysis sandbox are placed in the _low_ job queue, keeping others in the _default_ one. Subsequently, when a webhook comes in from GitHub indicating a new pull request, it is processed before any potentially waiting `SandboxWebhook`s.

Even more importantly, incoming webhooks from Paddle are placed in the _critical_ queue, making sure subscription changes are always processed first.

Thanks to [Kasper Timm Hansen](https://twitter.com/kaspth) for pointing me in the right direction! 🙏