Robust queue job handling in Craft CMS
Tags: craftcms, devops, queue
Published: 2020-03-04 05:44:21

---
title: Robust queue job handling in Craft CMS
published: true
date: 2019-08-03 15:37:00 UTC
tags: craftcms,devops,queue
canonical_url: https://nystudio107.com/blog/robust-queue-job-handling-in-craft-cms
---

# Robust queue job handling in Craft CMS

### Craft CMS uses queue jobs for long-run­ning tasks. Here’s how to avoid stalled or failed queue jobs by using an inde­pen­dent async queue runner

Andrew Welch / [nystudio107](https://nystudio107.com)

![Craft cms yii2 async queue](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_1200x675_crop_center-center_82_line/craft-cms-yii2-async-queue.jpg)

Queu­ing in line is some­thing few peo­ple are fond of, but it allows each per­son to be ser­viced in order, in an effi­cient and fair manner.

[Craft CMS](https://craftcms.com/) also uses a queue, too. Any lengthy task is pack­aged off into a  **job** , and jobs are put into a  **queue** to be exe­cut­ed in order.

![Queue enqueue dequeue craft cms](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_1200x630_crop_center-center_100_line/queue-enqueue-dequeue-craft-cms.png)

In com­put­er sci­ence terms, this is a First In, First Out (FIFO) data struc­ture that ser­vices things in what we’d con­sid­er to be a nat­ur­al order.

Here are some lengthy tasks that end up as jobs in the Craft CMS queue:

- Updat­ing entry drafts and revisions
- Delet­ing stale tem­plate caches
- Gen­er­at­ing pend­ing image transforms
- Re-sav­ing Elements
- Updat­ing search indexes
- Plu­g­in cre­at­ed queue jobs

The queue exists to allow these often lengthy jobs to run in the back­ground with­out block­ing access to the Craft CMS Con­trol Pan­el (CP) while they execute.

## See How They Run

So now what we know what type of jobs will end up in the queue, let’s have a look at how these jobs are run. This ends up being a cru­cial­ly impor­tant bit of infor­ma­tion to under­stand about queue jobs.

![Queue runners](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_1200x675_crop_center-center_82_line/queue-runners.jpg)

Giv­en that Craft CMS is built on top of the [Yii2 frame­work](https://www.yiiframework.com/), it makes sense that it lever­ages the exist­ing [Yii2-queue](https://github.com/yiisoft/yii2-queue) by lay­er­ing a [QueueIn­ter­face](https://github.com/craftcms/cms/blob/develop/src/queue/QueueInterface.php) on top of it.

It does this so that it can pro­vide a nice friend­ly sta­tus UI in the CP. The default Craft set­up uses a data­base (Db) dri­ver for queue jobs, stor­ing them in the <kbd>queue</kbd> table.

Although there are oth­er [queue dri­vers](https://github.com/yiisoft/yii2-queue/blob/master/docs/guide/README.md#queue-drivers) avail­able, you’re unlike­ly to see any gains from using any­thing oth­er than the Craft CMS Db dri­ver. And you’d lose the afore­men­tioned sta­tus UI in the CP.

<aside>
                                Where queue jobs are stored and how they are run are com­plete­ly sep­a­rate things.
                            </aside>

But how do the queue jobs run? Here’s how:

- When some­one access the CP
- Via web request

Here­in lies the rub with the default queue imple­men­ta­tion in Craft CMS. No queue jobs will run at all unless some­one logs into the CP, and even then, they are run via web request.

Here’s an excerpt from the Craft CMS <kbd>CP.js</kbd> JavaScript that kicks off the the web-based queue running:

```

runQueue: function() {
    if (!this.enableQueue) {
        return;
    }

    if (Craft.runQueueAutomatically) {
        Craft.queueActionRequest('queue/run', $.proxy(function(response, textStatus) {
            if (textStatus === 'success') {
                this.trackJobProgress(false, true);
            }
        }, this));
    }
    else {
        this.trackJobProgress(false, true);
    }
},

```

And here’s the [queue/​run con­troller method](https://github.com/craftcms/cms/blob/develop/src/controllers/QueueController.php#L57) that ends up get­ting exe­cut­ed via AJAX:

```

/**
 * Runs any waiting jobs.
 *
 * @return Response
 */
public function actionRun(): Response
{
    // Prep the response
    $response = Craft::$app->getResponse();
    $response->content = '1';
    // Make sure Craft is configured to run queues over the web
    if (!Craft::$app->getConfig()->getGeneral()->runQueueAutomatically) {
        return $response;
    }
    // Make sure the queue isn't already running, and there are waiting jobs
    $queue = Craft::$app->getQueue();
    if ($queue->getHasReservedJobs() || !$queue->getHasWaitingJobs()) {
        return $response;
    }
    // Attempt to close the connection if this is an Ajax request
    if (Craft::$app->getRequest()->getIsAjax()) {
        $response->sendAndClose();
    }
    // Run the queue
    App::maxPowerCaptain();
    $queue->run();
    return $response;
}

```

They are doing what they can to make sure the web-based queue jobs will run smooth­ly, but there’s only so much that can be done. Bran­don Kel­ly from Pix­el & Ton­ic even not­ed [​“Craft 2, and Craft3 by default, basi­cal­ly have a poor-man’s queue”](https://github.com/craftcms/cms/issues/3011#issuecomment-399521965).

<aside>
                                Run­ning lengthy oper­a­tions via web request rarely ends well
                            </aside>

Web requests are real­ly meant for rel­a­tive­ly short, atom­ic oper­a­tions like load­ing a web page. For this rea­son, both web servers and PHP process­es have time­outs to pre­vent them from tak­ing too long.

![Stuck queue jobs](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_1200x780_crop_center-center_100_line/stuck-queue-jobs.png)

This can result in bad things happening:

- Queue jobs stack up until some­one logs into the CP
- Queue jobs can fail or get stuck if they take too long and time out
- Queue jobs can fail or get stuck if they run out of memory
- Lengthy queue jobs can impact CP or fron­tend per­for­mance as they run

For­tu­nate­ly, we can alle­vi­ate all of these prob­lems, and it’s not hard to do!

If you do have tasks that have failed due to errors, check out the [Zen and the Art of Craft CMS Log File Read­ing](https://dev.to/gaijinity/zen-and-the-art-of-craft-cms-log-file-reading-3dp3-temp-slug-5047256) arti­cle to learn how to debug it.

## Queue Run­ning Solutions

For­tu­nate­ly Yii2, and by exten­sion Craft CMS, comes with a robust set of com­mand line tools that I detail in the [Explor­ing the Craft CMS 3 Con­sole Com­mand Line Inter­face (CLI)](https://dev.to/gaijinity/exploring-the-craft-cms-3-console-command-line-interface-cli-452-temp-slug-1468335) article.

![Shift into high gear](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_1200x675_crop_center-center_82_line/shift-into-high-gear.jpg)

The <kbd>queue</kbd> com­mand gives us the abil­i­ty to shift our queue jobs into high gear by run­ning them direct­ly. PHP code that is run via the con­sole com­mand line inter­face (CLI) uses a sep­a­rate <kbd>php.ini</kbd> file than the web-ori­ent­ed <kbd>php-fpm</kbd>. Here’s an exam­ple from my server:

```

/etc/php/7.1/fpm/php.ini
/etc/php/7.1/cli/php.ini

```

The rea­son this is impor­tant is that while the web-based <kbd>php-fpm</kbd> might have rea­son­able set­tings for [memory\_​limit](https://www.php.net/manual/en/ini.core.php#ini.memory-limit) and [max\_​execution\_​time](https://www.php.net/manual/en/info.configuration.php#ini.max-execution-time), the CLI-based PHP will typ­i­cal­ly have no <kbd>memory_limit</kbd> and no <kbd>max_execution_time</kbd>.

This is because web process­es are exter­nal­ly trig­gered and there­fore untrust­ed, where­as things run­ning via the CLI is inter­nal and trusted.

So run­ning our queue jobs via PHP CLI gives us the fol­low­ing benefits:

- Can be run any time, not just when some­one access­es the CP
- Won’t be ter­mi­nat­ed due to lengthy exe­cu­tion time
- Are unlike­ly to run out of memory

The queue con­sole com­mands we use are:

- <kbd>queue/run</kbd> — Runs all jobs in the queue, then exits
- <kbd>queue/listen</kbd> — Per­pet­u­al­ly runs a process that lis­tens for new jobs added to the queue and runs them

There are sev­er­al ways we can run these com­mands, dis­cussed in the [Yii2 Work­er Start­ing Con­trol](https://www.yiiframework.com/extension/yiisoft/yii2-queue/doc/guide/2.0/en/worker) arti­cle, and we’ll go into detail on them here.

## Solu­tion #1: Async Queue Plugin

If you want a no fuss, no muss solu­tion that just involves installing a plu­g­in, check out Oliv­er Stark’s [Async Queue plu­g­in.](https://github.com/ostark/craft-async-queue) From the plug­in’s description:

_[Async Queue plu­g­in] replaces Craft’s default queue han­dler and moves queue exe­cu­tion to a non-block­ing back­ground process. The com­mand craft <kbd>queue/run</kbd> gets exe­cut­ed right after you push a Job to the queue._

If you want a sim­ple solu­tion that does­n’t require doing any­thing more than installing a plu­g­in, go this route. You’ll end up with a more robust solu­tion for run­ning your queue jobs.

**That’s it, you’re done!**

## Solu­tion #2: Forge Daemons

If you use the won­der­ful [Lar­avel Forge](https://forge.laravel.com/) to pro­vi­sion your servers (dis­cussed in detail in the [How Agen­cies & Free­lancers Should Do Web Host­ing](https://dev.to/gaijinity/how-agencies-freelancers-should-do-web-hosting-ih4-temp-slug-5481912) arti­cle), Forge pro­vides a nice GUI inter­face to what it calls **Dae­mons**.

Go to your Serv­er, then click on Dae­mons, and you’ll see a GUI that looks like this:

![Forge new daemon](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_1200x505_crop_center-center_100_line/forge-new-daemon.png)

Here’s the com­mand we’ve added for the [dev​Mode​.fm](https://devmode.fm/) website:

```

/usr/bin/nice -n 10 /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose

```

Let’s break this down:

- <kbd>/usr/bin/nice -n 10</kbd> — this uses the Unix [nice](https://en.wikipedia.org/wiki/Nice_(Unix)) com­mand to run the process using a low­er pri­or­i­ty, so it won’t inter­fere with the CP or frontend
- <kbd>/usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose</kbd> — this starts a PHP CLI process to lis­ten for any queue jobs, and run them

You should swap in your own path to the <kbd>craft</kbd> CLI exe­cutable; ours is <kbd>/home/forge/devmode.fm/craft</kbd>

We tell it that we want to run this as the user <kbd>forge</kbd> and we want <kbd>2</kbd> of these process­es run­ning. Click **Start Dae­mon** and you’ll be greet­ed with:

![Forge daemon added](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_1200x246_crop_center-center_100_line/forge-daemon-added.png)

Then ensure that you’ve set the [run­QueueAu­to­mat­i­cal­ly](https://docs.craftcms.com/v3/config/config-settings.html#runqueueautomatically) Craft CMS <kbd>general.php</kbd> con­fig set­ting to <kbd>false</kbd> for your <kbd>live</kbd> pro­duc­tion environment:

```

    // Live (production) environment
    'live' => [
        // Craft defined config settings
        'runQueueAutomatically' => false,
    ],

```

This makes sure that Craft will no longer try to run its queue via web requests.

**That’s it, you’re done!**

The fol­low­ing is an entire­ly option­al peek under the hood at how this works with Lar­avel Forge. Feel free to skip it if you’re not inter­est­ed in the gory details.

What Forge calls Dae­mons are actu­al­ly com­mands run via the [Super­vi­sor](http://supervisord.org/) com­mand. When we cre­at­ed the above dae­mon in the Forge GUI, it cre­at­ed a file in <kbd>/etc/supervisor/conf.d</kbd>:

```

forge@nys-production /etc/supervisor/conf.d $ ls -al
-rw-r--r-- 1 root root 293 Aug 3 14:55 daemon-157557.conf

```

The con­tents of the file look like this:

```

[program:daemon-157557]
command=/usr/bin/nice -n 10 /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose

process_name=%(program_name)s_%(process_num)02d
autostart=true
autorestart=true
user=forge
numprocs=2
redirect_stderr=true
stdout_logfile=/home/forge/.forge/daemon-157557.log

```

And indeed, this is exact­ly one of the meth­ods rec­om­mend­ed in the in the [Yii2 Work­er Start­ing Con­trol](https://www.yiiframework.com/extension/yiisoft/yii2-queue/doc/guide/2.0/en/worker) arti­cle. Huzzah!

## Solu­tion #3 Using Heroku

If you’re using [Heroku](https://www.heroku.com/) for your host­ing needs, you can use [Work­er Dynos](https://devcenter.heroku.com/articles/background-jobs-queueing) to run your <kbd>queue/listen</kbd> command:

```

./craft queue/listen --verbose

```

Then ensure that you’ve set the [run­QueueAu­to­mat­i­cal­ly](https://docs.craftcms.com/v3/config/config-settings.html#runqueueautomatically) Craft CMS <kbd>general.php</kbd> con­fig set­ting to <kbd>false</kbd> for your <kbd>live</kbd> pro­duc­tion environment:

```

    // Live (production) environment
    'live' => [
        // Craft defined config settings
        'runQueueAutomatically' => false,
    ],

```

This makes sure that Craft will no longer try to run its queue via web requests.

**That’s it, you’re done!**

## Solu­tion #4: Uti­liz­ing systemd

If you’re using a Lin­ux serv­er, but you’re not using Forge or Heroku, or per­haps you just want to con­fig­ure things your­self, we can accom­plish the same thing using [sys­temd](https://www.linode.com/docs/quick-answers/linux-essentials/what-is-systemd/).

<kbd>systemd</kbd> is a way to start, stop, and oth­er­wise man­age dae­mon process­es that run on your serv­er. You’re already using it whether you know it or not, for things like your web serv­er, MySQL, and so on.

Here’s what I did for [dev​Mode​.fm](https://devmode.fm/) to use <kbd>systemd</kbd> to run our queue jobs. You’ll need <kbd>sudo</kbd> access to be able to do this, but for­tu­nate­ly most mod­ern VPS host­ing ser­vices give you this ability.

First I cre­at­ed a <kbd>.service</kbd> file in <kbd>/etc/systemd/system/</kbd> named <kbd>devmode-queue@.service</kbd> via:

```

forge@nys-production ~ $ sudo nano /etc/systemd/system/devmode-queue@.service

```

Here’s what the con­tents of the file looks like: 

```

[Unit]
Description=devMode.fm Queue Worker %I
After=network.target
After=mysql.service
Requires=mysql.service

[Service]
User=forge
Group=forge
ExecStart=/usr/bin/nice -n 10 /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose
Restart=always

[Install]
WantedBy=multi-user.target

```

Let’s break this down:

- <kbd>/usr/bin/nice -n 10</kbd> — this uses the Unix [nice](https://en.wikipedia.org/wiki/Nice_(Unix)) com­mand to run the process using a low­er pri­or­i­ty, so it won’t inter­fere with the CP or frontend
- <kbd>/usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose</kbd> — this starts a PHP CLI process to lis­ten for any queue jobs, and run them

You should swap in your own path to the <kbd>craft</kbd> CLI exe­cutable; ours is <kbd>/home/forge/devmode.fm/craft</kbd>

We tell it that we want to run this as the user <kbd>forge</kbd> , we want it to restart the process on fail­ure and that it depends on both the <kbd>network</kbd> and <kbd>mysql</kbd> ser­vices being available.

The strange <kbd>@</kbd> in the <kbd>.service</kbd> file name allows us to have mul­ti­ple instances of this ser­vice run­ning. We can then use the range syn­tax <kbd>{1..2}</kbd> to spec­i­fy how many of these process­es we want.

We’ve cre­at­ed the file, but we need to start the ser­vice via:

```

forge@nys-production ~ $ sudo systemctl start devmode-queue@{1..2}

```

And then just like our oth­er ser­vices such as <kbd>nginx</kbd> and <kbd>mysqld</kbd> we want it to start up auto­mat­i­cal­ly when we restart our server:

```

forge@nys-production ~ $ sudo systemctl enable devmode-queue@{1..2}

```

Then ensure that you’ve set the [run­QueueAu­to­mat­i­cal­ly](https://docs.craftcms.com/v3/config/config-settings.html#runqueueautomatically) Craft CMS <kbd>general.php</kbd> con­fig set­ting to <kbd>false</kbd> for your <kbd>live</kbd> pro­duc­tion environment:

```

    // Live (production) environment
    'live' => [
        // Craft defined config settings
        'runQueueAutomatically' => false,
    ],

```

This makes sure that Craft will no longer try to run its queue via web requests.

**That’s it, you’re done!**

Now you can start, stop, enable, dis­able, etc. your Craft CMS queue run­ner just like you would any oth­er sys­tem ser­vice. You can also mon­i­tor it via the [jour­nalctl](https://www.linode.com/docs/quick-answers/linux/how-to-use-journalctl/) command:

```

forge@nys-production ~ $ sudo journalctl -f -u devmode-queue@*.service
-- Logs begin at Sat 2019-08-03 13:23:17 EDT. --
Aug 03 14:06:04 nys-production nice[1364]: Processing element 42/47 - Neutrino: How I Learned to Stop Worrying and Love Webpack
Aug 03 14:06:04 nys-production nice[1364]: Processing element 43/47 - Google AMP: The Battle for the Internet's Soul?
Aug 03 14:06:04 nys-production nice[1364]: Processing element 44/47 - The Web of Future Past with John Allsopp
Aug 03 14:06:04 nys-production nice[1364]: Processing element 45/47 - Web Hosting with ArcusTech's Nevin Lyne
Aug 03 14:06:04 nys-production nice[1364]: Processing element 46/47 - Tailwind CSS utility-first CSS with Adam Wathan
Aug 03 14:06:04 nys-production nice[1364]: Processing element 47/47 - Talking Craft CMS 3 RC1!
Aug 03 14:06:04 nys-production nice[1364]: 2019-08-03 14:06:04 [1427] Generating episodes sitemap (attempt: 1) - Done (1.377 s)
Aug 04 09:25:25 nys-production nice[1364]: 2019-08-04 09:25:25 [1429] Generating calendar sitemap (attempt: 1) - Started
Aug 04 09:25:25 nys-production nice[1364]: Processing element 1/1 - Calendar of Upcoming Episodes
Aug 04 09:25:25 nys-production nice[1364]: 2019-08-04 09:25:25 [1429] Generating calendar sitemap (attempt: 1) - Done (0.174 s)

```

The above com­mand will tail the <kbd>systemd</kbd> logs, only show­ing mes­sages from our <kbd>devmode-queue</kbd> service.

You can learn more about using <kbd>journalctl</kbd> in the [How To Use Jour­nalctl to View and Manip­u­late Sys­temd Logs](https://www.digitalocean.com/community/tutorials/how-to-use-journalctl-to-view-and-manipulate-systemd-logs) article.

## Get­ting Dotenvious

This is an entire­ly option­al sec­tion you can skip if you’re not inter­est­ed in learn­ing how to not use <kbd>.env</kbd> files in pro­duc­tion, as rec­om­mend­ed by the [php­dotenv](https://github.com/vlucas/phpdotenv) pack­age authors:

_php­dotenv is made for devel­op­ment envi­ron­ments, and gen­er­al­ly should not be used in pro­duc­tion. **In pro­duc­tion, the actu­al envi­ron­ment vari­ables should be set so that there is no over­head of load­ing the .env file on each request.** This can be achieved via an auto­mat­ed deploy­ment process with tools like Vagrant, chef, or Pup­pet, or can be set man­u­al­ly with cloud hosts like Pagod­abox and Heroku._

<aside>
                                You do not need to do any of this unless you do not use .env in production
                            </aside>

Addi­tion­al­ly some setups have envi­ron­men­tal vari­ables intro­duced by alter­nate means. I made a [Doten­vy](https://github.com/nystudio107/dotenvy) pack­age that makes it easy to gen­er­ate <kbd>.env</kbd> key/​value vari­able pairs as Apache, Nginx, and shell equivalents.

In this case, we’ll need to mod­i­fied the com­mand we run slight­ly, so that we’re defin­ing our envi­ron­men­tal vari­ables in the shell before run­ning Craft via the CLI.

All we need to do is change our com­mand to this:

```

/bin/bash -c ". /home/forge/devmode.fm/.env_cli.txt && /usr/bin/nice -n 10 /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose"

```

All we’re doing is exe­cut­ing the <kbd>.env_cli.txt</kbd> file (cre­at­ed by [Doten­vy](https://github.com/nystudio107/dotenvy)) that con­tains our envi­ron­men­tal vari­ables we <kbd>export</kbd> to the shell environment:

```

forge@nys-production ~/devmode.fm (master) $ cat .env_cli.txt
# CLI (bash) .env variables
# Paste these inside your .bashrc file in your $HOME directory:
export ENVIRONMENT="live"
export SECURITY_KEY="XXXXX"
export DB_DRIVER="pgsql"
export DB_SERVER="localhost"
export DB_USER="devmode"
export DB_PASSWORD="XXXXX"
export DB_DATABASE="devmode"
export DB_SCHEMA="public"
export DB_TABLE_PREFIX=""
export DB_PORT="5432"
export SITE_URL="https://devmode.fm/"
export BASE_PATH="/home/forge/devmode.fm/web/"
export BASE_URL="https://devmode.fm/"
export REDIS_HOSTNAME="localhost"
export REDIS_PORT="6379"
export REDIS_DEFAULT_DB="0"
export REDIS_CRAFT_DB="3"
export PUBLIC_PATH="/dist/"
export DEVSERVER_PUBLIC="http://192.168.10.10:8080"
export DEVSERVER_HOST="0.0.0.0"
export DEVSERVER_POLL="1"
export DEVSERVER_PORT="8080"
export DEVSERVER_HTTPS="0"

```

The rest is the same as the com­mands we’ve dis­cussed above for run­ning <kbd>queue/listen</kbd>, and then we wrap it all in <kbd>/bin/sh -c</kbd> so it appears as a sin­gle com­mand that can be executed.

## Deeply Queued

Here’s some queue job triv­ia that you may find interesting.

![Craft cms queue job trivia](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_1200x675_crop_center-center_82_line/craft-cms-queue-job-trivia.jpg)

We men­tioned ear­li­er that the only way queue jobs are run nor­mal­ly is if you vis­it the CP. This is _most­ly true_.

There’s actu­al­ly a case where queue jobs can be run via fron­tend page load. If you have [gen­er­ate­Trans­forms­Be­forePage­Load](https://docs.craftcms.com/v3/config/config-settings.html#generatetransformsbeforepageload) set to <kbd>false</kbd> (which is the default), and there are pend­ing trans­forms that need to be generated.

What hap­pens is the <kbd>Assets::getAssetUrl()</kbd> method push­es the image trans­form queue job, which calls <kbd>Queue::push()</kbd>:

```

    /**
     * @inheritdoc
     */
    public function push($job)
    {
        // Capture the description so pushMessage() can access it
        if ($job instanceof JobInterface) {
            $this->_jobDescription = $job->getDescription();
        } else {
            $this->_jobDescription = null;
        }

        if (($id = parent::push($job)) === null) {
            return null;
        }

        // Have the response kick off a new queue runner if this is a site request
        if (Craft::$app->getConfig()->getGeneral()->runQueueAutomatically && !$this->_listeningForResponse) {
            $request = Craft::$app->getRequest();
            if ($request->getIsSiteRequest() && !$request->getIsAjax()) {
                Craft::$app->getResponse()->on(Response::EVENT_AFTER_PREPARE, [$this, 'handleResponse']);
                $this->_listeningForResponse = true;
            }
        }

        return $id;
    }

```

So if this is a site (fron­tend) request, and it’s not an AJAX request, Craft will add a call to <kbd>Queue::handleResponse()</kbd> that gets fired when the <kbd>Response</kbd> is about to be sent back to the client.

In this case, Craft will inject some JavaScript into the fron­tend request that pings the <kbd>queue/run</kbd> con­troller via AJAX:

```

    /**
     * Figure out how to initiate a new worker.
     */
    public function handleResponse()
    {
        // Prevent this from getting called twice
        $response = Craft::$app->getResponse();
        $response->off(Response::EVENT_AFTER_PREPARE, [$this, 'handleResponse']);

        // Ignore if any jobs are currently reserved
        if ($this->getHasReservedJobs()) {
            return;
        }

        // Ignore if this isn't an HTML/XHTML response
        if (!in_array($response->getContentType(), ['text/html', 'application/xhtml+xml'], true)) {
            return;
        }

        // Include JS that tells the browser to fire an Ajax request to kick off a new queue runner
        // (Ajax request code adapted from http://www.quirksmode.org/js/xmlhttp.html - thanks ppk!)
        $url = Json::encode(UrlHelper::actionUrl('queue/run'));
        $js = <<<EOD
<script type="text/javascript">
/*<![CDATA[*/
(function(){
    var XMLHttpFactories = [
        function () {return new XMLHttpRequest()},
        function () {return new ActiveXObject("Msxml2.XMLHTTP")},
        function () {return new ActiveXObject("Msxml3.XMLHTTP")},
        function () {return new ActiveXObject("Microsoft.XMLHTTP")}
    ];
    var req = false;
    for (var i = 0; i < XMLHttpFactories.length; i++) {
        try {
            req = XMLHttpFactories[i]();
        }
        catch (e) {
            continue;
        }
        break;
    }
    if (!req) return;
    req.open('GET', $url, true);
    if (req.readyState == 4) return;
    req.send();
})();
/*]]>*/
</script>
EOD;

        if ($response->content === null) {
            $response->content = $js;
        } else {
            $response->content .= $js;
        }
    }

```

While tech­ni­cal­ly this could hap­pen for any queue job that is <kbd>Queue::pushed()</kbd>​‘d in the queue dur­ing a fron­tend request, asset trans­forms are the only case I’m aware of that this hap­pens in practice.

For this rea­son, if you’re post­ing any XHR requests to Craft con­trollers or API end­points, make sure you add the header:

```

'X-Requested-With': 'XMLHttpRequest'

```

Craft looks for this head­er to deter­mine whether the request is AJAX or not, and will not attempt to attach its queue run­ner JavaScript to the request if the request is AJAX.

This head­er is set auto­mat­i­cal­ly by jQuery for <kbd>$.ajax()</kbd> requests, but it is not set auto­mat­i­cal­ly by pop­u­lar libraries such as [Axios](https://www.npmjs.com/package/axios).

## Wrap­ping Up

So which method do I use? If I’m spin­ning up a VPS with Forge (which I nor­mal­ly am), then I’ll use **Solu­tion #2: Forge Dae­mons**. If it’s not a Forge box, but I do have <kbd>sudo</kbd> access, then I’ll use **Solu­tion #3: Uti­liz­ing sys­temd**.

I use **Solu­tion #1: Async Queue Plu­g­in** on client sites where nei­ther of the above are the case. Oliv­er’s plu­g­in is great, and much bet­ter than the default way that queue jobs are run, but I like lever­ag­ing exist­ing sys­tems if pos­si­ble, and hav­ing <kbd>nice</kbd> con­trol over the process pri­or­i­ty is, well, nice.

![There is always hope](https://nystudio107-ems2qegf7x6qiqq.netdna-ssl.com/img/blog/_1200x675_crop_center-center_82_line/there-is-always-hope.jpg)

Replac­ing the default web-based queue job sys­tem in Craft CMS with a more robust CLI-based sys­tem will pay dividends.

Your queue jobs will run more smooth­ly, fail less often, and have less of an impact on the CP/​frontend.

It’s worth it. Do it.

Hap­py queuing!

## Further Reading

_If you want to be notified about new articles, follow [nystudio107](https://twitter.com/nystudio107) on Twitter._

<small>Copyright ©2020 nystudio107. Designed by nystudio107</small>