Monitoring Laravel’s background queues in realtime using Pusher
Tags: laravel, pusher, realtime
Published: 2018-03-01 19:01:20

---
title: Monitoring Laravel’s background queues in realtime using Pusher
published: true
description: 
tags: laravel, pusher, realtime
---


When building large applications, making it scale is usually a concern. Stats like how long it takes for the page to load is usually very important. Thus, doing things like processing large images, sending emails and SMS can be pushed to a background queue and processed at a later time.

However, because queues work in the background, they can fail sometimes. It may then be necessary to be able to monitor background queues.

In this article, we will consider ways to monitor Laravel’s background queues in realtime. We will assume we created an application that sends emails. The emails are to be queued in the background and sent later. We will then have a report page with the emails that have been sent and those that haven’t.

This is a screen recording of what we will be building:

![](https://blog.pusher.com/wp-content/uploads/2017/12/monitoring-laravel-background-queues-realtime-pusher-demo.gif)

## Tutorial Requirements

To follow along in this tutorial, we would need the following things:

- PHP 7.0+ installed on your machine.
- [Laravel CLI](https://laravel.com/docs/5.5/installation) installed on your machine.
- [Composer](https://getcomposer.org/doc/00-intro.md#installation-linux-unix-osx) installed on your machine.
- Knowledge of PHP and Laravel.
- [Node.js and NPM](https://docs.npmjs.com/getting-started/installing-node) installed on your machine.
- Basic knowledge of Vue.js and JavaScript.
- A Pusher application. Create one [here](https://pusher.com).
- A Mailtrap account to test emails sent. Create one [here](https://mailtrap.io).

Once you have these requirements ready, let’s start.

## Setting up Your Laravel Application

Open the terminal and run the command below to create a Laravel application:

```php 
    $ laravel new app_name

```

**Setting Up a Database Connection and Migration**
When installation is complete, we can move on to setting up the database. Open the `.env` file and replace the configuration below:

```
    DB_CONNECTION=mysql
    DB_HOST=127.0.0.1
    DB_PORT=3306
    DB_DATABASE=homestead
    DB_USERNAME=homestead
    DB_PASSWORD=secret

```with:

```
    DB_CONNECTION=sqlite

```

This will set SQLite as our default database connection (you can use MySQL or any other database connection you want).

In the terminal `cd` to the root directory of your project. Run the command below to create the SQLite database file:

```
    $ touch database/database.sqlite

```

The command above will create an empty file that will be used by SQLite. Run the command below to create a migration:

```php 
    $ php artisan make:migration create_queued_emails_table

```

Open up the migration file that was just created by the command above and replace the `up` method with the code below:

```php 
    public function up()
    {
        Schema::create('queued_emails', function (Blueprint $table) {
            $table->increments('id');
            $table->string('email');
            $table->string('description');
            $table->boolean('run')->default(false);
            $table->timestamps();
        });
    }

```

Now run the command below to migrate our database:

```php 
    $ php artisan migrate

```

**Setting Up Mailtrap for Email Testing**
Open your `.env` file and enter the keys you got from the Mailtrap dashboard. The relevant keys are listed below:

```
    MAIL_DRIVER=smtp
    MAIL_HOST=smtp.mailtrap.io
    MAIL_PORT=2525
    MAIL_USERNAME=null
    MAIL_PASSWORD=null
    MAIL_ENCRYPTION=null
    MAIL_FROM="john@doe.com"
    MAIL_NAME="John Doe"

```

Now when emails are sent, the emails will be visible in the Mailtrap inbox.

**Setting Up Authentication**
The next thing we need to do is set up authentication. Open your terminal and enter the command below:

```php 
    $ php artisan make:auth

```

This will generate an authentication scaffold. That is all that you need to do regarding authentication.

**Configuring Pusher**
Replace the `PUSHER_*` keys in the `.env` file with the correct keys you got from your Pusher dashboard:

```
    PUSHER_APP_ID="PUSHER_APP_ID"
    PUSHER_APP_KEY="PUSHER_APP_KEY"
    PUSHER_APP_SECRET="PUSHER_APP_SECRET" 

```

Open the terminal and enter the command below to install the Pusher PHP SDK:

```
    $ composer require pusher/pusher-php-server "~3.0"

```

After installation is complete, open the `config/broadcasting.php` file and scroll to the `pusher` section. Replace the `options` key with the following:

```php 
    'options' => [
        'encrypt' => true,
        'cluster' => 'PUSHER_APP_CLUSTER'
    ],

```

**Configuring Other Miscellaneous Things**
Open the `.env` file and change the `BROADCAST_DRIVER` to `pusher`, and the `QUEUE_DRIVER` to `database`. To make sure we have the tables necessary to use `database` as our `QUEUE_DRIVER` run the command below to generate the database migration:

```php 
    $ php artisan queue:table

```

Then run the migrate command to migrate the database:

```php 
    $ php artisan migrate

```

This will create the database table required to use our database as a queue driver.

> 💡 **In a production environment, it is better to use an in-memory storage like Redis or Memcached as the queue driver. In-memory storage is faster and thus has better performance than using a relational database.**

## Building the Backend of Our Application

Now let’s create the backend of our application. Run the command below in your terminal:

```php 
    $ php artisan make:model QueuedEmails

```

This will create a new model in the `app` directory. Open the file and replace the contents with the following:

```php 
    <?php

    namespace App;

    use Illuminate\Database\Eloquent\Model;

    class QueuedEmails extends Model
    {
        protected $fillable = ['description', 'run', 'email'];
        protected $casts = ['run' => "boolean"];
    }

```

In the code above, we define the `fillable` property of the class. This will prevent a mass assignment exception error when we try to create a new entry to the database. We also specify a `casts` property which will instruct Eloquent to typecast attributes to data types.

Next, open the `HomeController` and and replace the contents with the code below:

```php 
    <?php
    namespace App\Http\Controllers;

    use Mail;
    use App\QueuedEmails;
    use App\Mail\SimulateMail;
    use Faker\Factory as Faker;

    class HomeController extends Controller
    {
        /**
         * Create a new controller instance.
         *
         * @return void
         */
        public function __construct()
        {
            $this->middleware('auth');

            $this->faker = Faker::create();
        }

        /**
         * Show the application dashboard.
         *
         * @return \Illuminate\Http\Response
         */
        public function index()
        {
            return view('home', ['jobs' => $this->jobs()]);
        }

        /**
         * Return all the jobs.
         *
         * @return array
         */
        public function jobs()
        {
            return QueuedEmails::orderBy('created_at', 'DESC')->get()->toArray();
        }

        /**
         * Simulate sending the email.
         *
         * @return mixed
         */
        public function simulate()
        {
            $email = $this->faker->email;

            Mail::to($email)->send(
                new SimulateMail([
                    "email" => $email,
                    "description" => $this->faker->sentence()
                ])
            );

            return redirect()->route('home');
        }
    }

```

In the controller above, we have 4 methods that are mostly self-explanatory. In the class we use the [Faker library](https://github.com/fzaninotto/Faker) which helps us generate random fake values. In the `simulate` method, we are using the faker library to generate a fake email address and description. We instantiate a `SimulateMail` [mailable](https://laravel.com/docs/5.5/mail#generating-mailables).

Open the terminal and enter the command below:

```php 
    $ php artisan make:mail SimulateMail

```

Open the `SimulateMail` class and enter the code below:

```php 
    <?php
    namespace App\Mail;

    use App\QueuedEmails;
    use Illuminate\Bus\Queueable;
    use Illuminate\Mail\Mailable;
    use Illuminate\Queue\SerializesModels;
    use App\Events\{EmailQueued, EmailSent};
    use Illuminate\Contracts\Queue\ShouldQueue;
    use Illuminate\Contracts\Queue\Factory as Queue;
    use Illuminate\Contracts\Mail\Mailer as MailerContract;

    class SimulateMail extends Mailable implements ShouldQueue
    {
        use Queueable, SerializesModels;

        protected $mail;

        /**
         * Create a new message instance.
         *
         * @return void
         */
        public function __construct(array $mail)
        {
            $this->mail = QueuedEmails::create($mail);
        }

        /**
         * Build the message.
         *
         * @return $this
         */
        public function build()
        {
            return $this->subject("Queuer: Welcome to queuer")->view('email.welcome');
        }

        /**
         * Send the mail
         */
        public function send(MailerContract $mailer)
        {
            $this->mail->update(['run' => true]);

            event(new EmailSent($this->mail));

            parent::send($mailer);
        }

        /**
         * Queue the email
         */
        public function queue(Queue $queue)
        {
            event(new EmailQueued($this->mail));

            return parent::queue($queue);
        }
    }

```

> 💡 **By implementing the**`**ShouldQueue**` **interface, we are telling Laravel that the email should be queued and not sent immediately.**

In the class above, we have a constructor that creates a new entry into the `queued_emails` table. In the `build` method, we build the mail message we are going to be sending.

In the `send` method, we mark the `queued_emails` entry’s `run` column to `true`. We also fire an event called `EmailSent`. In the `queue` method, we also trigger an event called `EmailQueued`.

Let’s create the events we triggered in the methods above. In your terminal run the command below:

```php 
    $ php artisan make:event EmailSent
    $ php artisan make:event EmailQueued

```

In the `EmailSent` event class, paste the following code:

```php 
    <?php
    namespace App\Events;

    use App\QueuedEmails;
    use Illuminate\Broadcasting\Channel;
    use Illuminate\Queue\SerializesModels;
    use Illuminate\Foundation\Events\Dispatchable;
    use Illuminate\Broadcasting\InteractsWithSockets;
    use Illuminate\Contracts\Broadcasting\ShouldBroadcast;

    class EmailSent implements ShouldBroadcast
    {
        use Dispatchable, InteractsWithSockets, SerializesModels;

        public $mail;

        public function __construct($mail)
        {
            $this->mail = $mail;
        }

        public function broadcastOn()
        {
            return new Channel('email-queue');
        }

        public function broadcastAs()
        {
            return 'sent';
        }
    }

```

In the code above, we just use [Broadcasting](https://laravel.com/docs/5.5/broadcasting#defining-broadcast-events) in Laravel to send some data to Pusher.

Open the `EmailQueued` event class and paste the code below:

```php 
    <?php

    namespace App\Events;

    use App\QueuedEmails;
    use Illuminate\Broadcasting\Channel;
    use Illuminate\Queue\SerializesModels;
    use Illuminate\Foundation\Events\Dispatchable;
    use Illuminate\Broadcasting\InteractsWithSockets;
    use Illuminate\Contracts\Broadcasting\ShouldBroadcast;

    class EmailQueued implements ShouldBroadcast
    {
        use Dispatchable, InteractsWithSockets, SerializesModels;

        public $mail;

        public function __construct($mail)
        {
            $this->mail = $mail;
        }

        public function broadcastOn()
        {
            return new Channel('email-queue');
        }

        public function broadcastAs()
        {
            return 'add';
        }
    }

```

This class is almost the same as the `EmailSent` event class. The minor difference is the `broadcastAs` method. It returns a different alias to broadcast the event as.

Finally, open the routes file `routes/web.php` and replace the code with this:

```php 
    Auth::routes();
    Route::name('jobs')->get('/jobs', 'HomeController@jobs');
    Route::name('simulate')->get('/simulate', 'HomeController@simulate');
    Route::name('home')->get('/home', 'HomeController@index');
    Route::view('/', 'welcome');

```

Great! Now let’s move on to the frontend of the application.

## Building the Frontend of Our Application

Now that we have set up most of the backend, we will create the frontend of the application. Open the `resources/views/home.blade.php` file and replace the code with the following:

```php 
    @extends('layouts.app')

    @section('content')
    <div class="container">
        <div class="row">
            <div class="col-md-12">
                <div class="panel panel-default">
                    <div class="panel-heading clearfix">
                        <span class="pull-left">Queue Reports</span>
                        <a href="{{ route('simulate') }}" class="btn btn-sm btn-primary pull-right">Simulate</a>
                    </div>
                    <div class="panel-body">
                        <jobs :jobs='@json($jobs)'></jobs></jobs>
                    </div>
                </div>
            </div>
        </div>
    </div>
    @endsection

```

The noteworthy aspect of the code above is the `jobs` tag. This is a reference to the Vue component we will create next. We also have a “Simulate” button that leads to a `/simulate` route. This route simulates queuing an email to be sent.

Open your terminal and type in the command below:

```
    $ npm install --save laravel-echo pusher-js

```

This will install [Laravel Echo](https://laravel.com/docs/5.5/broadcasting#installing-laravel-echo) and the [Pusher JS SDK](https://github.com/pusher/pusher-js). When the installation is complete, run the command below to install the other NPM dependencies:

```
    $ npm install

```

**Building our Vue Component**
Let’s build the `jobs` Vue component we referenced earlier. Open the `resources/assets/js/app.js` file and replace the code below:

```javascript 
    Vue.component('example', require('./components/ExampleComponent.vue'));

```with:

```javascript 
    Vue.component('jobs', require('./components/JobsComponent.vue'));

```

Now create a new `JobsComponent.vue` file in the `resources/assets/js/components/` directory. In the file, paste in the following code:

```javascript 
    <template>
        <table class="table">
            <tbody>
                <tr v-for="(job, index) in allJobs" :key="index" v-bind:class="{success: job.run, danger: !job.run}">
                    <td width="80%">{{ job.description }}</td>
                    <td>{{ job.created_at }}</td>
                </tr>
            </tbody>
        </table>
    </template>

    <script>
    export default {
        props: ['jobs'],
        data() {
            return {allJobs: this.jobs}
        },
        created() {
            let vm = this
            vm.refreshAllJobs = (e) => axios.get('/jobs').then((e) => (vm.allJobs = e.data))
            Echo.channel('email-queue')
                .listen('.add', (e)  => vm.refreshAllJobs(e))
                .listen('.sent', (e) => vm.refreshAllJobs(e))
        }
    }
    </script>

```

In the Vue component above, we have defined a `template`. In there, we loop through the `jobs` array and list each job’s description and timestamp.

In the `created` method of the Vue component `script`, we have a `refreshAllJobs` function that uses [Axios](https://github.com/axios/axios) (a HTTP request library built-in Laravel by default) to make a request to the `/jobs` route. We then assign the response to the `allJobs` property.

In the same method, we use [Laravel Echo](https://laravel.com/docs/5.5/broadcasting#installing-laravel-echo) to listen to a Pusher channel and wait for an event to be triggered. Whenever the events `.add` and `.sent` are triggered, we call the `refreshAllJobs` method.

> 💡 **The event names have a dot before them because,** \*\*\*\*\*\*in Laravel, whenever you use the\*\* `**broadcastAs**` **method to define an alias** **you need to add the dot\*\*\*\*. Without the dot your event will not be caught by the listener. If you do not provide an alias, Laravel will use the namespace + class as the name of the broadcast event.**

Open the `resources/assets/js/bootstrap.js` file. At the bottom of the file, add the following code:

```javascript 
    import Echo from 'laravel-echo'

    window.Pusher = require('pusher-js');

    window.Echo = new Echo({
        broadcaster: 'pusher',
        key: 'PUSHER_APP_KEY',
        encrypt: true,
        cluster: 'PUSHER_APP_CLUSTER'
    });

```

> ⚠️ **Make sure you replace the**`**PUSHER_APP_KEY**` **and**`**PUSHER_APP_CLUSTER**` **with your Pusher application key and cluster.**

Finally, run the command below to build your assets:

```
    $ npm run dev

```

## Testing Our Application

After the build is complete, start a PHP server if you have not already by running the command below:

```php 
    $ php artisan serve

```

This will create a PHP server so we can preview our application. The URL will be provided on the terminal but the default is http://127.0.0.1:8000.

When you see the Laravel homepage, create a new account using the ”Register” link on the top right corner of the page. Now click the “Simulate” button and you should see a new queued email entry.

Now we will manually execute the processes on our queue using the `queue:listen` artisan command. Open a new terminal window and run the command below:

```php 
    $ php artisan queue:listen

```

This should start executing any queues it sees. As long as the terminal is open and the `queue:listen` command is running, when you click the “Simulate” button the queue will run immediately. If you kill the `queue:listen` command, the queue entries will remain there and not be triggered.

> 💡 **In a production environment, you cannot keep**`**queue:listen**` **running and you might need a worker running on a background process; something like** [**Supervisor**](http://supervisord.org/index.html)\*\*. You can read more about how you can do that\*\* [**here**](https://laravel.com/docs/5.5/queues#running-the-queue-worker)**.**

## Conclusion

In this article, we have been able to create a realtime Laravel queue monitor using Pusher and Vue. Having queues that you can track and quantify can be useful. Hopefully, you picked something from this article. If you have any questions or feedback, feel free to ask in the comments section.

The source code is available on [GitHub](https://github.com/neoighodaro/monitoring-laravel-queues).

This post first appeared on the [Pusher blog](https://blog.pusher.com/monitoring-laravel-background-queues-realtime-pusher/).