Implementing an email queue with bullJs and hosting on Heroku!
Tags: heroku, bull, node, redis
Published: 2020-03-20 05:27:26

---
title: Implementing an email queue with bullJs and hosting on Heroku!
published: true
description: In this article, I show how to add a bullJs email queue to an existing node application and how to host the application on Heroku while running a Redis instance.
tags: #heroku, #bull, #node, #redis
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/euihpo9h5sit9c2scgko.png
---

# The problem...
If you've ever picked up a task that reads like this `Setup an email service with SendGrid and nodemailer` you probably did what most of us would do... Setup SendGrid for production and nodemailer for development and test environments. If you did that successfully and stopped no one would fault you but oftentimes email sending in some applications require just a bit more to keep em running smoothly. Few questions we often forget to ask is 

- How does the increase in volume affect my email setup?
- Would it hold out or work when we have 10,000 users, sending at least one email every minute?
- What happens when my email fails to send on the first attempt for various reasons...

Well, in this article we'll be looking at setting up an email queuing system using bullJs in a nodeJs application and also getting this setup to a production-like environment using Heroku.

# Prerequisite

- Basic knowledge of Javascript and nodeJs
- Heroku account(you can sign up [here](https://signup.heroku.com/))
- Clone this repo 👉🏽 https://github.com/vic3king/bulljs-email-setup.git and follow the readme setup instructions.
- Redis server running on `localhost:6379`. You can set one up quickly by following the instructions [here](https://redis.io/topics/quickstart)

You can also set up your own nodeJs application the way you like it and integrate the code from this example and it should work regardless. But for this article, I will be working with the cloned repo.


The cloned repo contains some boilerplate code set up with node, express, and Sequelize.  I added a registration endpoint that requires the user to input their `username` and `email` address(`server/controllers/auth`) to register. If you followed the setup instructions on the `readMe` you should be able to hit the registration endpoint at `http://127.0.0.1:3333/v1/auth/register` and register a user.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/qvh6n72jy2xrhdzwtcs1.png)

I have SendGrid and nodemailer preconfigured on the application as well in form of a function we can call whenever we need to send an email (`server/config/emailSetup.js`), I also added a base email template function that takes in a `username`, `email`, and a `link` as arguments. What we want to do now is to make sure that when a user registers they get an email that will be queued and processed by the bullJs package.

> Bull is a Node library that implements a fast and robust queue system based on [redis] (https://redis.io/).

Although it is possible to implement queues directly using Redis commands, this library provides an API that takes care of all the low-level details and enriches Redis' basic functionality so that more complex use-cases can be handled easily. [Read more] (https://optimalbits.github.io/bull/)

# Bull Implementation
Navigate to the `server/config` folder and add a file called `bullConfig.js`.Paste in the following code.



```javascript
import Queue from 'bull';
import dotenv from 'dotenv';

dotenv.config();

const { REDIS_URL } = process.env;

// Initiating the Queue with a redis instance
const sendMailQueue = new Queue('sendMail', REDIS_URL);

export default Sendmail queue;
```
Make sure your `.env` file contains `REDIS_URL='redis://127.0.0.1:6379'`. In this file, we're importing `Queue` from our package `bull`. with that, we create a new instance of the bull queue class and called it `sendMailQueue`. We could create multiple queues for a different use case like

```javascript
const imageQueue = new Queue('image transcoding');
```
> Make sure you have a Redis server running in the background

Next, we need to create a `notifications.js` file in the `server/helper` folder. In this file, we'll introduce two key concepts from `bullJs` `producer` and `consumer`. we'll also create a function that takes in a receiver's email address, name, and a link to verify their email address. we could extend this implementation by adding functions like a `forgotPassword` function that takes the same parameters as the `registrationEmail` function. for each new notification we send out, we need to trigger the `producer` by calling `sendMailQueue.add(data, options);`.

```javascript
import emailService from '../config/emailSetup';
import sendMailQueue from '../config/bullConfig';
import template from './template';

/**
 *
 * @param {*} emailTo
 * @param {*} link
 * @param {*} name
 * @returns {*} sends an email to a new user
 */
const registrationEmail = (emailTo, link, name) => {
  const subject = 'Welcome to Karneek';
  const body = `<p>Dear ${name},</p>
  <p>We are thrilled to have you.</p>
  <p>Some random message with link</p>
      <a href="${link}" class="button">Confirm email</a>`;
  const message = template(subject, body, emailTo);

  const options = {
    attempts: 2,
  };
  const data = { emailTo, subject, message };

  // Producer: adds jobs to que, in this case emails to be sent out upon signup
  sendMailQueue.add(data, options);
};

// Consumer: this gets called each time the producer receives a new email.
sendMailQueue.process(async job => {
  emailService.mailSender(job.data);
});

const Notifications = { registrationEmail };

export default Notifications;
```
Finally, we need to call the `registrationEmail` function in our registration route. Navigate to `server/controllers/auth.js`. Import our notifications file

```javascript
import notifications from '../helpers/notifications';
```

In the `registerUser` function, just after the `verificatioToken` variable add

```
      const { REGISTRATION_URL } = process.env;
      const verificationLink = `${REGISTRATION_URL}?token=${verificationToken}`;
      await notifications.registrationEmail(email, verificationLink, username);
```

That's it. To test things our run the server `npm run start:dev` and create a new user! On your console, you should see

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/sr20ywl1ym9rewx9n9sl.png)

#Hosting this setup on Heroku

 The challenge with this setup comes when we need to host this project on a remote server like Heroku. To get this working, we need to have an instance of Redis running by the side on Heroku. First, we need to add an addon by Heroku called `Heroku Redis`, we also need to add `Heroku Postgres` because this project uses a Postgres DB.

Navigate to your dashboard on Heroku, and create a new app. After that navigate to the overview tab of the newly created Heroku app and click on `Configure Add-ons`

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ezyu97ujfzj56dp59wz0.png)

This would navigate to a new page with a search box. Search for and add `Heroku Redis`, and `Heroku Postgres`

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/x5ovfkkjw542dodjg7gu.png)

Copy the name of your Heroku app and on your terminal run the following commands.

```bash
heroku git:remote -a <name of heroku app>
heroku addons | grep heroku-redis
heroku addons:create heroku-redis:hobby-dev -a <name of heroku app>
```
After the last command, you should see something close to this

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/jdy9sasoz7caw79hs8xz.png)

You need to copy your new Redis addon name at this point. for me, it was `redis-deep-25660` (see screenshot)

run the next commands

```bash
heroku addons:info <your addon name> 
heroku config | grep REDIS
git push heroku master 
```

If you're seeing anything like this on your logs, then you're done! 😳
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/opdiqdzzrqlr1w4vzrif.png)

> To learn more about how `Heroku Redis` works ... https://devcenter.heroku.com/articles/heroku-redis#provisioning-the-add-on

# Conclusion

In this article, we've been able to use bullJs for email queueing and also host our tiny app on Heroku by provisioning and running a Redis server on our Heroku app. I hope you found this useful, If you run into any problems you can check out the complete code [here] (https://github.com/vic3king/bulljs-email-setup/tree/ft-implement-bull) or leave a comment.