Simple Node.js task queue with bee-queue and redis
Tags: node, javascript, webdev, tutorial
Published: 2020-06-26 16:27:53

<sup>Cover Photo by <a href="https://unsplash.com/@bimoluki02?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Bimo Luki</a> on <a href="/s/photos/waiter?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></sup>

As you saw in the [previous article](https://dev.to/sarbikbetal/task-queues-and-why-do-we-need-them-26mj) that task queues are pretty awesome 🌟 and in this tutorial we would be using a task queue in our own application let's get our hands dirty and write some code.

We will be building our restaurant as explained in the previous article.

This tutorial would be much of a demonstration rather than a working application, so stick with me if you wanna check out how to plug in a task queue into your app.
In the next article we will be building a real application. (I know it's exciting and you can't wait for that 😉).

**👨‍💻  Link to the entire github repo for this project is at the end of the article ⬇**

## Let's get started.
### Pre-requisites
+ Install [Node.js](https://nodejs.org/en/) on your machine, and then run the following command to verify proper installation.
```bash
$ node --version
> v12.16.1
```
+ Redis running on your pc or the cloud. Install [Redis](https://redis.io/) or create an instance on [RedisLabs](https://redislabs.com/) for free.

And we're good to go :grin:

## Initialization
Run:
```bash
$ npm init
```
After that install the necessary packages by running 
```bash
$ npm install express bee-queue dotenv
```
In case you are wondering what each package does, here's some info:
+ **`express`** helps us to create a server and handle incoming requests with ease.
+ **`bee-queue`** is our task queue manager and will help to create and run jobs
+ **`dotenv`** helps us to load environment variables from a local `.env` file

After that create a file `restaurant.js` and edit your `package.json` so it looks something like this
```json
{
  ...
  "main": "restaurant.js",
  "scripts": {
    "start": "node restaurant.js"
  }
  ...
}
```
## Time for some real code
Open `restaurant.js` in the editor of your choice and add the following lines of code 
```js
require('dotenv').config();
const express = require('express');
const http = require('http');

// Inits
const app = express();
app.use(express.json());
app.use(express.urlencoded({ extended: false }));

// Routes
app.get('/', (req, res) => {
    res.send("😋 We are serving freshly cooked food 🍲");
});


// Create and start the server
const server = http.createServer(app);
const PORT = process.env.PORT || 5000;
server.listen(PORT, () => {
    console.log(`Restaurant open at:${PORT}`);
});
```
What it does is basically start a local webserver at the specified port (here, 5000) and listens for incoming `GET` requests on the base url `/` and replies with a text.

Run the following command to start the server and head over to `localhost:5000` in your browser.
```bash
$ npm start
> restaurant@1.0.0 start /mnt/code/dev/queue
> node restaurant.js

Restaurant open at port:5000
```
You will get a blank page with a neat little `😋 We are serving freshly cooked food 🍲` message
<!-- ![simple express server](https://dev-to-uploads.s3.amazonaws.com/i/o3rszn0ihvhppmt772e2.png) -->

---

### **Now it's time to create our task queue**

First create a file named `.env` and paste in it your database credentials like so, (You can use your local redis instance here as well) and remember, never to commit `.env` to your source control.
```sh
DB_HOST=redis-random-cloud.redislabs.com
DB_PORT=14827
DB_PASS=pTAl.not-my-password.rUlJq
```
And you are done with the basic configuration.

Let's go ahead and create our `waiter`. Start by creating a file `waiter.js` and add the following chunk of code:
```js

const Queue = require('bee-queue');

const options = {
    removeOnSuccess: true,
    redis: {
        host: process.env.DB_HOST,
        port: process.env.DB_PORT,
        password: process.env.DB_PASS,
    },
}

const cookQueue = new Queue('cook', options);
const serveQueue = new Queue('serve', options);


const placeOrder = (order) => {
    return cookQueue.createJob(order).save();
};

serveQueue.process((job, done) => {
    console.log(`🧾 ${job.data.qty}x ${job.data.dish} ready to be served 😋`);
    // Notify the client via push notification, web socket or email etc.
    done();
})
    // Notify the client via push notification, web socket or email etc.
    done();
})


module.exports.placeOrder = placeOrder;
```
**🤯 Whoa! what was that?** Well, let me explain.

We first import the `bee-queue` package as `Queue`,
and then pass in the database configuration to our two new `Queue` objects. One of the queue will have the list of orders to be prepared by the cook and the other will have the list of orders that are ready to be served by the waiter.

We then create a new function `placeOrder` that takes in an `order` as the parameter. We will define this order object later, but keep in mind that it has a structure like this 
```js
order = {
    dish: "Pizza 🍕", 
    qty: 2,
    orderNo: "kbv9euic"
}
```
The `placeOrder` function takes this order and adds it to the queue by calling `.createJob(order).save()` method on the `cookQueue` Queue object. This acts as the task **publisher**.

and lastly the `process` method on `serveQueue` Queue object executes the handler function `(job, done) => {...}` everytime an order is prepared and ready to be served. This acts as the task **consumer**.

We call `done()` to acknowledge out task queue that the job in done so that it can send the next task to be processed from the queue. We simply call `done()` to indicate the task was successful and call `done(err)` i.e with the first parameter (where `err` is an error message) to indicate job failure. You can also call `done(null, msg)` to indicate job success with the second parameter `msg` being the success message.

And our waiter 👨‍💼 is ready

---

Now its time for the kitchen with the cooks 👨‍🍳
create another file `kitchen.js` and paste in it the following lines of code: 
```js
const Queue = require('bee-queue');

const options = {
    removeOnSuccess: true,
    redis: {
        host: process.env.DB_HOST,
        port: process.env.DB_PORT,
        password: process.env.DB_PASS,
    },
}

const cookQueue = new Queue('cook', options);
const serveQueue = new Queue('serve', options);

cookQueue.process(3, (job, done) => {
    setTimeout(() => console.log("Getting the ingredients ready 🥬 🧄 🧅 🍄"), 1000);
    setTimeout(() => console.log(`🍳 Preparing ${job.data.dish}`), 1500);
    setTimeout(() => {
        console.log(`🧾 Order ${job.data.orderNo}: ${job.data.dish} ready`);
        done();
    }, job.data.qty * 5000);
});

cookQueue.on('succeeded', (job, result) => {
    serveQueue.createJob(job.data).save();
});
```
**😌 Well that looks familiar.**

Yeah exactly, but the only change is that here our cooks are **consuming** from the `cookQueue` and **publishing** to the `serveQueue` for the waiters to take and serve the the orders.

One thing to note here is that, anything published via `createJob(order)` is available to the cosumer as `job.data` in the `Queue.process()` method's handler function `(job, done) => {...}`, and if you look closely, there's something different in `cookQueue.process(3, (job, done) => {...})` too. Yeah, we pass in a number, before the actual handler function. It is known as concurrency (the number of tasks in the queue that can be processed simultaneously). Here we have set it to 3 because our kitchen has 3 cooks, who can work together.

And we use the `cookQueue.on('succeeded', (job, result) => {...})` method to call the handler function whenever a task is successful (i.e. whenever you have called `done()` in the `process()` method).

**Believe me we are almost done 🤞**

---

### **Final step: Hook everything together**

Open `restaurant.js` and add these final lines of code

```js
// ...
// Add these lines before the Inits.
require('./kitchen');
const { placeOrder } = require('./waiter');

// Inits
// ...
// Routes

// ...

app.post('/order', (req, res) => {
    let order = {
        dish: req.body.dish,
        qty: req.body.qty,
        orderNo: Date.now().toString(36)
    }

    if (order.dish && order.qty) {
        placeOrder(order)
            .then(() => res.json({ done: true, message: "Your order will be ready in a while" }))
            .catch(() => res.json({ done: false, message: "Your order could not be placed" }));
    } else {
        res.status(422);
    }
})

// Create and start the server
// ...
```

What we've done here is imported our `kitchen` and `waiter` and added a POST route `/order` to receive orders from our customers. Remember the order object?
```js
order = {
    dish: "Pizza 🍕", 
    qty: 2,
    orderNo: "kbv9euic"
}
```

We are creating a order object from the JSON body of the POST request and passing it to our waiter and sending a JSON response to acknowledge our customer. In case the request is not properly made, we will send some error message also. And we're done ✌ .

---

**Yeah really, we are done. Now it's time to test it out 😁**

+ Start the server by running `$ npm start` on your terminal.
+ Send a get request to `localhost:5000` and see if you get a response like this:
![Restaurant open](https://dev-to-uploads.s3.amazonaws.com/i/oiywtpv07pxi583qvpe3.png)
+ Next send a POST request to `localhost:5000/order` and check the response and look at your console.
![Testing out the API](https://dev-to-uploads.s3.amazonaws.com/i/xr5zxkjby02e1ol2prjl.gif)

You can send multiple requests one after another to check that it does not hang to any request.

**Let's add another `POST` route, to compare it with a normal restaurant without a task queue.**

Add these lines to `restaurant.js`:
```js
//  ...
app.post('/order-legacy', (req, res) => {
    let order = {
        dish: req.body.dish,
        qty: req.body.qty,
        orderNo: Date.now().toString(36)
    }
    if (order.dish && order.qty) {
        setTimeout(() => console.log("Getting the ingredients ready... 🥬 🧄 🧅 🍄"), 1000);
        setTimeout(() => console.log(`🍳 Preparing ${order.dish}`), 1500);
        setTimeout(() => {
            console.log(`🧾 Order ${order.orderNo}: ${order.dish} ready`);
            res.json({ done: true, message: `Your ${order.qty}x ${order.dish} is ready` })
        }, order.qty * 5000);
    } else {
        console.log("Incomplete order rejected");
        res.status(422).json({ done: false, message: "Your order could not be placed" });
    }
});


// Create and start the server
// ...
```
+ Next send a POST request to `localhost:5000/order-legacy` and check the response and look at your console.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/y6qcmwm1vx3w653qxc17.gif)

**Note the difference in the response time 🤯**

> **With the task queue**
![With task queue](https://dev-to-uploads.s3.amazonaws.com/i/rorr36dsu0t64uvhpo03.png) 

> **Without task queue**
![without task queue](https://dev-to-uploads.s3.amazonaws.com/i/cgn4u13gq8a1sgzryxph.png)


---

#### **Here is the Github repo, containing the complete project**
{% github sarbikbetal/nodejs-task-queue no-readme %}

Please comment down below if you have any questions or suggestions and feel free to reach me out :smile: and also check out the section below for Q&A.

| :camera_flash:[Instagram](https://www.instagram.com/sarbikbetal) | :incoming_envelope:[Email](mailto:sarbikb55@gmail.com) | :man_office_worker:[LinkedIn](https://www.linkedin.com/in/sarbikbetal) | :man_technologist:[Github](https://github.com/sarbikbetal) |
| --- | --- | --- | --- |

---

## 🤔  Hmmm.. I have some questions though.
I know, so here are some common ones, feel free to ask more in the comments section below.

+ ### How do we send the food to our customer once it is prepared?
For that we need to implement some additional logic to our server side and client side application. Example of how we can achieve that, is through Websockets, push-notifications, emails, etc. Don't worry though I'll be covering that in details in the next article.

+ ### Aren't there better things out there like RabbitMQ?
Yeah sure there is, but for small scale projects that doesn't need a lot of advanced features but still want to maintain a decent back-end infrastructure RabbitMQ would be an overkill and bee-queue might just turn out simple and easy to use.