Building a Simple Bot Protection With NGINX JavaScript Module (NJS) and TypeScript
Tags: nginx, njs, javascript, typescript
Published: 2021-05-25 08:00:16

---
title: Building a Simple Bot Protection With NGINX JavaScript Module (NJS) and TypeScript
published: true
date: 2021-05-24 19:57:34 UTC
tags: nginx,njs,javascript,typescript
canonical_url: https://fsjohnny.medium.com/building-a-simple-bot-protection-with-nginx-javascript-module-njs-and-typescript-386b2207ba90
cover_image: https://cdn-images-1.medium.com/max/1024/1*l3YxGXaHfJRZLunoXU_hDg.jpeg
---

#### Cover Photo by [Phillip Glickman](href="https://unsplash.com/@phillipglickman?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText) on [Unsplash](https://unsplash.com/s/photos/robots?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText)

I love Lua. I also love NGINX. The three of us get along just great. Like every relationship, we’ve had our highs and lows (yes, I’m looking at you Lua patterns), but overall life was perfect. Then, NGINX JavaScript Module (NJS for short) came along.

![](https://cdn-images-1.medium.com/max/750/1*UJZhiTDBGO2mOFx01iUL8Q.jpeg)<figcaption>I ❤️ JS/TS</figcaption>

NGINX JavaScript module was first introduced in 2015 but recently received a big boost in functionality with the 0.5.x update. Since I'm a sucker for anything JS, I decided to test it out by building a simple (read naive and **not production ready** ) bot protection module 🤖.

### Configuring NGINX

Before diving into bot fight, we have to set up NGINX to support the JavaScript module. The instructions below are for my setup (Ubuntu 20.4/Nginx 1.18), so YMMV, but the general idea should be the same for most setups.

1. Start by adding the NGINX PPA key by running:  
curl -s [https://nginx.org/keys/nginx\_signing.key](https://nginx.org/keys/nginx_signing.key) | sudo apt-key add -

2. Setup the repository key by running:

```bash
sudo sh -c 'echo "deb http://nginx.org/packages/ubuntu/ focal nginx" >> /etc/apt/sources.list.d/nginx.list'
```

3. Update the repository list by runningsudo apt update.

4. Install NJS by running sudo apt install nginx-module-njs.

If all went well, at this point, you should get this lovely message on your terminal:

![](https://cdn-images-1.medium.com/max/1024/1*TR-aBRr-V8qILaSHvmVqeQ.png)<figcaption>Big success 🥂</figcaption>

5. Enable NJS by adding the following to the top of your main **nginx.conf** file:

```
load_module modules/ngx_http_js_module.so;
```

6. Restart NGINX to load NJS into the running instance:

```bash
sudo nginx -s reload
```

Now your NGINX is ready for some JS love, so let’s move on and create our first line of defense — IP filtering!

![](https://cdn-images-1.medium.com/max/888/1*G71NQ2tKLzd-3BmYLgNJ3A.jpeg)<figcaption>damn right you are!</figcaption>

### Opening Act — Creating the Project

Our bot protection project is going to be written in TypeScript. For that, we need to create a project that will transpile TypeScript to ES5 JavaScript, which NJS can understand. As you may have guessed, NodeJS is a must here, so make sure you are all set up before continuing.

1. Create the new project folder and initialize it:

```bash
mkdir njs-bot-protection && cd njs-bot-protection
npm init -y
```

2. Install the required packages:

```bash
npm i -D @rollup/plugin-typescript @types/node njs-types rollup typescript
```

3. Add the **build** script to the **package.json** ’s **scripts**  section:

```json
{
    ...
    "scripts": {
        "build": "rollup -c"
    },
    ...
}
```

4. To compile the project, you’ll need to tell the TypeScript compiler how to do that with the **tsconfig.json** file. Create a new **tsconfig.json** file in the root of the project, and add the following content to it:

{% gist https://gist.github.com/pxjohnny/6619770d740872e7ee7ad248e8ed68e1 %}

5. Lastly, let’s add the rollup config, which will wrap everything up and produce the endgame js file that NJS will read.   
Create a new **rollup.config.js** file in the root of the project, and add the following content to it:

{% gist https://gist.github.com/pxjohnny/362541a564632a7cdc86e2b075d3a8d8 %}

And with that, our boilerplate is all loaded and ready to go. That means it’s time to kick some bots!

### Round 1 — IP Filtering

Our first line of bot defense is IP blocking; we compare the IP of an incoming request with a list of known IPs with bad reputations, and if we find a match, we redirect the request to a “block” page.

We’ll begin with creating the JavaScript module:

1. In the project root folder, create a new folder called **src,** and then inside of it create a new **bot.ts**  file.
2. Add the following code snippet to  **bot.ts** :

{% gist https://gist.github.com/pxjohnny/7dc86d7d29cef74ae767b88a4ab161fe %}

💡 So what do we have here?

- **Line 1** : Imports the built-in module for the file system (i.e., fs). This module deals with the file system, allowing us to read and write files, among other activities.
- **Line 2** : Calls the loadFile function, passing it the name of the file we wish to load.
- **Lines 4–12** : The implementation of loadFile. First, we initialize the data variable to an empty string array (line 5), then we try to read and parse a text file containing a list of bad IP addresses into the data object (line 7), and finally we return the data object (line 11).
- **Lines 14–21** : The implementation of verifyIP — the heart of our module (for now). This is the function we will expose to NGINX to verify the IP. We first check if the array of bad reputation IPs contains the current request client IP (line 15). If yes, redirect the request to the block page and end processing (lines 16 and 17). If not , redirect internally to the pages location (line 20).
- **Line 23** : Exports (read exposes) verifyIPexternally.

3. Build the module by running npm run build in your terminal. If all goes well, you should find the compiled **bot.js** file in the **dist** folder 🎉

With the file in hand, let’s configure NGINX to be able to use it:

1. In your NGINX folder ( **/etc/nginx** in my case) create a folder named **njs** and copy **bot.js** from the previous section inside it.
2. Create a new folder called **njs** under **/var/lib** , create a file called **ips.txt** inside it, and populate it with a list of bad reputation IPs (one IP per line). You can either add your own list of IPs or use something like [https://github.com/stamparm/ipsum](https://github.com/stamparm/ipsum).
3. In your **nginx.conf** , under the http section, add the following:

```bash
js_path "/etc/nginx/njs/";
js_import bot.js;
```

💡 So what do we have here?

- **js\_path**  — Sets the path for the NJS modules folder.
- **js\_import**  — Imports a module from the NJS modules folder. If not specified, the imported module namespace will be determined by the file name (in our case, bot)

4. Under the server section (mine is on **/etc/nginx/conf.d/default.conf** ) modify the / location as follows:

```bash
location / {
    js_content bot.verifyIP;
}
```

By calling verifyIP using the js\_content directive we set it as the content handler, which means verifyIP can control the content we send back to the caller (in our case, either show a block page or pass the request to the origin)

5. Still under the server section, add the block.html location and the pages named location:

```bash
location [@pages](http://twitter.com/pages) {
    root /usr/share/nginx/html;
    proxy_pass [http://localhost:8080](http://localhost:8080);
}

location /block.html {
    root /usr/share/nginx/html;
}
```

(The namedpages location will be used by our NJS module to internally redirect the request if it shouldn't be blocked. You likely have your own logic for this redirection so change this to fit your needs)

6. At the bottom of the file, add the server block for port 8080:

```bash
server {
        listen 8080;
        location / {
        root /usr/share/nginx/html;
        index index.html index.htm;
    }
}
```

7. Under the **/usr/share/nginx/html** folder, add the **block.html** file as follows:

{% gist https://gist.github.com/pxjohnny/f8e96cc825906cd59c1280ba69287925 %}

And with that, our IP protection is ready! Add your own IP to the **ips.txt** file and restart NGINX (sudo nginx -s reload). Browse to your instance and you should be greeted with the following:

![](https://cdn-images-1.medium.com/max/1024/1*ly0aRwgPudTC-TcdLZr7mg.png)<figcaption>Take that Mr. Evil Bot! 🤖 ⛔</figcaption>

### Round 2 — JavaScript Detection

Our second protection layer is JavaScript detection. We use this detection to determine if the visitor coming to our site is running JavaScript (which every normal browser should do) or not (a warning sign that this visitor might not be a legitimate user). We begin with injecting a JavaScript snippet to the pages that will bake a cookie on the root path:

1. Add the following code snippets to  **bot.ts** :

{% gist https://gist.github.com/pxjohnny/1eeb2a6642a11112b413eb800de553d7 %}

💡 So what do we have here?

- **Line 1** : Imports the built-in Crypto module. This module deals with cryptography, and we will soon use it for creating an HMAC.
- **Lines 5–18** : The implementation of getCookiePayload. The function sets a date object to one hour ahead of the current time (lines 6–8), then uses the date object to HMAC (using the crypto module) the signature we passed to the function (the value object) with the date object (lines 10–14). Lastly, the function returns the cookie information in a string format (name, value, expiration, etc.). You may notice the cookie value contains not only the hashed signature but also the date object we used to HMAC the signature with. You’ll see why we do that soon.
- **Lines 20–30** : The implementation of addSnippet. The function buffers the request data, and once it finishes (line 23) it:  
- Creates a signature based on the client IP and the User-Agent header (line 24).  
- Replaces the closing head tag with a script section that inserts a cookie (from the getCookiePayload function) on the browser side using JavaScript’s document.cookie property. (lines 25–28).   
- Sends the modified response back to the client (line 29).

2. Export the new addSnippet function by updating the export statement at the bottom of the file:

```javascript
export default { verifyIP, addSnippet };
```

3. Under the @pages location block, modify the / location as follows:

```bash
location [@pages](http://twitter.com/pages) {
    js_body_filter bot.addSnippet;
    proxy_pass [http://localhost:8080](http://localhost:8080);
}
```

Unlike verifyIP, we don't want addSnippet to manage the content of the response, we want it to inject content (a script tag in our case) to whatever response comes back from the origin. This is where js\_body\_filter comes into play. Using the js\_body\_filter directive we tell NJS that the function we provide will modify the original response from the origin and return it once finished.

4. Restart NGINX and browse to a page on your instance. You should see our new script added just before the closing head tag:

![](https://cdn-images-1.medium.com/max/1024/1*CB7hOzUOS7r5rsWU-HfwlQ.png)

If the client is running JavaScript, a new cookie called **njs** will be baked. Next, let’s create the validation for this cookie/lack of cookie:

1. Add the verifyCookie function (and its supportive functions/variables), to  **bot.ts** :

{% gist https://gist.github.com/pxjohnny/774401df722d488ea4c82fb4164f1d1f %}

💡 So what do we have here?

- **Lines 5–11** : The implementation of the updateFile function, which uses the fs module to save an array of strings to a file.
- **Lines**  **13–52** : The motherload implementation. When validating the **njs** cookie, we have a flow of verification and consequences we have to follow:

a. We begin with extracting the **njs** cookie from the request’s **Cookie** header (lines 14–20).

b. If we don’t have a cookie (or we do and it’s malformed), we compare the client IP against our list of client IPs that have reached us without a cookie. If we find a match from within the last hour, we fail the request (returning false, lines 26–27). If we don’t, we delete the IP (if it's on the list but past one hour) and pass the request (lines 29–34).

c. If we do have a cookie, we split it into a timestamp and a payload and use the timestamp to create our own HMAC hash based on the request’s User-Agent header and client IP. If our own HMAC matches the HMAC of the **njs** cookie, we pass the request. Otherwise, we fail it (lines 38–45).

d. If anything goes wrong during the validation, we fail open (meaning pass) the request (lines 48–51).

2. Add the new verify function, which calls the new verifyCookie function, and act according to its result:

{% gist https://gist.github.com/pxjohnny/1e39b40ea237780e96eeb75b5ab283ae %}

🔥 At the point you might be thinking to yourself at this point that this verify function looks eerily similar to the verifyIP function from the earlier — you are absolutely right, and I will touch on that in a minute!

3. To test our new cookie validation functionality, open up your configuration file (mine is at **/etc/nginx/conf.d/default.conf** ) and change the js\_content directive from verifyIP to verify:

```bash
location / {
    js_content bot.verify;
}
```

4. Restart NGINX and try to visit the site twice without the njs cookie — ✋ 🎤- you are blocked!

![](https://cdn-images-1.medium.com/max/1024/1*ly0aRwgPudTC-TcdLZr7mg.png)<figcaption>Not today Mr. Evil Bot! 🤖 ⛔</figcaption>

### Final Round — Bringing It All Together

So now we have the cookie verification, but we took off our IP verification because we can only have one js\_content directive, how do we go around fixing that?

You may remember that a few minutes ago we created the verify function (which eagle-eyed readers may have noticed is VERY similar to the verifyIP function we used before). If we update our verifyIP function so that it returns a boolean response as verification, and add that verification to verify, we get the best of both worlds with one big function that verifies requests for both IPs and cookies!

1. Refactor the verifyIP function as follows:

{% gist https://gist.github.com/pxjohnny/5a210b2a71a5774a9acdbc2f029eb052 %}

2. Update the verify function to call verifyIP as follows:

{% gist https://gist.github.com/pxjohnny/7a3cd343273874e8125ebb96cac0c8b5 %}

3. Update the export statement, as we no longer need to expose verifyIP:

```javascript
export default { addSnippet, verify };
```

4. Restart NGINX and enjoy your home-made bot protection using NJS and TypeScript 🎉

![](https://cdn-images-1.medium.com/max/556/1*ocJSogOMSWAcaTBdaLD4Rw.jpeg)<figcaption>Ewwwww, so cute ❤️</figcaption>

🍾 The module source code is available on [GitHub](https://github.com/jtordgeman/njs-bot-protection)!