Create Nginx extensions in JavaScript
Tags: nginx, javascript, webdev, programming
Published: 2022-07-23 06:29:37

If you need to adapt content, authorize, filter, change the behavior of Nginx, so `njs` can be the solution. Nginx proposes a JavaScript backend to manipulate requests and responses and even streams. This is [`njs`](https://nginx.org/en/docs/njs/) and it is very powerful.

[Nginx](https://nginx.org/) is a very good HTTP server and reverse proxy. It is simple to configure, easy to start, and it "does the job". We often use it as a proxy to backends. But, sometimes, you can feel some limitations. For example if you need to authorize users to several backends with a specific identity provider with a weird API. Or, sometimes, this is the backend which is problematic.

For example, I serve a Rancher backend, and Rancher provides  its `kubeconfig` YAML file (dynamically generated for each user) with a certificate inside that I need to remove.

In short, I need to "filter" the content, because I cannot change the behavior of the backend. 

> And this is exactly when `njs` can be used!

What I could do is simply to detect that the user claims the KubeConfig file, remove the certificate entry in the YAML, and serve the response file. This is one of the vast variety of manipulation that you can do with `njs`.

What it `njs`? 

`njs` is a JavaScript engine integrated to Nginx. `njs` provides a different approach that starts a JS VM on each needed process. It actually can use common JavaScript modules (like `fs` for example), and it's ES6 compliant. That means that you will code the scripts without the need of changing your habits.

## What you can do

There are plenty of things that you can do with `njs`:

- Make authorization (with or without backend)
- Manipulate the output content, headers, status...
- Interact with streams
- Use cryptography
- And so on...

You must keep in mind that `njs` is not intend to create an application. Nginx is not an application server, it's a web server and a reverse proxy. So, it will not substitute a "real" web framework. But, it will help to fix some things that are hard to do.


## Read this before!

Go to [the documentation page](https://nginx.org/en/docs/njs/reference.html) here to check the global variables which are already defined by Nginx/njs, you'll see that there are many methods to trace, crypt, decode, or manipulate the requests. 

Do not spend too much of time to read the page, but just take a look to be aware of the possibilities.

## It's not activated by default

`njs` is not activated by default. It's a module that you need to load at startup.

The legacy method to activate it is to add `load_module modules/ngx_http_js_module.so;` on top of `nginx.conf` file.

For docker, you can change the "command" to force the module to be loaded:

```
docker run --rm -it nginx:alpine \
nginx -g "daemon off; load_module modules/ngx_http_js_module.so;"
```

Then in your `http` services, you can now load a script and call functions for different situation.

## Prepare the tests

To follow this article, and be able to test, we will start a Nginx service and a "Ghost" blog engine.

This will help to startup:

```bash
mkdir -p ~/Projects/nginx-tests
cd ~/Projects/nginx-tests
mkdir -p nginx/conf.d
touch nginx/conf.d/default.conf

cat << EOF | tee docker-compose.yaml
version: "3"
services:
  # our nginx reverse proxy
  # with njs activated
  http:
    image: nginx:alpine
    command: nginx -g "daemon off; load_module modules/ngx_http_js_module.so;"
    volumes:
      - ./nginx/conf.d:/etc/nginx/conf.d:z
    ports:
      - 80:80

  # a backend
  ghost:
    image: ghost
    depends_on:
      - http
    environment:
      url: http://example.localhost
EOF
```

## Debug

Sometimes, your script fails to return somthing, you've got an error 500 and nothing is clearly displayd in the nginx logs.

What you can do is to add this in the "location":

```nginx
error_log /var/log/nginx/error.log debug;
```

For docker/podman users, use:

```bash
# see the logs in real time
docker-compose logs -f http
```

And you can use in your script :

```javascript
r.log("something here")
r.log(njs.dump(variable))
```

When you finally found the problems, you can then remove the "debug" suffix of the "`error_log`" directive.

## There are many ways, many options, many situations

Nginx manages 2 types of configuration: http and stream. By chance, `njs` can be used for both.

In this article, we will only speak about the "http" mode.

When we work with "http" mode, we can tell to Nginx to use JavaScript to manipulate:

- The content (that means that we will generate the entire request before the call)
- The headers only (after request)
- The body only (after request)
- Set variables
- ...

The pipeline is not hard: import a javascript file, then use a Nginx directive to use a function for a situation. Nothing more.


## First example, create a content

A first, let's create a file named `example.js`. To make it easy to test, put this in `nginx/conf.d/example.js` – in production environment, it's better to change the location of your scripts.

OK, so, this is the content:

```javascript
function information(r) {
    r.return(200, JSON.stringify({
        "hello": "world",
    }));
}

// make the function(s) available
export default {information}
```

The `r` variable (argument of the function) is a "request" object provided by Nginx. There are many methods and properties that we can use, here we only use `r.return()`.

It's now time to make the JavaScript function to be called as the content maker. In `nginx/conf.d/default.conf`, append this:

```nginx
js_import /etc/nginx/conf.d/example.js;

server {
    listen 80;
    server_name example.localhost;
    location / {
        js_content example.information;
    }
}
```

Yes, that's all. We import the script, and we use the function as `js_content`. That means that Nginx will release the request to the script, and the script can yield the content.

Start (or restart) the container and hit the "example.localhost" domain:

```bash
$ docker-compose up -d
$ curl example.localhost
{"hello": "world"}
```

That's the first example. Here, we only generate a JSON output.

We can now do some nice things, like replacing the content.

## Replacing the content

`njs` proposes several `fetch` APIs to get, sub-request or internally redirect the request.

In this example, we will replace the "/about/" page content by inserting a message inside.

The following configuration is not perfect (actually we could match `location /about` and call our JavaScript, but I want to show you the internal redirection) — but it will show you some cool stuffs that `njs` allows.

Change the `nginx/conf.d/default.conf` file with this:

```nginx
js_import /etc/nginx/conf.d/example.js;

upstream ghost_backend {
    server ghost:2368;
}

server {
    listen 80;
    server_name example.localhost;
    gunzip on;

    # call our js for /exemple url
    location = /exemple {
        js_content example.information;
    }

    # match everything
    location / {
        js_content example.replaceAboutPage;
    }

    # call the "ghost blog" backend
    # note the "arobase" that creates a named location
    location @ghost_backend {
        # important !
        subrequest_output_buffer_size 16k;

        proxy_pass http://ghost_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering off;
    }
}
```

In this file, you have to pay attention on this:

> We force "gunzip" to uncompress proxied responses. This is important if the backend forces `gzip` responses, and you want to make replacements.
> We also make the "subrequest output buffer" size to 16k, you can set it to 128k if needed (for large CSS files if you subrequest them for example).

> This is important because we will make subrequests later, and the buffer will be filled too fast

Then, in the `example.js` file, add the `replaceAboutPage` function and export it :

```javascript
function information(r) {
  r.return(
    200,
    JSON.stringify({
      hello: "world",
    })
  );
}

async function replaceAboutPage(r) {
  if (r.uri == "/about/") {
    r.headersOut["content-type"] = "text/html";
    r.return(200, "Changed");
  } else {
    r.internalRedirect("@ghost_backend");
  }
}

export default { information, replaceAboutPage };
```


Take a minute to read the `replaceAboutPage` function. In this example, we only:

- return a page with "Changed" inside if the URI is "/about"
- either we use `@ghost_backend` location to proxy the blog

Restart the nginx container:
```bash
docker-compose restart http
```

And visit http://example.locahost. Then go to the "/about" page using the "About" link on top.

![We changed the about page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t9sdzflmlqpqhyyz5n51.png)

Nice, so now, we can do a better replacement.

We will need to "subrequest" the page. But a subrequest needs a "real URI path". So, let's add a location in `default.conf` first.


```nginx
# a reversed uri.
# We remove the prefix and use the @ghost_backend
location /__reversed {
    internal;
    rewrite ^/__reversed/(.*)$ /$1 break;
    try_files $uri @ghost_backend;
}
```

Then, let's go to `example.js` and replace the `replaceAboutPage` function to this one:

```javascript
//...
async function replaceAboutPage(r) {
  if (r.uri == "/about/") {
    r.subrequest(`/__reversed${r.uri}`) // call the reversed url
      .then((res) => {
        // copy the response headersOut
        Object.keys(res.headersOut).forEach((key) => {
          r.headersOut[key] = res.headersOut[key];
        });
        // replace the end of "header" tag to append a title
        const responseBuffer = res.responseBuffer
          .toString()
          .replace("</header>", "</header><h1>Reversed</h1>");
        r.return(res.status, responseBuffer);
      })
      .catch((err) => {
        r.log(err);
        r.return(500, "Internal Server Error");
      });
  } else {
    // in any other case
    r.internalRedirect("@ghost_backend");
  }
}
//...
```

One more time, restart `http` container:
```bash
docker-compose restart http
```

Then visit http://example.locahost/about/ – you should see:

![The modified about page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j13dj8r77ejwnevhre9s.png)


## Second method, use `js_body_filter`
A probably better solution is to use `js_body_filter` instead of `js_content`. This leaves Nginx makes the proxy pass, then we can manipulate the body.

So, let's change the `default.conf` file to this:
```nginx
js_import /etc/nginx/conf.d/example.js;

upstream ghost_backend {
    server ghost:2368;
}

server {
    listen 80;
    server_name example.localhost;
    gunzip on;

    # call our js for /exemple url
    location = /exemple {
        js_content example.information;
    }

    # call the "ghost blog" backend
    location / {
        js_body_filter example.replaceAboutPage;
        subrequest_output_buffer_size 16k;
        proxy_pass http://ghost_backend;
        proxy_set_header Host $host;
        proxy_set_header Accept-Encoding "";
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering off;
    }
}
```

Very important, here we force the `Accept-Encoding` to be empty, because Ghost will return a gzipped content that is impossible (at this time) to decompress from javascript.

Then change the JavaScript `replaceAboutPage` function to this:
```javascript
async function replaceAboutPage(r, data, flags) {
  if (r.uri == "/about/") {
    r.sendBuffer(data.toString().replace("</header>", "</header>Reversed"), flags);
  } else {
    r.sendBuffer(data, flags);
  }
}
```

- `r` is the request
- `data` is the data to send, here it's the content taken from the Ghost backend
- `flags` is an object with "last" flags set to `true` or `false`

The function needs to use `sendBuffer()` to send the data to the client.

Of course there are many others things to do, like changing the "Content-Length" header, but this works.

It's very efficient and that's a good method to make some replacement, content checks or fixes to a response without the need to make a `subrequest`.

## Make a `fetch` to outside

`njs` provides a global `ngx` variable. This object proposes the `fetch` API which is more or less the same as you can find in modern browsers.

Let's add a method to call the Dev.to API.

> Please, do not abuse the API, it's a simple example, and you are pleased to not overload the servers

The following function will get the list of my articles.

```javascript
async function getMetal3D(r) {
  ngx
    .fetch("https://dev.to/api/articles?username=metal3d", {
      headers: {
        "User-Agent": "Nginx",
      },
    })
    .then((res) => res.json())
    .then((response) => {
      const titles = response.map((article) => article.title);
      r.headersOut["Content-Type"] = "application/json";
      r.return(200, JSON.stringify(titles));
    })
    .catch((error) => {
      r.log(error);
    });
}

// don't forget to export functions
export default { information, replaceAboutPage, getMetal3D };
```

OK, now you think that you'll only need to add a `js_content` call inside the `default.conf` file. But... there will be some problems.

- `ngx` object need a "resolver" to know how to resolve the domain name
- also, you need to define where it can find certificate authorities

But, that's not so hard to do:

```nginx
location /metal3d {
    resolver 9.9.9.9;
    js_fetch_trusted_certificate /etc/ssl/certs/ca-certificates.crt;
    js_content example.getMetal3D;
}
```

So, one more time, restart the `http` container and call the `/metal3d` URI:

```bash
docker-compose restart http

curl http://example.localhost/metal3d

# response:
[
    "Python, the usefulness of \"dataclass\"",
    "Fixing a Promise return object to be typed in Typescript",
    "Flask / Quart - manage module loading and splitting",
    "Change local container http(s) reverse proxy to Pathwae"
]
```


## Conclusion

Nginx proposes a very useful JavaScript extension system. It's easy to realize the large possibilities that can be made with it.

It's possible to create a custom cache system, to manipulate headers, reading token, validate authorizations, change the content, create a proxy on API, and many others things.

> We only checked how to manipulate content, but it's also possible to manipulate "streams" with events.


Go to the documentation pages:

- NJS doc: https://nginx.org/en/docs/njs/
- JS object that you can use: https://nginx.org/en/docs/njs/reference.html
- Module directive:
  - http https://nginx.org/en/docs/http/ngx_http_js_module.html
  - stream https://nginx.org/en/docs/stream/ngx_stream_js_module.html
- Examples on GitHub: https://github.com/nginx/njs-examples/