How to Rate Limit Your API
Tags: javascript, node, webdev, tutorial
Published: 2020-01-20 00:58:45

**Subscribe to my email list now at http://jauyeung.net/subscribe/**

**Follow me on Twitter at https://twitter.com/AuMayeung**

**Many more articles at https://medium.com/@hohanga**

To prevent overloading your servers, adding a rate limit to your API is a good option to solve this problem. We can block excessive requests from being accepted by blocking it with route middleware to prevent route code from executing if too many requests are sent from one IP address.

Express apps can use the express-rate-limit package to limit the number of requests being accepted by the back end app. It is very simple to use. We just have to specify the rate limit per some amount of time for request to be accepted.

For example, to limit a request to accepting 5 requests per minute from one IP address, we put:

```javascript
const rateLimit = require("express-rate-limit");
 
 
const limiter = rateLimit({
  windowMs: 60 * 1000, 
  max: 5
});
 
app.use("/api/", limiter, (req, res) => {...});
```

The `limiter` is a middleware that is added before the route callback and it is executed before the route callback is called if the rate limit is not reach.

In this article, we will build a video converter to convert the source video into the format of the user’s choice. We will use the fluent-ffmpeg package for running the conversions. Because the jobs are long running, we will also create a job queue so that it will run in the background. The rate limit per minute can be set by us in an environment variable.

FFMPEG is a command line video and audio processing program that has a lot of capabilities. It also supports lots of formats for video conversion.

Developers have done the hard work for us by creating a Node.js wrapper for FFMPEG. The package is called fluent-ffmpeg. gIt is located at [https://github.com/fluent-ffmpeg/node-fluent-ffmpeg](https://github.com/fluent-ffmpeg/node-fluent-ffmpeg). This package allows us to run FFMPEG commands by calling the built in functions.

We will use Express for the back end and React for the front end.

Back End
========

To get started, create a project folder and a `backend` folder inside it. In the `backend` folder, run `npx express-generator` to generate the files for the Express framework.

Next run `npm i` in the `backend` folder to download the packages in `package.json`.

Then we have to install our own packages. We need Babel to use `import` in our app. Also, we will use the Bull package for background jobs, CORS package for cross domain requests with front end, fluent-ffmpeg for converting videos, Multer for file upload, Dotenv for managing environment variables, express-rate-limit for limiting requests to our app, Sequelize for ORM and SQLite3 for or database.

Run `npm i @babel/cli @babel/core @babel/node @babel/preset-env bull cors dotenv fluent-ffmpeg multer sequelize sqlite3 express-rate-limit` to install all the packages.

Next add `.babelrc` file to the `backend` folder and add:

```json
{
    "presets": [
        "@babel/preset-env"
    ]
}
```

to enable the latest JavaScript features, and in the `scripts` section of `package.json` , replace the existing code with:

```
"start": "nodemon --exec npm run babel-node --  ./bin/www",  
"babel-node": "babel-node"
```

to run with Babel instead of the regular Node runtime.

Next we create the Sequelize code by running `npx sequelize-cli init`.

Then in `config.json` that’s just created by running the command above, we replace the existing code with:

```json
{  
  "development": {  
    "dialect": "sqlite",  
    "storage": "development.db"  
  },  
  "test": {  
    "dialect": "sqlite",  
    "storage": "test.db"  
  },  
  "production": {  
    "dialect": "sqlite",  
    "storage": "production.db"  
  }  
}
```

Next we need to create our model and migration. We run:

```bash
npx sequelize-cli --name VideoConversion --attributes filePath:string,convertedFilePath:string,outputFormat:string,status:enum
```

To create the model and migration for the `VideoConversions` table.

In the newly created migration file, replace the existing code with:

```javascript
"use strict";
module.exports = {
  up: (queryInterface, Sequelize) => {
    return queryInterface.createTable("VideoConversions", {
      id: {
        allowNull: false,
        autoIncrement: true,
        primaryKey: true,
        type: Sequelize.INTEGER
      },
      filePath: {
        type: Sequelize.STRING
      },
      convertedFilePath: {
        type: Sequelize.STRING
      },
      outputFormat: {
        type: Sequelize.STRING
      },
      status: {
        type: Sequelize.ENUM,
        values: ["pending", "done", "cancelled"],
        defaultValue: "pending"
      },
      createdAt: {
        allowNull: false,
        type: Sequelize.DATE
      },
      updatedAt: {
        allowNull: false,
        type: Sequelize.DATE
      }
    });
  },
  down: (queryInterface, Sequelize) => {
    return queryInterface.dropTable("VideoConversions");
  }
};
```

to add the constants for our enum.

Then in `models/videoconversion.js` , replace the existing code with:

```javascript
"use strict";
module.exports = (sequelize, DataTypes) => {
  const VideoConversion = sequelize.define(
    "VideoConversion",
    {
      filePath: DataTypes.STRING,
      convertedFilePath: DataTypes.STRING,
      outputFormat: DataTypes.STRING,
      status: {
        type: DataTypes.ENUM("pending", "done", "cancelled"),
        defaultValue: "pending"
      }
    },
    {}
  );
  VideoConversion.associate = function(models) {
    // associations can be defined here
  };
  return VideoConversion;
};
```

to add the enum constants to the model.

Next run `npx sequelize-init db:migrate` to create our database.

Then create the `files` folder in the back end folder for storing the files.

Next we create our video processing job queue. Create a `queues` folder and inside it, create `videoQueue.js` file and add:

```javascript
const Queue = require("bull");
const videoQueue = new Queue("video transcoding");
const models = require("../models");
var ffmpeg = require("fluent-ffmpeg");
const fs = require("fs");
const convertVideo = (path, format) => {
  const fileName = path.replace(/\.[^/.]+$/, "");
  const convertedFilePath = `${fileName}_${+new Date()}.${format}`;
  return new Promise((resolve, reject) => {
    ffmpeg(`${__dirname}/../files/${path}`)
      .setFfmpegPath(process.env.FFMPEG_PATH)
      .setFfprobePath(process.env.FFPROBE_PATH)
      .toFormat(format)
      .on("start", commandLine => {
        console.log(`Spawned Ffmpeg with command: ${commandLine}`);
      })
      .on("error", (err, stdout, stderr) => {
        console.log(err, stdout, stderr);
        reject(err);
      })
      .on("end", (stdout, stderr) => {
        console.log(stdout, stderr);
        resolve({ convertedFilePath });
      })
      .saveToFile(`${__dirname}/../files/${convertedFilePath}`);
  });
};
videoQueue.process(async job => {
  const { id, path, outputFormat } = job.data;
  try {
    const conversions = await models.VideoConversion.findAll({ where: { id } });
    const conv = conversions[0];
    if (conv.status == "cancelled") {
      return Promise.resolve();
    }
    const pathObj = await convertVideo(path, outputFormat);
    const convertedFilePath = pathObj.convertedFilePath;
    const conversion = await models.VideoConversion.update(
      { convertedFilePath, status: "done" },
      {
        where: { id }
      }
    );
    Promise.resolve(conversion);
  } catch (error) {
    Promise.reject(error);
  }
});
export { videoQueue };
```

In the `convertVideo` function, we use `fluent-ffmpeg` to get the video file, then set the FFMPEG and FFProbe paths from the environment variables. Then we call `toFormat` to convert it to the format we specify. We log in the start, error, and end handlers to see the outputs and resolve our promise on the end event. When conversion is done, we save it to a new file.

`videoQueue` is a Bull queue that processes jobs in the background sequentially. Redis is required to run the queue, we will need a Ubuntu Linux installation. We run the follwing commands in Ubuntu to install and run Redis:

```bash
$ sudo apt-get update  
$ sudo apt-get upgrade  
$ sudo apt-get install redis-server  
$ redis-server
```

In the callback of the `videoQueue.process` function, we call the `convertVideo` function and update the path of the converted file and the status of the given job when the job is done.

Next we create our routes. Create a `conversions.js` file in the `routes` folder and add:

```javascript
var express = require("express");
var router = express.Router();
const models = require("../models");
var multer = require("multer");
const fs = require("fs").promises;
const path = require("path");
import { videoQueue } from "../queues/videoQueue";
const rateLimit = require("express-rate-limit");
const limiter = rateLimit({
  windowMs: 60000,
  max: process.env.CALL_PER_MINUTE || 10,
  message: {
    error: "Too many requests"
  }
});
const storage = multer.diskStorage({
  destination: (req, file, cb) => {
    cb(null, "./files");
  },
  filename: (req, file, cb) => {
    cb(null, `${+new Date()}_${file.originalname}`);
  }
});
const upload = multer({ storage });
router.get("/", async (req, res, next) => {
  const conversions = await models.VideoConversion.findAll();
  res.json(conversions);
});
router.post("/", limiter, upload.single("video"), async (req, res, next) => {
  const data = { ...req.body, filePath: req.file.path };
  const conversion = await models.VideoConversion.create(data);
  res.json(conversion);
});
router.delete("/:id", async (req, res, next) => {
  const id = req.params.id;
  const conversions = await models.VideoConversion.findAll({ where: { id } });
  const conversion = conversions[0];
  try {
    await fs.unlink(`${__dirname}/../${conversion.filePath}`);
    if (conversion.convertedFilePath) {
      await fs.unlink(`${__dirname}/../files/${conversion.convertedFilePath}`);
    }
  } catch (error) {
  } finally {
    await models.VideoConversion.destroy({ where: { id } });
    res.json({});
  }
});
router.put("/cancel/:id", async (req, res, next) => {
  const id = req.params.id;
  const conversion = await models.VideoConversion.update(
    { status: "cancelled" },
    {
      where: { id }
    }
  );
  res.json(conversion);
});
router.get("/start/:id", limiter, async (req, res, next) => {
  const id = req.params.id;
  const conversions = await models.VideoConversion.findAll({ where: { id } });
  const conversion = conversions[0];
  const outputFormat = conversion.outputFormat;
  const filePath = path.basename(conversion.filePath);
  await videoQueue.add({ id, path: filePath, outputFormat });
  res.json({});
});
module.exports = router;
```

In the POST `/` route, we accept the file upload with the Multer package. We add the job and save the file to the `files` folder that we created before. We save it with the file’s original name in the `filename` function in the object we passed into the `diskStorage` function and specified the file be saved in the `files` folder in the `destination` function.

The GET route `/` route gets the jobs added. DELETE `/` deletes the job with the given ID along with the source file of the job. PUT `/cancel/:id` route sets `status` to `cancelled` .

And the GET `/start/:id` route add the job with the given ID to the queue we created earlier.

We added `limiter` object here to use `express-rate-limit` to limit the number of API calls to POST `/` route and the GET `/start/:id` route to prevent too many jobs from be added and started respectively.

In `app.js` , we replace the existing code with:

```javascript
require("dotenv").config();
var createError = require("http-errors");
var express = require("express");
var path = require("path");
var cookieParser = require("cookie-parser");
var logger = require("morgan");
var cors = require("cors");
var indexRouter = require("./routes/index");
var conversionsRouter = require("./routes/conversions");
var app = express();
// view engine setup
app.set("views", path.join(__dirname, "views"));
app.set("view engine", "jade");
app.use(logger("dev"));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, "public")));
app.use(express.static(path.join(__dirname, "files")));
app.use(cors());
app.use("/", indexRouter);
app.use("/conversions", conversionsRouter);
// catch 404 and forward to error handler
app.use(function(req, res, next) {
  next(createError(404));
});
// error handler
app.use(function(err, req, res, next) {
  // set locals, only providing error in development
  res.locals.message = err.message;
  res.locals.error = req.app.get("env") === "development" ? err : {};
// render the error page
  res.status(err.status || 500);
  res.render("error");
});
module.exports = app;
```

to add the CORS add-on to enable cross domain communication, expose the `files` folder to the public, and expose the `conversions` routes we created earlier to the public.

To add the environment variables, create an `.env` file in the `backend` folder and add:

```bat
FFMPEG\_PATH='c:\\ffmpeg\\bin\\ffmpeg.exe'  
FFPROBE\_PATH='c:\\ffmpeg\\bin\\ffprobe.exe'  
CALL\_PER\_MINUTE=5
```

Change the paths to the FFMPEG and FFProbe paths in your computer and configure `CALL_PER_MINUTE` to whatever you like as long as it is more than zero.

Front End
=========

With back end done, we can move on to the front end. In the project’s root folder, run `npx create-react-app frontend` to create the front end files.

Next we install some packages. We need Axios for making HTTP requests, Formik for form value handling, MobX for state management, React Router for routing URLs to our pages, and Bootstrap for styling.

Run `npm i axios bootstrap formik mobx mobx-react react-bootstrap react-router-dom` to install the packages.

Next we replace the existing code in `App.js` with:

```jsx
import React from "react";
import { Router, Route } from "react-router-dom";
import "./App.css";
import { createBrowserHistory as createHistory } from "history";
import HomePage from "./HomePage";
import { ConversionsStore } from "./store";
import TopBar from "./TopBar";
const conversionsStore = new ConversionsStore();
const history = createHistory();
function App() {
  return (
    <div className="App">
      <TopBar />
      <Router history={history}>
        <Route
          path="/"
          exact
          component={props => (
            <HomePage {...props} conversionsStore={conversionsStore} />
          )}
        />
      </Router>
    </div>
  );
}
export default App;
```

We add the top bar and the routes in this file.

In `App.css` , we replace the existing code with:

```css
.page {
  padding: 20px;
}
.button {
  margin-right: 10px;
}
```

to adding padding and margins to our page and buttons.

Next create `HomePage.js` in the `src` folder and add:

```jsx
import React from "react";
import Table from "react-bootstrap/Table";
import Button from "react-bootstrap/Button";
import ButtonToolbar from "react-bootstrap/ButtonToolbar";
import { observer } from "mobx-react";
import {
  getJobs,
  addJob,
  deleteJob,
  cancel,
  startJob,
  APIURL
} from "./request";
import { Formik } from "formik";
import Form from "react-bootstrap/Form";
import Col from "react-bootstrap/Col";
function HomePage({ conversionsStore }) {
  const fileRef = React.createRef();
  const [file, setFile] = React.useState(null);
  const [fileName, setFileName] = React.useState("");
  const [initialized, setInitialized] = React.useState(false);
  const onChange = event => {
    setFile(event.target.files[0]);
    setFileName(event.target.files[0].name);
  };
  const openFileDialog = () => {
    fileRef.current.click();
  };
  const handleSubmit = async evt => {
    if (!file) {
      return;
    }
    let bodyFormData = new FormData();
    bodyFormData.set("outputFormat", evt.outputFormat);
    bodyFormData.append("video", file);
    try {
      await addJob(bodyFormData);
    } catch (error) {
      alert(error.response.statusText);
    } finally {
      getConversionJobs();
    }
  };
  const getConversionJobs = async () => {
    const response = await getJobs();
    conversionsStore.setConversions(response.data);
  };
  const deleteConversionJob = async id => {
    await deleteJob(id);
    getConversionJobs();
  };
  const cancelConversionJob = async id => {
    await cancel(id);
    getConversionJobs();
  };
  const startConversionJob = async id => {
    await startJob(id);
    getConversionJobs();
  };
  React.useEffect(() => {
    if (!initialized) {
      getConversionJobs();
      setInitialized(true);
    }
  });
  return (
    <div className="page">
      <h1 className="text-center">Convert Video</h1>
      <Formik onSubmit={handleSubmit} initialValues={{ outputFormat: "mp4" }}>
        {({
          handleSubmit,
          handleChange,
          handleBlur,
          values,
          touched,
          isInvalid,
          errors
        }) => (
          <Form noValidate onSubmit={handleSubmit}>
            <Form.Row>
              <Form.Group
                as={Col}
                md="12"
                controlId="outputFormat"
                defaultValue="mp4"
              >
                <Form.Label>Output Format</Form.Label>
                <Form.Control
                  as="select"
                  value={values.outputFormat || "mp4"}
                  onChange={handleChange}
                  isInvalid={touched.outputFormat && errors.outputFormat}
                >
                  <option value="mov">mov</option>
                  <option value="webm">webm</option>
                  <option value="mp4">mp4</option>
                  <option value="mpeg">mpeg</option>
                  <option value="3gp">3gp</option>
                </Form.Control>
                <Form.Control.Feedback type="invalid">
                  {errors.outputFormat}
                </Form.Control.Feedback>
              </Form.Group>
            </Form.Row>
            <Form.Row>
              <Form.Group as={Col} md="12" controlId="video">
                <input
                  type="file"
                  style={{ display: "none" }}
                  ref={fileRef}
                  onChange={onChange}
                  name="video"
                />
                <ButtonToolbar>
                  <Button
                    className="button"
                    onClick={openFileDialog}
                    type="button"
                  >
                    Upload
                  </Button>
                  <span>{fileName}</span>
                </ButtonToolbar>
              </Form.Group>
            </Form.Row>
            <Button type="submit">Add Job</Button>
          </Form>
        )}
      </Formik>
      <br />
      <Table>
        <thead>
          <tr>
            <th>File Name</th>
            <th>Converted File</th>
            <th>Output Format</th>
            <th>Status</th>
            <th>Start</th>
            <th>Cancel</th>
            <th>Delete</th>
          </tr>
        </thead>
        <tbody>
          {conversionsStore.conversions.map((c, i) => {
            return (
              <tr key={i}>
                <td>{c.filePath}</td>
                <td>{c.status}</td>
                <td>{c.outputFormat}</td>
                <td>
                  {c.convertedFilePath ? (
                    <a href={`${APIURL}/${c.convertedFilePath}`}>Open</a>
                  ) : (
                    "Not Available"
                  )}
                </td>
                <td>
                  <Button
                    className="button"
                    type="button"
                    onClick={startConversionJob.bind(this, c.id)}
                  >
                    Start
                  </Button>
                </td>
                <td>
                  <Button
                    className="button"
                    type="button"
                    onClick={cancelConversionJob.bind(this, c.id)}
                  >
                    Cancel
                  </Button>
                </td>
                <td>
                  <Button
                    className="button"
                    type="button"
                    onClick={deleteConversionJob.bind(this, c.id)}
                  >
                    Delete
                  </Button>
                </td>
              </tr>
            );
          })}
        </tbody>
      </Table>
    </div>
  );
}
export default observer(HomePage)
```

This is the home page of our app. We have a drop down for selecting the format of the file, a upload button for select the file for conversion, and a table for displaying the video conversion jobs with the status and file names of the source and converted files.

We also have buttons to start, cancel and delete each job.

To add file upload, we have a hidden file input and in the `onChange` handler of the file input, we set the file. The Upload button’s `onClick` handler will click the file input to open the upload file dialog.

We get latest jobs by calling `getConversionJobs` we we first load the page, and when we start, cancel and delete jobs. The job data are stored in the MobX store that we will create later. We wrap `observer` in our `HomePage` in the last line to always get the latest values from the store.

Next create `request.js` and the `src` folder and add:

```javascript
const axios = require("axios");
export const APIURL = "http://localhost:3000";
export const getJobs = () => axios.get(`${APIURL}/conversions`);
export const addJob = data =>
  axios({
    method: "post",
    url: `${APIURL}/conversions`,
    data,
    config: { headers: { "Content-Type": "multipart/form-data" } }
  });
export const cancel = id => axios.put(`${APIURL}/conversions/cancel/${id}`, {});
export const deleteJob = id =>
  axios.delete(`${APIURL}/conversions/${id}`);
export const startJob = id => axios.get(`${APIURL}/conversions/start/${id}`);
```

The HTTP requests that we make to the back end are all here. They were used on the `HomePage`.

Next create the MobX store by creating `store.js` file in the `src` folder. In there add:

```javascript
import { observable, action, decorate } from "mobx";
class ConversionsStore {
  conversions = [];
setConversions(conversions) {
    this.conversions = conversions;
  }
}
ConversionsStore = decorate(ConversionsStore, {
  conversions: observable,
  setConversions: action
});
export { ConversionsStore };
```

This is a simple store which stores the contacts the `conversions` array is where we store the contacts for the whole app. The `setConversions` function let us set contacts from any component where we pass in the this store object to.

Next create `TopBar.js` in the `src` folder and add:

```jsx
import React from "react";
import Navbar from "react-bootstrap/Navbar";
import Nav from "react-bootstrap/Nav";
function TopBar() {
  return (
    <Navbar bg="primary" expand="lg" variant="dark">
      <Navbar.Brand href="#home">Video Converter</Navbar.Brand>
      <Navbar.Toggle aria-controls="basic-navbar-nav" />
      <Navbar.Collapse id="basic-navbar-nav">
        <Nav className="mr-auto">
          <Nav.Link href="/">Home</Nav.Link>
        </Nav>
      </Navbar.Collapse>
    </Navbar>
  );
}
export default TopBar;
```

This contains the React Bootstrap `Navbar` to show a top bar with a link to the home page and the name of the app.

In `index.html` , we replace the existing code with:

```html
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <link rel="shortcut icon" href="%PUBLIC_URL%/favicon.ico" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <meta name="theme-color" content="#000000" />
    <meta
      name="description"
      content="Web site created using create-react-app"
    />
    <link rel="apple-touch-icon" href="logo192.png" />
    <!--
      manifest.json provides metadata used when your web app is installed on a
      user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
    -->
    <link rel="manifest" href="%PUBLIC_URL%/manifest.json" />
    <!--
      Notice the use of %PUBLIC_URL% in the tags above.
      It will be replaced with the URL of the `public` folder during the build.
      Only files inside the `public` folder can be referenced from the HTML.
Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
      work correctly both with client-side routing and a non-root public URL.
      Learn how to configure a non-root public URL by running `npm run build`.
    -->
    <title>Video Converter</title>
    <link
      rel="stylesheet"
      href="https://maxcdn.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css"
      integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T"
      crossorigin="anonymous"
    />
  </head>
  <body>
    <noscript>You need to enable JavaScript to run this app.</noscript>
    <div id="root"></div>
    <!--
      This HTML file is a template.
      If you open it directly in the browser, you will see an empty page.
You can add webfonts, meta tags, or analytics to this file.
      The build step will place the bundled scripts into the <body> tag.
To begin the development, run `npm start` or `yarn start`.
      To create a production bundle, use `npm run build` or `yarn build`.
    -->
  </body>
</html>
```

to add the Bootstrap CSS and change the title.

After writing all that code, we can run our app. Before running anything, install `nodemon` by running `npm i -g nodemon` so that we don’t have to restart back end ourselves when files change.

Then run back end by running `npm start` in the `backend` folder and `npm start` in the `frontend` folder, then choose ‘yes’ if you’re asked to run it from a different port.