Learn Node.js and MongoDB by building a URL Shortener app
Tags: node, mongodb, javascript
Published: 2019-02-02 08:27:54

---
title: Learn Node.js and MongoDB by building a URL Shortener app
published: true
tags: node,mongodb,javascript
canonical_url: https://freshman.tech/url-shortener/
---

In this article, you’ll learn how to build a URL Shortener application with Node.js and MongoDB. Here’s a [live demo](https://freshman-shortener.herokuapp.com/) of what we’ll be building. You can find the complete source code for this project in this [GitHub repo](https://github.com/freshman-tech/url-shortener).

## Prerequisites

I assume basic familiarity with JavaScript as well as the command line. If you haven’t built a basic Node application before, you might want to [start here](/microservice) first, then return to this tutorial at a later time.

You also need to have Node.js and npm installed on your computer. You can visit the [Node.js website](https://nodejs.org/en/download/) to view installation instructions for your operating system. [npm](https://npmjs.com) comes bundled with Node, so once you install Node, you’ll have access to the `npm` command too.

The versions I used while building this project are as follows:

- Node.js v11.2.0
- npm v6.6.0

You can view the version of Node and `npm` you have installed by running the following commands in your terminal:

```
node -v
npm -v
```

## Grab the starter files

Grab the starter files for this project at this [GitHub repository](https://github.com/Freshman-tech/url-shortener-starter-files). Clone the repo to your computer and `cd` into the created directory. Then run `npm install` from the project root to install all the dependencies specified in the `package.json` file. I’ll get into what each of the dependencies do later on as we progress.

## Install MongoDB

MongoDB is a free and open-source NoSQL document database commonly used in modern web applications. You’ll need to have it installed on your machine. At the time of writing, the latest stable version is **4.0.5**. This is the version I used throughout this tutorial.

Here are the installation instructions for [Linux](https://docs.mongodb.com/manual/administration/install-on-linux/), [macOS](https://docs.mongodb.com/manual/tutorial/install-mongodb-on-os-x/) and [Windows](https://docs.mongodb.com/v3.2/tutorial/install-mongodb-on-windows/). If you’re on Ubuntu like myself, you can install MongoDB using `apt`:

```
sudo apt update
sudo apt install -y mongodb
```

You can checkout what version of mongoDB you have installed using `mongo --version`.

The database server should be automatically started after the installation process, but you should verify this before moving on from this step. On Ubuntu, you can use the following command to check the status of the mongoDB server:

```
sudo systemctl status mongodb
```

You should see this output:

![Terminal showing active status of MongoDB server](https://freshman.tech/assets/dist/images/learn-mongodb/mongodb-status-ubuntu.png)

If not, you can start it using the following command:

```
sudo systemctl start mongodb
```

## Set up a basic Node server

Looking through the `src` folder within the project directory, you’ll see that we’ve got a `server.js` file and a `public` folder containing the markup and styles for the application frontend. The `server.js` file is where the bulk of the application code will be written in.

Unlike the previous Node tutorial where I used the built-in http module to set up the Node server, we’ll be using [Express](https://expressjs.com/), a popular Node.js web application framework in this instance.

There are other web frameworks out there, but Express is simple enough, well documented and well supported so you shouldn’t run into many issues when using it in your applications.

If you look through the `package.json` file, you will see the `express` package is part of the dependencies that we installed earlier. Let’s go ahead and use it to set up the Node server in `server.js`:

```javascript
const express = require('express');

const app = express();

app.set('port', process.env.PORT || 4100);
const server = app.listen(app.get('port'), () => {
  console.log(`Express running → PORT ${server.address().port}`);
});
```

You can start the server by running `npm start` in the terminal. I am making use of the [Nodemon](https://www.npmjs.com/package/nodemon) package to auto restart the Node server when changes are made to it so we don’t have to do so ourselves.

## Set up the application frontend

As mentioned earlier, the app frontend lives in the `public` folder. We need to set up a new route on the server so that when a user visits the application, the HTML file will be sent and rendered in the browser.

Change the `server.js` file to look like this:

```javascript
const express = require('express');
const path = require('path');

const app = express();

app.get('/', (req, res) => {
  const htmlPath = path.join(__dirname, 'public', 'index.html');
  res.sendFile(htmlPath);
});

app.set('port', process.env.PORT || 4100);
const server = app.listen(app.get('port'), () => {
  console.log(`Express running → PORT ${server.address().port}`);
});
```

[path](https://nodejs.org/api/path.html) is a built-in module in Node.js. It allows us to link to directories and file paths in Node.js. The `sendFile()` method takes an absolute path to the file, so ` __dirname` is used to avoid hardcoding the path. `__ dirname` is the directory in which the executing file is located, so `path.join(__dirname, 'public', 'index.html')` will resolve to `src/public/index.html`.

Navigate to `http://localhost:4100` in your browser. Notice that the HTML is rendered correctly. However, the styles are missing even though `style.css` was linked correctly in `index.html`.

![Missing styles on the Application frontend](https://freshman.tech/assets/dist/images/learn-mongodb/frontend-no-styles.png)

When the browser encounters the reference to `style.css`, it fires off a request to the server for that file. But since we haven’t configured our server to handle requests for static files (such as images, CSS and JavaScript), the server does nothing, and the request fails leaving the page without styles.

To fix this situation, we need to configure express to handle requests for static files correctly. We can do this using the a [built-in middleware function](https://expressjs.com/en/starter/static-files.html) in express as follows:

```javascript
// beginning of the file

const app = express();

app.use(express.static(path.join(__dirname, 'public')))

// rest of the file
```

Now reload the page. It should work correctly:

![Application styles are properly loaded](https://freshman.tech/assets/dist/images/learn-mongodb/frontend-with-styles.png)

## Submit the form to the server

We need to write client side JavaScript to submit the contents of the form input to the server when a user submits the form.

We can do this without using any client side JavaScript by setting the `action` attribute of the form to a route on the server and setting the `method` attribute to `POST`, but I’ve opted to use JavaScript here so that we can handle the response and display the shortened url to the user without a full page refresh.

Create a new `main.js` file in the `public` directory and add the following code into it:

```javascript
const form = document.querySelector('.url-form');
const result = document.querySelector('.result-section');
form.addEventListener('submit', event => {
  event.preventDefault();

  const input = document.querySelector('.url-input');
  fetch('/new', {
    method: 'POST',
    headers: {
      'Accept': 'application/json',
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      url: input.value,
    })
  })
    .then(response => {
      if (!response.ok) {
        throw Error(response.statusText);
      }
      return response.json();
    })
    .then(data => {
      while (result.hasChildNodes()) {
        result.removeChild(result.lastChild);
      }

      result.insertAdjacentHTML('afterbegin', `
        <div class="result">
          <a target="_blank" class="short-url" rel="noopener" href="/${data.short_id}">
            ${location.origin}/${data.short_id}
          </a>
        </div>
      `)
    })
    .catch(console.error)
});
```

This code listens for the `submit` event on the form, prevents the form submission and fires of a POST request to the server with the value of the form input in request body. The reason we wrap the body object in `JSON.stringify` is so that we can consume it as JSON on the server.

Note that we’re posting the data to the `/new` route which hasn’t been created on the server yet. We’ll create it in the next section. Before that, make sure you reference `main.js` in your `index.html` file before the closing body tag:

```html
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <meta http-equiv="X-UA-Compatible" content="ie=edge">
  <title>URL Shortener</title>
  <link rel="stylesheet" href="style.css" type="text/css">
</head>
<body>
  // rest of the code

  <script src="main.js"></script>
</body>
</html>
```

## Access the form body on the server

Let’s go ahead and create the `/new` route that will process the URLs to be shortened. Add this below the root route in `server.js`:

```javascript
app.post('/new', (req, res) => {

});
```

The first thing we need to do is access the JSON data that was sent from the client in the request body. To do this, we need to use of the [body-parser](https://www.npmjs.org/package/body-parser) package. This package parses all incoming request bodies and makes them accessible on `req.body`.

Since this package has been installed already, we can use it right away in `server.js`:

```javascript
const express = require('express');
const path = require('path');
const bodyParser = require('body-parser');

const app = express();
app.use(express.static(path.join(__dirname, 'public')))
app.use(bodyParser.json());

// rest of the file
```

Now, we should be able to access the request body in the `req.body` property in the `/new` route:

```javascript
app.post('/new', (req, res) => {
  console.log(req.body);
});
```

You can try this out by entering a URL into the form and submitting it. Then navigate to the terminal where your server is running to see the JSON data printed in the terminal.

![Request body is printed to the terminal](https://freshman.tech/assets/dist/images/learn-mongodb/json-data-in-terminal.png)

This means we can access the url using `req.body.url`.

## Validate the URL

Before we shorten the URL, we need to validate if the URL that was submitted is valid. Client side validation is handled for us in the browser because we’ve set the type of the input to `url` so the form won’t be submitted if the value doesn’t have a valid URL structure.

To make the application more robust, we need to validate the URL on the server as well. There are several npm packages that can handle this, but I’ve opted to do so using a few built-in Node modules.

The first thing we need to do is check if the URL has a valid structure, then we perform a DNS lookup to see if the domain is operational. A domain like `https://google.com` will pass both tests, but `http://jidfsdm.com` will fail the second one since that site does not exist.

Require the built-in [dns](https://nodejs.org/api/dns.html) module at the top of `server.js`:

```javascript
const dns = require('dns');
```

Then change up the `/new` route as follows:

```javascript
app.post('/new', (req, res) => {
  let originalUrl;
  try {
    originalUrl = new URL(req.body.url);
  } catch (err) {
    return res.status(400).send({ error: 'invalid URL' });
  }

  dns.lookup(originalUrl.hostname, (err) => {
    if (err) {
      return res.status(404).send({ error: 'Address not found' });
    };
  });
});
```

The [URL class](https://nodejs.org/api/url.html#url_the_whatwg_url_api) returns a new `URL` object with several properties if the input URL has a valid structure. Otherwise, it throws an error which we can `catch` and send back to the client.

If the URL input passes the first test, we then check to see if the domain is operational by passing the [hostname](https://developer.mozilla.org/en-US/docs/Web/API/URL/hostname) portion of the URL (the domain) to `dns.lookup` which checks if the domain is live. If so, we can connect to our MongoDB instance and create the shortened version of the url as you’ll see.

## Set up environmental variables

Environmental variables are a great way to configure how your program should work. They are key-value pairs that are stored on the local system where your program is being run, and are accessible from within your code.

It’s considered best practice to set app configuration data such as API keys, tokens, passwords and other sensitive details as environmental variables instead of hardcoding it into the program itself. This prevents you from accidentally exposing it to others, and also makes the values really easy to change without having to touch your code.

In Node.js, you can access variables defined in your environment via the `process.env` object. You can check the contents of this project via the Node.js REPL as shown below:

![Node REPL showing contents of process.env](https://freshman.tech/assets/images/learn-mongodb/process-env-poster.png)

Aside from the operating system variables that are present by default, we can create project specific variables using a `.env` file. To avoid checking these files into source control, you should add <code>.env</code> to your <code>.gitignore</code>


Create a `.env` file at the root of your project directory and paste the following code into it:

```
DATABASE=mongodb://localhost:27017
```

Here, we’ve added the URL of our local MongoDB instance as an environmental variable. Port 27017 is the port that MongoDB runs on. The next thing to do is load the values defined in `.env` into `process.env`. The easiest way to do this is with the [dotenv](https://www.npmjs.com/package/dotenv) package which is already part of our app dependencies.

Add the following at the very top of `server.js`:

```javascript
require('dotenv').config();

// rest of the file
```

This will read the contents of the `.env` file in the root of your project, parse its contents and initialize the values on `process.env`. Now, you’ll be able to access any set variable under `process.env.<KEY>`.

## Connect to MongoDB

Let’s go ahead and connect to our local MongoDB instance in `server.js` as shown below:

```javascript
require('dotenv').config()

const express = require('express');
const bodyParser = require('body-parser');
const path = require('path');
const dns = require('dns');
const { MongoClient } = require('mongodb');

const databaseUrl = process.env.DATABASE;

const app = express();
app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());

app.use(express.static(path.join(__dirname, 'public')))

MongoClient.connect(databaseUrl, { useNewUrlParser: true })
  .then(client => {
    app.locals.db = client.db('shortener');
  })
  .catch(() => console.error('Failed to connect to the database'));

// rest of the file
```

First, we’re importing `MongoClient` from [mongodb](https://mongodb.github.io/node-mongodb-native/) which is the the native driver for interacting with a MongoDB instance in Node.js. Then we connect to the MongoDB instance specified in the `DATABASE` environmental variable.

If the connection is successful, we get a reference to the MongoDB instance client and can select a database using the `client.db()` method. Note that this method creates the database if it does not already exist.

Here, we are selecting a reference to the `shortener` database and storing that reference in `app.locals` which is an object provided by `express`. This object allows us to set local variables that persist throughout the life of the application, and can be accessed in other middleware functions (functions that have access to the `req` or `res` objects) via `req.app.locals`.

The reason we’re storing a reference to the database in `app.locals.db` is so that we can reuse the db object without having to open another connection to the MongoDB instance.

## Shorten URLs

The next step is to actually shorten the URL and store it in the database. To create a unique short id for each url, we will be making use of the [nanoid](https://github.com/ai/nanoid) package.

Require it at the top of `server.js` below the other `require` statements:

```javascript
const nanoid = require('nanoid');
```

Then create a new `shortenURL` function in `server.js` as follows:

```javascript
// beginning of the file

MongoClient.connect(databaseUrl, { useNewUrlParser: true })
  .then(client => {
    app.locals.db = client.db('shortener');
  })
  .catch(() => console.error('Failed to connect to the database'));

const shortenURL = (db, url) => {
  const shortenedURLs = db.collection('shortenedURLs');
  return shortenedURLs.findOneAndUpdate({ original_url: url },
    {
      $setOnInsert: {
        original_url: url,
        short_id: nanoid(7),
      },
    },
    {
      returnOriginal: false,
      upsert: true,
    }
  );
};

// rest of the file
```

We need to reference a collection before we can add any data into the database. We can do so using the using the `db.collection()` method. If the collection does not exist yet, it’s created.

Before we shorten the URL and add it to the database, we need to check if the URL has not been shortened already to prevent duplicate database entries for one URL. We can do this using the `findOneAndUpdate()` method on the collection which allows us to modify a document that already exists in the database collection or create it if it doesn’t exist.

This method takes a few arguments. The first one is an object that is used to filter the collection. Here we are passing an object whose `original_url` property matches the url we are about to shorten. If a document in the database matches this filter, it will be returned and updated according to the [update operators](https://docs.mongodb.com/manual/reference/operator/update/) set in the second argument.

The `$setOnInsert` operator allows us to set the value of the document only if its being inserted. This means that the document will not be modified if it already exists, but if it doesn’t, it will be created with its value set to whatever we set in `$setOnInsert`.

In this case, the document will have two properties: `original_url` which is the url to be shortened, and `short_id` which is a unique 7 character id for that url.

We also need to set the `upsert` option to `true`. This ensures that the document is created if it doesn’t exist. Otherwise, `$setOnInsert` has no effect. Finally, setting `returnOriginal` to `false` ensures that `findOneAndUpdate` returns the new document if one is upserted, which is what we want in this case.

We can make use of the `shortenURL()` function in the `/new` route like this:

```javascript
app.post('/new', (req, res) => {
  let originalUrl;
  try {
    originalUrl = new URL(req.body.url);
  } catch (err) {
    return res.status(400).send({error: 'invalid URL'});
  }

  dns.lookup(originalUrl.hostname, (err) => {
    if (err) {
      return res.status(404).send({error: 'Address not found'});
    };

    const { db } = req.app.locals;
    shortenURL(db, originalUrl.href)
      .then(result => {
        const doc = result.value;
        res.json({
          original_url: doc.original_url,
          short_id: doc.short_id,
        });
      })
      .catch(console.error);
  });
});
```

At this point, the document that was inserted will be sent to the client as JSON. It will be displayed on the page like this:

![URL is shortened](https://freshman.tech/assets/dist/images/learn-mongodb/shortened-url.png)

## Set up a catch-all route for all shortened URLs

Apart from the `/` and `/new` routes, we need to handle the other requests for the shortened URLs. Specifically, we need to redirect them to the original URLs. Here’s how we can do so using Express:

```javascript
app.get('/:short_id', (req, res) => {
  const shortId = req.params.short_id;

});
```

Here, we’re using named route parameters to capture the value of the `short_id` part of the URL. This value can then be accessed in the `req.params` object under the `short_id` property.

Once we have the short id, we need to check if a url with that short id exists in the database. Let’s create a new function for this purpose just below `shortenURL`:

```javascript
const checkIfShortIdExists = (db, code) => db.collection('shortenedURLs')
  .findOne({ short_id: code });
```

The `findOne` method returns a document that matches the filter object passed to it or `null` if no documents match the filter.

We can then use the function in our catch-all route like this:

```javascript
app.get('/:short_id', (req, res) => {
  const shortId = req.params.short_id;

  const { db } = req.app.locals;
  checkIfShortIdExists(db, shortId)
    .then(doc => {
      if (doc === null) return res.send('Uh oh. We could not find a link at that URL');

      res.redirect(doc.original_url)
    })
    .catch(console.error);
});
```

If the short id exists in our database collection, we will redirect the user to the `original_url` associated with that short id. Otherwise, an error message is sent to the user.

Now, you should be able to shorten long urls, visit the shortened url and be redirected to the original url.

## Visualise your database with a GUI

A GUI will allow us to connect to our MongoDB instance and visualise all the data that is present. You also be able to create new data, update existing data and perform other similar operations.

There are several MongoDB GUIs out there, but the one I like to use is [NoSQLBooster](https://nosqlbooster.com/). It is available for Linux, macOS and Windows, and you can download it [here](https://nosqlbooster.com/downloads).

Once you install and open the app, it should connect to your local MongoDB instance by default. Otherwise, you can click the **Connect** button in the top left and connect to it from there.

Once connected, you’ll able to view all the collections present in the database, and interact with the data present in those collections.

![NoSQLBooster for Linux](https://freshman.tech/assets/dist/images/learn-mongodb/nosqlbooster.png)

## Deploy to Heroku

Before we deploy the app to Heroku, we need to create a cloud hosted MongoDB instance first. We can do so using [MongoDB Atlas](https://www.mongodb.com/cloud/atlas). Create an account at that link and create a new cluster when you are redirected to the **Create New Cluster** page.

Next, click the **Security** tab and hit the **Add new user** button. Give the user a name and password, then click the **Add user** button.

![Add user in MongoDB Atlas](https://freshman.tech/assets/dist/images/learn-mongodb/atlas-add-user.png)

Following that, click **IP Whitelist** and then **Add IP Address**. This allows you to choose what IP addresses can access your cluster. Since this is just a demo application, you can allow access from anywhere by clicking the **Allow Access From Anywhere** button.

![Add IP Address in MongoDB Atlast](https://freshman.tech/assets/dist/images/learn-mongodb/atlas-ip.png)

Next, go back to the Overview, and hit the **Connect** button. Choose **Connect your Application** and then **Short SRV connection string**. Take note of that string as we’ll be using it shortly on Heroku.

![Atlas SRV connection string](https://freshman.tech/assets/dist/images/learn-mongodb/atlas-srv.png)

Head over to the Heroku website and [sign up for a free account](https://signup.heroku.com/). Once your account is activated, [follow this link](https://dashboard.heroku.com/new-app?org=personal-apps) to create a new app. Give it a unique name. I called mine **freshman-shortener**.

Next, [follow the instructions here](https://devcenter.heroku.com/articles/heroku-command-line) to install the Heroku CLI on your machine. Then run the `heroku login` command in the terminal to login to your Heroku account.

Make sure you’ve initialised a git repository for your project. If not, run the `git init` command at the root of your project directory, then run the command below to set heroku as a remote for your git repo. Replace `<app name>` with the name of your application.

```
heroku git:remote -a <app name>
```

Next, create a `Procfile` in the root of your project directory (`touch Procfile`) and paste in the following contents:

```
web: node src/server.js
```

Following that, specify the version of Node you are running in your `package.json` file under the `engines` key. I specified version `11.2.0` since that’s the version I’m running on my computer. You should change that value to match the version of Node you have on your machine.

```json
{
  "name": "url-shortener",
  "version": "1.0.0",
  "description": "URL Shortener Demo",
  "main": "src/server.js",
  "scripts": {
    "start": "npx nodemon src/server.js"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/freshman-tech/url-shortener.git"
  },
  "keywords": [
    "url shortener",
    "mongodb",
    "nodejs"
  ],
  "author": "Ayo Isaiah",
  "license": "MIT",
  "bugs": {
    "url": "https://github.com/freshman-tech/url-shortener/issues"
  },
  "homepage": "https://github.com/freshman-tech/url-shortener#readme",
  "devDependencies": {
    "nodemon": "^1.18.9"
  },
  "dependencies": {
    "body-parser": "^1.18.3",
    "dotenv": "^6.2.0",
    "express": "^4.16.4",
    "mongodb": "^3.1.10",
    "nanoid": "^2.0.1"
  },
  "engines": {
    "node": "11.2.0"
  }
}
```

Before you deploy the app, head over to the **Settings** tab in the Heroku dashboard and hit **Reveal Config Vars**. This is where you will set the environmental variables for your app.

As we did earlier on the local `.env` file, enter `DATABASE` as the KEY and the SRV string from MongoDB Atlas as the value, then click **Add**. Remember to replace `<PASSWORD>` in that string with password for the user you created earlier.

![Set up Heroku config values](https://freshman.tech/assets/dist/images/learn-mongodb/heroku-config.png)

Notice how easy it is to change the database location depending on the environment the program is running without touching the application code. This is one huge advantage of using environmental variables for project configuration.

Finally, commit your code and push it to the Heroku remote using the following commands:

```
git add .
git commit -m "Initial commit"
git push heroku master
```

Once the deployment process is done, you can open `https://<your-app-name>.heroku.com` to view and test your project.

## Wrap up

We’ve successfully created a fully featured URL shortener and learnt the basics of Express and MongoDB along the way. We also learnt how to set up a cloud based MongoDB instance on MongoDB Atlas and how to deploy the app to Heroku.

I hope this exercise was helpful to you. If you have any questions regarding this tutorial, please leave a comment below and I’ll get back to you.

Thanks for reading!

*Originally published at [freshman.tech](https://freshman.tech/url-shortener/) on January 24, 2019. [Subscribe to my newsletter](https://freshman.tech/newsletter/) to get my latest tutorials on JavaScript and CSS delivered to your inbox.*