Writing & organizing Node.js API Tests the right way
Tags: node, beginners, typescript, testing
Published: 2022-11-22 14:20:22

---
title: Writing & organizing Node.js API Tests the right way
published: true
description: An architecture for writing and organizing your Unit & Integration Tests for Node.js REST APIs using Jest, Supertest & Chai.
tags: node, beginners, typescript, testing
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hbakaqkfuyzazt8nlqgg.jpg
# Use a ratio of 100:42 for best results.
# published_at: 2022-11-22 14:10 +0000
---

This post was originally published on my [blog](https://larswaechter.dev/blog/nodejs-api-testing/).

---

Testing the code you write is an important step in the process of software engineering. It ensures that your software works as expected and reduces the risk of shipping bugs and vulnerabilities to production. Automated tests in particular play an important role when it comes to testing frequently and consistently. Continous integration makes them even more powerful.

In this blog post I'll show you an architecture for testing Node.js REST APIs that use a **database** in the background. There are some things you should consider in this scenario that we'll talk about. You'll see how to separate and organize your application's components in a way that you can test them independently. Therefore, we'll use **two different approaches**. On the one hand we setup a test environment in which we run ours tests against a **test database**. On the other hand we **mock the database layer** using [mock functions](https://jestjs.io/docs/mock-functions) so that we can run them in an environment in which we have no access to a database.

We'll start by writing unit tests for testing single components of our application. In the next step we combine those components and test them using integration tests. Last but not least, we setup a CI/CD pipeline using GitHub actions and run the tests on each push that is made.

Note that this is not a guide on how testing in general works. There are thousands of articles about frameworks like Jest, Mocha, Supertest and so on.
This is more a guide on how to prepare your Node application and test environment in a way that you can write tests effortlessly and efficiently **with or without a database connection**. There's also an example repo on [GitHub](https://github.com/larswaechter/nodejs-api-testing). You should definitely check it out.

> Disclamer: I know there really is no right or wrong when it comes to the architecture. The following is my prefered one.

Let's start with some of the tools we'll use. You should know most of them:

- **Language**: [Typescript](https://www.npmjs.com/package/typescript)
- **Server**: [Express](https://www.npmjs.com/package/express)
- **Database**: [Postgres](https://www.npmjs.com/package/pg)
- **Testing**: [Jest](https://www.npmjs.com/package/jest) & [Supertest](https://www.npmjs.com/package/supertest) & [Chai](https://www.npmjs.com/package/chai)
- **CI/CD**: [GitHub Actions](https://github.com/features/actions)
- **Containerization**: [Docker](https://www.docker.com/)

One benefit of this architecture is that you can use it with other databases than Postgres too, like [MySQL](https://www.npmjs.com/package/mysql2) for example. In this architecture we don't use any kind of ORM. Moreover, you can replace Jest with [Mocha](https://www.npmjs.com/package/mocha) if that is your desired testing framework.

## Application Architecture

The architecture of our application looks roughly like this:

```
node-api
├── api
│   ├── components
│   │   ├── user
|   |   │   ├── tests               // Tests for each component
|   |   |   │   ├── http.spec.ts
|   |   |   │   ├── mock.spec.ts
|   |   │   |   └── repo.spec.ts
|   |   │   ├── controller.ts
|   |   │   ├── dto.ts
|   |   │   ├── repository.ts
|   |   │   └── routes.ts
│   └── server.ts
├── factories                       // Factories to setup tests
|   ├── helper.ts
|   ├── abs.factory.ts
|   ├── http.factory.ts
|   └── repo.factory.ts
└── app.ts
```

> Note: The example repo contains some more code.

Each component consists of the following four files:

- **controller.ts**: HTTP Handler
- **dto.ts**: Data Transfer Object ([more](https://docs.nestjs.com/controllers#request-payloads))
- **repository.ts**: Database Layer
- **routes.ts**: HTTP Routing

The `tests` directory includes the tests of the according component. If you want to read more about this architecture, checkout [this](https://larswaechter.dev/blog/nodejs-rest-api-structure/) article of mine.

## Config

We'll start by creating an `.env.test` file that contains the secret environment variables for testing. The npm Postgres package uses them automatically when establishing a new database connection. All we have to do is to make sure that they are loaded using [dotenv](https://www.npmjs.com/package/dotenv).

```
NODE_PORT=0
NODE_ENV=test

PGHOST=localhost
PGUSER=root
PGPASSWORD=mypassword
PGDATABASE=nodejs_test
PGPORT=5432
```

Setting `NODE_PORT=0` lets Node choose the first randomly available port that it finds. This can be useful if you run multiple instances of a HTTP server during testing. You can also set a fixed value other than `0` here. Using `PGDATABASE` we provide the name of our test database.

Next, we setup Jest. The config in `jest.config.js` looks as follows:

```js
module.exports = {
  preset: "ts-jest",
  testEnvironment: "node",
  roots: ["src"],
  setupFiles: ["<rootDir>/setup-jest.js"],
}
```

And `setup-jest.js` like this:

```js
require("dotenv").config({
  path: ".env.test",
})
```

This snippet ensures that the appropriate environment variables are loaded from the provided `.env` file before running the tests.

## Testing with database

Let's start with the assumption that we have a test database that we can use. This might be one in a GitHub Actions CI/CD pipeline for example. Later on I'll show you how to test your application without a database connection.

### 3 rules

At the beginning I said that there are some important things we want to consider that make life much easier when testing Node APIs:

1. Separate the database layer
2. Outsource the database connection initialization
3. Outsource the HTTP server initialization

What do I mean by that?

**Separate the database layer**

You should have your own layer, separated from your business logic, that takes care of the communication with the database. In the example Git repo you can find this layer in a component's `repository.ts` file. This allows us to easily mock a layer's methods when we have no database available for testing.

Moreover, it's easier to replace your database system with another one.

```ts
export class UserRepository {
  readAll(): Promise<IUser[]> {
    return new Promise((resolve, reject) => {
      client.query<IUser>("SELECT * FROM users", (err, res) => {
        if (err) {
          Logger.error(err.message)
          reject("Failed to fetch users!")
        } else resolve(res.rows)
      })
    })
  }
```

**Outsource the database connection initialization**

You should already know that when writing tests against a database, you do not run them against the production one. Instead, you setup a test database. Otherwise, you run the risk messing up your production data.

Most of the time, your application connects to the database in a startup script, like `index.js`. After the connection is established, you start the HTTP server. That's what we want to do in our test setup too. This way we can connect to the database and disconnect from it gracefully before and after each test case.

**Outsource the HTTP server initialization**

It's a good practice, whether you use a database or not, to start the HTTP server from inside your tests. Just as we do for the database connection, we create a new HTTP server before and stop it after each test case.

This might look as follows: (you'll see the concrete implementation later on)

```ts
describe("Component Test", () => {
  beforeEach(() => {
    // Connect to db pool && start Express Server
  });

  afterEach(() => {
    // Release db pool client && stop Express Server
  });

  afterAll(() => {
    // End db pool
  });
```

In particular the execution order is:

1. Connect to the database pool
2. Run the SQL seed (create tables)
3. Start the Express server
4. **Run the tests**
5. Shutdown the Express server & release db pool client
6. Repeat step 1 - 5 for each test case and close the pool at the end

### Tests

Each component consists of two test files:

- repo.spec.ts
- http.spec.ts

Both of them make use of so called `TestFactories` which prepare the test setup. You'll see their implementation in the next chapter.

> Note: If you have a look at the example Git repo, you'll see that there are two more: mock.spec.ts and dto.spec.ts. Former one is discussed later on. The latter is not covered in this article.

#### repo.spec.ts

A repository is an additional abstract layer that is responsible for interacting with the database like reading and inserting new data. That layer is what we test in here. Since a database is required in this case, a new pool client is created to connect to the database using the `RepoTestFactory` before each test case. And it is released, right after the test case is completed. At the end, when all test cases are finished, the pool connection is closed.

Example on [GitHub](https://github.com/larswaechter/nodejs-api-testing/blob/main/src/api/components/user/tests/repo.spec.ts)

```ts
describe("User component (REPO)", () => {
  const factory: RepoTestFactory = new RepoTestFactory()
  const dummyUser: IUser = userFactory.build()
  const dummyUserDTO: UserDTO = userDTOFactory.build()

  // Connect to pool
  beforeEach(done => {
    factory.prepareEach(done)
  })

  // Release pool client
  afterEach(() => {
    factory.closeEach()
  })

  // End pool
  afterAll(done => {
    factory.closeAll(done)
  })

  test("create new user", async () => {
    const repo = new UserRepository()
    const user = await repo.create(dummyUserDTO)

    expect(user).to.be.an("object")
    expect(user.id).eq(1)
    expect(user.email).eq(dummyUser.email)
    expect(user.username).eq(dummyUser.username)

    const count = await factory.getTableRowCount("users")
    expect(count).eq(1)
  })
})
```

#### http.spec.ts

Here, we test the integration of the user component's routes, controller and repository. Before each test case, a new pool client is created just as we did above. In addition, a new Express server is started using the `HttpTestFactory`. At the end, both are closed again.

Example on [GitHub](https://github.com/larswaechter/nodejs-api-testing/blob/main/src/api/components/user/tests/http.spec.ts)

```ts
ddescribe("User component (HTTP)", () => {
  const factory: HttpTestFactory = new HttpTestFactory()
  const dummyUser: IUser = userFactory.build()
  const dummyUserDTO: UserDTO = userDTOFactory.build()

  // Connect to pool && start Express Server
  beforeEach(done => {
    factory.prepareEach(done)
  })

  // Release pool client && stop Express Server
  afterEach(done => {
    factory.closeEach(done)
  })

  // End pool
  afterAll(done => {
    factory.closeAll(done)
  })

  test("POST /users", async () => {
    const res = await factory.app
      .post("/users")
      .send(dummyUserDTO)
      .expect(201)
      .expect("Content-Type", /json/)

    const user: IUser = res.body
    expect(user).to.be.an("object")
    expect(user.id).eq(dummyUser.id)
    expect(user.email).eq(dummyUser.email)
    expect(user.username).eq(dummyUser.username)

    const count = await factory.getTableRowCount("users")
    expect(count).eq(1)
  })
})
```

## Factories

The test factories are actually the heart of our tests. They are responsible for setting up and preparing the environment for each test case. That includes:

- Droping & creating all db tables
- Initializing the db connection
- Initializing the HTTP server
- Closing both of them again

There are four [factories](https://github.com/larswaechter/nodejs-api-testing/tree/main/src/factories) in total: `AbsTestFactory`, `RepoTestFactory`, `HttpTestFactory` and `MockTestFactory`. Each of them has its own Typescript class.

The last one is discussed in the chapter "Testing without database".

### AbsTestFactory

The first one `AbsTestFactory` is an abstract base class that is implemented by the other three. It includes among others a method for connecting to the database pool and one for disconnecting from it.

```ts
export abstract class AbsTestFactory implements ITestFactory {
  public poolClient: PoolClient

  abstract prepareEach(cb: (err?: Error) => void): void
  abstract closeEach(cb: (err?: Error) => void): void

  public async getTableRowCount(name: string) {
    const { rows } = await this.poolClient.query(
      `SELECT COUNT(*) FROM ${this.poolClient.escapeIdentifier(name)};`
    )
    return rows.length ? +rows[0].count : 0
  }

  protected connectPool(cb: (err?: Error) => void) {
    pool
      .connect()
      .then(poolClient => {
        this.poolClient = poolClient
        this.poolClient.query(this.seed, cb)
      })
      .catch(cb)
  }

  protected releasePoolClient() {
    this.poolClient.release(true)
  }

  protected endPool(cb: (err?: Error) => void) {
    pool.end(cb)
  }

  private seed = readFileSync(
    join(__dirname, "../../db/scripts/create-tables.sql"),
    {
      encoding: "utf-8",
    }
  )
}
```

Using the `create-tables.sql` script, the factory **drops and recreates** all the tables after the connection is established:

```sql
DROP TABLE IF EXISTS users;

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(50) UNIQUE NOT NULL,
    username VARCHAR(30) UNIQUE NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```

### RepoTestFactory

The `RepoTestFactory` is used by each component's repository test (`repo.spec.ts`) that you just saw above. All it does is use the parent class `AbsTestFactory` to connect to the database.

```ts
export class RepoTestFactory extends AbsTestFactory {
  prepareEach(cb: (err?: Error) => void) {
    this.connectPool(cb)
  }

  closeEach() {
    this.releasePoolClient()
  }

  closeAll(cb: (err?: Error) => void) {
    this.endPool(cb)
  }
}
```

The methods `prepareEach`, `closeEach` and `closeAll` are called for each test case in the Jest `beforeEach`, `afterEach` and `afterAll` lifecycle.

### HttpTestFactory

The last one `HttpTestFactory` is used by each component's HTTP test (`http.spec.ts`). Just like `RepoTestFactory`, it uses the parent class for the database connection. Furthermore, it initializes the Express server.

```ts
export class HttpTestFactory extends AbsTestFactory {
  private readonly server: Server = new Server()
  private readonly http: HttpServer = createServer(this.server.app)

  get app() {
    return supertest(this.server.app)
  }

  prepareEach(cb: (err?: Error) => void) {
    this.connectPool(err => {
      if (err) return cb(err)
      this.http.listen(process.env.NODE_PORT, cb)
    })
  }

  closeEach(cb: (err?: Error) => void) {
    this.http.close(err => {
      this.releasePoolClient()
      cb(err)
    })
  }

  closeAll(cb: (err?: Error) => void) {
    this.endPool(cb)
  }
}
```

## Helpers

In [helper.ts](https://github.com/larswaechter/nodejs-api-testing/blob/main/src/factories/helper.ts), there are two [fishery](https://www.npmjs.com/package/fishery) objects which we can use to create dummy data during the tests.

```ts
export const userFactory = Factory.define<IUser>(({ sequence, onCreate }) => {
  onCreate(
    user =>
      new Promise((resolve, reject) => {
        pool.query<IUser>(
          "INSERT INTO users (email, username) VALUES($1, $2) RETURNING *",
          [user.email, user.username],
          (err, res) => {
            if (err) return reject(err)
            resolve(res.rows[0])
          }
        )
      })
  )

  return {
    id: sequence,
    email: "john@doe.com",
    username: "johndoe",
    created_at: new Date(),
  }
})

export const userDTOFactory = Factory.define<UserDTO>(
  () => new UserDTO("john@doe.com", "johndoe")
)
```

Usage:

```ts
// Returns new `IUser` instance
const dummyUser1 = userFactory.build()

// Returns new `IUser` instance & creates db entry
const dummyUser2 = await userFactory.create()
```

## Rewind

Let's jump back to the `repo.spec.ts` and `http.spec.ts` test files from above. In both of them we used the factories' `prepareEach` method before each and its `afterEach` method after right each test case. The `closeAll` method is called at the very end of the test file. As you have just seen, depending on the type of factory, we establish the database connection and start the HTTP server if needed.

```ts
describe("Component Test", () => {
  beforeEach(done => {
    factory.prepareEach(done)
  })

  afterEach(() => {
    factory.closeEach()
  })

  afterAll(done => {
    factory.closeAll(done)
  })
})
```

One important thing you should keep in mind is that for each test case that uses the database, the factory drops all the tables and recreates them using the provided SQL script afterwards. This way we have a clean database with empty tables in each test case.

## Testing without database

So far we have run our tests against a test database, but what if we have no access to a database? In this case, we need to [mock](https://jestjs.io/docs/es6-class-mocks) our database layer implementation (`repository.ts`), which is quite easy if you have separated it from the business logic, as I recommended in rule #1.

With mocks, the layer does not depend on an external data source any more. Instead, we provide a custom implementation for the class and each of its methods. Be aware that this does not affect the behavior of our [controller](https://github.com/larswaechter/nodejs-api-testing/blob/main/src/api/components/user/controller.ts) since it does not care about where the data comes from.

Example on [GitHub](https://github.com/larswaechter/nodejs-api-testing/blob/main/src/api/components/user/tests/mock.spec.ts)

```ts
const dummyUser = userFactory.build()
const dummyUserDTO = userDTOFactory.build()

const mockReadAll = jest.fn().mockResolvedValue([dummyUser])

const mockReadByID = jest
  .fn()
  .mockResolvedValueOnce(dummyUser)
  .mockResolvedValueOnce(dummyUser)
  .mockResolvedValue(undefined)

const mockCreate = jest.fn().mockResolvedValue(dummyUser)

const mockReadByEmailOrUsername = jest
  .fn()
  .mockResolvedValueOnce(undefined)
  .mockResolvedValueOnce(dummyUser)

const mockDelete = jest.fn().mockResolvedValue(true)

jest.mock("../repository", () => ({
  UserRepository: jest.fn().mockImplementation(() => ({
    readAll: mockReadAll,
    readByID: mockReadByID,
    readByEmailOrUsername: mockReadByEmailOrUsername,
    create: mockCreate,
    delete: mockDelete,
  })),
}))
```

After mocking the database layer, we can write ours tests as usual. Using `toHaveBeenCalledTimes()` we make sure that our custom method implementation has been called.

```ts
describe("User component (MOCK)", () => {
  const factory: MockTestFactory = new MockTestFactory()

  // Start Express Server
  beforeEach(done => {
    factory.prepareEach(done)
  })

  // Stop Express Server
  afterEach(done => {
    factory.closeEach(done)
  })

  test("POST /users", async () => {
    const res = await factory.app
      .post("/users")
      .send(dummyUserDTO)
      .expect(201)
      .expect("Content-Type", /json/)

    const user: IUser = res.body
    cExpect(user).to.be.an("object")
    cExpect(user.id).eq(dummyUser.id)
    cExpect(user.email).eq(dummyUser.email)
    cExpect(user.username).eq(dummyUser.username)

    expect(mockCreate).toHaveBeenCalledTimes(1)
    expect(mockReadByEmailOrUsername).toHaveBeenCalledTimes(1)
  })
})
```

> Note: cExpect is a named import from the "chai" package.

### MockTestFactory

Just as we did in the other tests files, we use a test factory here as well. All the `MockTestFactory` does is run a new Express HTTP instance. It **does not** establish a database connection since we mock the database layer.

```ts
export class MockTestFactory extends AbsTestFactory {
  private readonly server: Server = new Server()
  private readonly http: HttpServer = createServer(this.server.app)

  get app() {
    return supertest(this.server.app)
  }

  prepareEach(cb: (err?: Error) => void) {
    this.http.listen(process.env.NODE_PORT, cb)
  }

  closeEach(cb: (err?: Error) => void) {
    this.http.close(cb)
  }
}
```

One drawback we have using this approach is that the layer (`repository.ts`) is not tested at all because we overwrite it. Nevertheless, we can still test the rest of our application, like the business logic for example. Great!

## Running

Using the the commands below we can run the tests with or without a database. Depending on the scenario, the files we do not want to test are excluded from execution.

```json
{
  "test:db": "jest --testPathIgnorePatterns mock.spec.ts",
  "test:mock": "jest --testPathIgnorePatterns \"(repo|http).spec.ts\""
}
```

## GitHub Actions

The final step is to create a CI/CD pipeline using GitHub actions that runs our tests. The according yaml file is available [here](https://github.com/larswaechter/nodejs-api-testing/blob/main/.github/workflows/actions.yml). There's also a very good [tutorial](https://docs.github.com/en/actions/using-containerized-services/creating-postgresql-service-containers) published on GitHub. You can decide whether to run the tests against a test database or use the mocked data layer. I decided to go with the former.

When running the pipeline with a test database, we need to make sure that we set the correct environment variables for it. [Here](https://github.com/larswaechter/nodejs-api-testing/actions/runs/3514430623/jobs/5888433172) you can find a test run.

## Last words

My last tip is to have a look at the example repository on [GitHub](https://github.com/larswaechter/nodejs-api-testing) and to read it carefully There are some more tests and code snippets that I did not cover in this article. Moreover, checkout the links below. Happy coding!

## Further resources

- [ES6 Class Mocks](https://jestjs.io/docs/es6-class-mocks)
- [Database seeding in Node.js](https://larswaechter.dev/blog/nodejs-database-seeding/)
- [How I structure my Node.js REST APIs](https://larswaechter.dev/blog/nodejs-rest-api-structure/)
- [Creating PostgreSQL service containers](https://docs.github.com/en/actions/using-containerized-services/creating-postgresql-service-containers)