Part 3: Adding a database
Tags: javascript, webdev, vue, tutorial
Published: 2022-05-05 19:53:22

> The full code for this part is in branch `part-3`, found [here](https://github.com/Evertvdw/creating-a-embeddable-chat-widget/tree/part-3)

In the previous parts we setup our project folders and connected everything together. In this part we will focus on the `server` package mostly, by adding a database to persist data when the server restarts or is updated.

## What database to use?
I went back and forth on this one, as the technology you use has to match your goals for the project. I primarily wanted something simple, easy to deploy and with little extra setup necessary for development. 

In the end I want to be able to host the whole project (server/portal/widget) in a single VM somewhere, without having to worry about external connections, databases and such. With that in mind I was looking at some sort of in-memory database with persistence to a local file of some sorts which would be loaded back in upon restarts/updates. 

I did want something performant so that I would (hopefully) not run into issues when there are around 100-ish clients connected at the same time. I looked at [`low-db`](https://github.com/typicode/lowdb) for a while but did not like that it would `JSON.stringify` my whole database on every change, which could become a problem when it becomes to big.

So instead I went with [`lokijs`](https://www.npmjs.com/package/lokijs) even though I find the documentation of it quite horrible, I have used it before and it works without issues and has lots of features. And I thought it would also be good to show how I use it so others don't have to figure that our on their own 😇

### Don't agree with me?
That is of course perfectly fine! In the end it does not matter what database technology you use, feel free to implement it using MongoDB or Firebase whatever you are comfortable with. The only thing that you would need to change is the database initialization and how to save/update/fetch something out of there.

## Let's code!
> The changes in this section are summarized in [this](https://github.com/Evertvdw/creating-a-embeddable-chat-widget/commit/6d89c32f683e74f2396f3612e5bcfada333052f9) commit

To keep things separated I will put all database related stuff inside the `/packages/server/database` folder. As the `/packages/server/admins.ts` we use seed into our database logically belong there, I moved that file into that folder, changing the top line to: `import { Admin } from './../types';`.

### Installing lokijs
To install lokijs run the following commands:
```bash
yarn workspace server add lokijs
yarn workspace server add -D @types/lokijs
```

### Initializing the database
I create a `packages/server/database/database.ts` file with the following:
```ts
import { join } from 'path';
import adminSeed from './admins';
import loki from 'lokijs';
import { Admin, Client, Database } from '../types';
const lsfa = require('lokijs/src/loki-fs-structured-adapter');

export default function initDB() {
  return new Promise<Database>((resolve) => {
    const adapter = new lsfa();
    const db = new loki(join(__dirname, './server.db'), {
      adapter,
      autoload: true,
      autosave: true,
      autosaveInterval: 4000,
      autoloadCallback: () => {
        db.removeCollection('admins');
        const admins = db.addCollection<Admin>('admins', {
          autoupdate: true,
        });
        adminSeed.forEach((admin) => {
          admins.insertOne(admin);
        });
        let clients = db.getCollection<Client>('clients');
        if (clients === null) {
          clients = db.addCollection<Client>('clients', {
            autoupdate: true,
            indices: ['id'],
          });
        }
        resolve({ admins, clients });
      },
    });
  });
}
```
Quick breakdown of what is happening:
1. Lokijs uses so called `adapters` to handle the persistence to files. We use the fastest and most scalable adapter called the `fs-structured-adapter`. You can read more about it [here](https://github.com/techfort/LokiJS/wiki/LokiJS-persistence-and-adapters#overview)
2. We export a `initDB` function which will setup the database and return a promise, resolving when it is done.
3. Inside the setup we provide some seed data to our database, we repopulate the admins from our seed file every time. Also we check if a collection for our clients exist, and if not, we create one. Collections are logically separated parts of the database, which also get persisted inside their own file.
4. On both collections we use the `autoupdate` setting, that will persist changes made to the collection automatically. By default you would have to call `.update()` manually to make sure the data in memory is also saved to file.

Inside our `.gitignore` file we have to add `/packages/server/database/*.db*` to make sure our created database files are ignored by git.

### Updating `packages/server/index.ts`
Now we have to use our just created `initDB` function inside our main entry file. First remove the current `database` initialization:
- `const db: Database ...` ❌
- `import admins from ./admins;` ❌

And add `import initDB from './database/database';` at the top somewhere.

Replace the `server.listen` call with:
```ts
let db: Database;
(async function () {
  try {
    db = await initDB();
    server.listen(5000, () => {
      console.log(
        `Server started on port ${5000} at ${new Date().toLocaleString()}`
      );
    });
  } catch (err) {
    console.log('Server failed to start.');
    console.error(err);
  }
})();
```
Which is our new initialize function that will startup the server once the database is setup.

At this point typescript is probably complaining that the `Database` type is no longer correct. Lets change `packages/server/types.ts`:
- add `import { Collection } from 'lokijs';` at the top 
- update the interface to:

```ts
export interface Database {
  clients: Collection<Client>;
  admins: Collection<Admin>;
}
```

### Update handlers
Our code in the `packages/server/handlers` still expects a plain object as a database, we have to update some code inside `adminHandler` and `clientHandler` to use our new database properly:
- Instead of `.find((admin) => admin.name === name)` we can now use `.findOne({name})`
- When we want to send all items of a collection we have to to `db.clients.find()` instead of just `db.clients`
- When adding a new client we use `.insert` instead of `.push`.

There is one gotcha when adding a new message to the clients `messages` array. As `lokijs` uses `Object.observe` on the whole client to determine if something needs updating. This does not work for array mutations (common Vue2 reactivity caveat as well, that got me quite a few times😅). So whenever we add a message we have to update manually by adding `db.clients.update(client);` afterwards.

## Store the client session
> Commit found [here](https://github.com/Evertvdw/creating-a-embeddable-chat-widget/commit/01b5a8c512b8bbbe1a755dacb03453257b013f49)

When a client connects now it will generate a new random name, and when that client refreshes it's browser window it will create a new client. This is of course not really feasible, we have to store the clients session somewhere, and if that same client reconnects we restore that session.

### Generate a random id for clients at the server
Inside `packages/server/index.ts` we add the following

```ts
// Socket middleware to set a clientID
const randomId = () => crypto.randomBytes(8).toString('hex');
io.use((socket, next) => {
  const clientID = socket.handshake.auth.clientID;
  if (clientID) {
    const client = db.clients.findOne({ id: clientID });
    if (client) {
      socket.clientID = clientID;
      return next();
    }
  }
  socket.clientID = randomId();
  next();
});
```

and add `import crypto from 'crypto';` at the top.

This is a piece of middleware that will run for every client that connects to our server. It will check a `auth` object on the handshake that the socket server does with the client, if the a clientID is present there we set that clientID on the socket object. If not it is a new client and we generate a new random ID. 

As we use typescript and we set a `clientID` property on the socket object it does not know about we have to add that to the type of `socket`.

To do that we add to `packages/server/types.ts`:
```ts
declare module 'socket.io' {
  interface Socket {
    clientID: string;
  }
}
```

### Using the new clientID inside the clientHandler
Inside the `packages/server/handlers/clientHandler.ts` we currently define the client and add it to the database. We have to check here if our client already exists and only add a new entry to the database if necessary.

Remove:
- `const client: Client = ...` ❌
- `db.clients.insert(client);` ❌

And add:

```ts
socket.join(socket.clientID);

socket.emit('client:id', socket.clientID);

let client: Client;
const DBClient = db.clients.findOne({ id: socket.clientID });
if (DBClient) {
  client = DBClient;
  client.connected = true;
  socket.emit('client:messages', client.messages);
} else {
  client = {
    ...data,
    messages: [],
    id: socket.clientID,
    connected: true,
  };
  db.clients.insert(client);
}
```
Breakdown:
1. By default socket.io will create a user id store it a `socket.id` and join a room with that specific id. We now have to join the `socket.cliendID` room as we define our clientID manually. 
2. We emit the clientID to the client so that it can store that in localStorage and send it along when reconnecting.
3. We check if the client exists, and if it does not we create and insert that client into the database.
4. If a client is already in the database we send the message history to the client.

Inside the same file we also have to update our `disconnect` event listener as we have to change the logic that determines if a client is connected. In theory we could have one client opening multiple tabs, which will each establish it's own socket connection. If one of them closes we need to check if there aren't any connections left open for that client before we update the connection status.

Change the `socket.on('disconnect')` handler to:
```ts
socket.on('disconnect', async () => {
  const matchingSockets = await io.in(socket.clientID).allSockets();
  const isDisconnected = matchingSockets.size === 0;
  if (isDisconnected) {
    client.connected = false;
    io.to('admins').emit('admin:client_status', {
      id: client.id,
      status: false,
    });
  }
});
```

### Save the clientID at the widget side
In the widget we will store the id and name of the client in the socket store, and generate a new name there if it is the first time connecting. 

Inside the `packages/widget/src/stores/socket.ts` we add to our state:
```ts
id: localStorage.getItem('clientID'),
name: localStorage.getItem('clientName') || '',
```
and to our actions:
```ts
SOCKET_messages(payload: Message[]) {
  this.messages = payload;
},
SOCKET_id(payload: string) {
  localStorage.setItem('clientID', payload);
  this.id = payload;
},
setName() {
  const name = faker.name.firstName();
  this.name = name;
  localStorage.setItem('clientName', name);
},
```
Also add `import faker from '@faker-js/faker/locale/en';` at the top of the file, and remove it from `packages/widget/src/App.vue`;

Now we have to use the name and id from the store when connecting to the socket server, change `const socket = io(URL);` to:
```ts
const socket = io(URL, {
  auth: {
    clientID: socketStore.id,
  },
});
watch(
  () => socketStore.id,
  (val) => {
    socket.auth = {
      clientID: val,
    };
  }
);
if (!socketStore.name) {
  socketStore.setName();
}
```
The watcher here is needed when for some reason the server disconnects (for a restart for example) and the socket connection is reset. In that case the socket will reconnect with the correct clientID provided. 

In the `addClient` object change `name` to `name: socketStore.name` and add `watch` to the list of imports from 'vue'. 

## Handle reconnects at the portal side
> Commit found [here](https://github.com/Evertvdw/creating-a-embeddable-chat-widget/commit/0700f213f2e3cccf4d79dbd052078d4d7437baa4)

The last thing we take care of in this part is handling reconnecting the portal to the server when the server is restarted or otherwise loses it's connection. In the portal currently we only call `admin:add` when we boot up our application. If the socket connection is lost and restored we have to call `admin:add` again to register the correct handlers on that socket.

Inside `packages/portal/src/boot/socket.ts` we change the `admin:add` call to:
```ts
// This will be called on the initial connection and also on reconnects
socket.on('connect', () => {
  socket.emit('admin:add', 'Evert');
});
```

We have to do the same inside our widget inside `packages/widget/src/App.vue` change `client:add` to:
```ts
// This will be called on the initial connection and also on reconnects
socket.on('connect', () => {
  socket.emit('client:add', addClient);
});
```

### Fix a small bug in the portal
There is a bug in the portal code that happens when the server reboots and the socket is reconnected. Even if we re-emit the `admin:add` event, if we already have a client selected, we cannot send new messages to that selected client. That is because when we reconnect we resend the whole client list and in the `SOCKET_list` action inside `packages/portal/src/stores/client.ts` we replace the `clients` array in the state with the newly received value.

However, if we already had a client selected, the `clientSelected` state pointed to an item in the old array we overwrote. So to keep things working we have to reset the `clientSelected` in there as well:
```ts
if (this.clientSelected) {
  const currentSelectedId = this.clientSelected.id;
  this.clientSelected =
    this.clients.find((client) => client.id === currentSelectedId) ||
    null;
}
```

## Wrapping up
That's it for this one! In the next part I will be adding a login page to the portal and generate a token to also secure the connection from the portal to the server. See you next time! 🚀