Tutorial part 4: key-value store - data streams
Tags: serverless, node, database, iot
Published: 2023-02-24 10:07:33

Welcome to this **part-4** of the [key-value database tutorial](https://codehooks.io/docs/tutorials/key-val-store/part-1) for [codehooks.io](https://codehooks.io) serverless [Node.js](https://nodejs.org) backend. In this part we'll focus on how to work with multiple key-value pairs in a [key-value database](https://en.wikipedia.org/wiki/Key%E2%80%93value_database).

The Codehooks.io key-value database stores all keys as a sorted string index. This is a very useful feature if you need to store and retrieve multiple values together, as a group or as a segment of key-values.

This article is also avaliable at [the official codehooks.io website](https://codehooks.io/docs/tutorials/key-val-store/part-4-working-with-multiple-values-and-streams)

## IoT device - time series data example

To illustrate this feature we'll create an example use-case for capturing data from multiple [IoT devices](https://en.wikipedia.org/wiki/Internet_of_things). In this example, each IoT device sends its location, identification and observations to a specific [REST API](https://en.wikipedia.org/wiki/Representational_state_transfer) network endpoint.


For example, a temperature sensor device in the "Alpha region West" reports its value each minute. And a wind force sensor in the "Beta region West" reports its values hourly, and so on.

To ensure that we can access and process the observation data sets effectively, we'll create a key string combination that captures all necessary information in one unique key-value pair. 

To illustrate this concept lets imagine a series of IoT device observations as shown in the example below. 

```bash
'/Alpha/West/density/2023-01-09T10:14:00.495Z'  => '4.4191'
'/Alpha/West/humidity/2023-01-09T17:01:00.821Z' => '0.7539'
'/Alpha/West/temp/2023-01-09T21:15:00.450Z'     => '87.100'
'/Beta/West/wind/2023-01-19T21:57:00.316Z'      => '5.9477'
...
```

Now that we understand the concept for our IoT database, lets begin the development of an example backend API for IoT sensors.

We'll create a serverless REST API endpoint to receive observations from many IoT devices. The API will store each observation with a logic key that effectively creates a [time series database](https://en.wikipedia.org/wiki/Time_series_database) with data points that we can explore and use in our application. 

Lets get started with creating the first API to receive data observations from a device.

## REST API endpoint to POST a single observation

The REST API route takes in 3 parameters for a given device, region, location and device. 
And the POST body must contain the actual device observation data point.
For example:
```bash
POST /observation/North/Plaza/humid03 

{"observation": "4.02"}
```
The route parameters are combined into a unique key with a time stamp which gives a correct time series for the data points. The following table shows some example device observations. 

| IoT devices time series | Data points  |
|---|---|
| /East/BuilingA/temp01/2023-02-22T18:30:14.441Z  | 21.4  |
| /North/Plaza01/humid3/2023-02-23T05:23:11.306Z  | 12.5  |
| /East/BuilingB/temp02/2023-02-22T18:32:38.991Z  | 19.0  |


The serverless function below implements a REST API endpoint route that can receive data from a device and insert it to the key-value database.


```js
app.post('/observation/:region/:location/:device', async (req, res) => {
    const {region, location, device} = req.params;
    const {observation} = req.body;    
    const key = `/${region}/${location}/${device}/${new Date().toISOString()}`;
    // string key e.g. '/North/Plaza/humid03/2023-02-23T05:23:11.306Z'
    const conn = await Datastore.open();
    const result = await conn.set(key, observation);
    console.log(result)
    res.status(201).end('Data received ok');
})
```

Test the REST API with curl.

```bash
curl -X POST \
  'https://<YOUR_DATABASE_ID>.api.codehooks.io/dev/observation/East/BuilingA/temp01' \
  --header 'x-apikey: <YOUR_API_TOKEN>' \
  --header 'Content-Type: application/json' \
  --data-raw '{
  "observation": "21.4"
}'
```


> Use the CLI command [`coho info --examples`](https://codehooks.io/docs/cli#info) to inspect your project API endpoint and Curl examples.

## Inspecting the key-value database with the CLI

The [Codehooks CLI](https://codehooks.io/docs/cli#get) can be used to manage the key-value database basic operations, `get`, `set` and `del`. For example, to retrieve all observations for the `East` locations you can run the following command:

```bash
coho get '/East*'
```
This will list all matching keys in the key-value database as the example output shows below.

```bash
{
  key: '/East/BuilingA/temp01/2023-02-23T05:48:19.646Z',
  value: '21.4'
}
{
  key: '/East/BuilingA/temp01/2023-02-23T05:49:06.614Z',
  value: '20.0'
}

2 keys
 ```


## Endpoint to GET observations subkey

Similiar to the CLI example above, we can use the [database API](https://codehooks.io/docs/key-value-database-api) `getAll` method to retrieve all key-value pairs that match a start segment of a particular key. In this example we are producing [CSV data format](https://en.wikipedia.org/wiki/Comma-separated_values) for easy spreadsheet analysis etc.

```js
app.get('/observations/:prefix', async (req, res) => {
    const conn = await Datastore.open();
    const {prefix} = req.params;
    let count = 0;
    res.set('content-type', 'text/csv');
    res.write('Device, Data\n');
    const stream = conn.getAll(`/${prefix}`);
    stream.on('data', (data) => {
        const outString = `"${data.key}", ${data.val}\n`;
        count++
        res.write(outString);
    }).on('end', () => {
        res.end();        
    })
})
```

Testing the API with curl again.

```bash
curl -X GET \
  'https://<YOUR_DATABASE_ID>.api.codehooks.io/dev/observations/Alpha' \
  --header 'x-apikey: <YOUR_API_TOKEN>' \
```

```csv
Device, Data
"/Alpha/Center/decibel/2023-01-09T11:11:31.218Z", 2.3428
"/Alpha/Center/decibel/2023-01-09T11:12:01.050Z", 6.0632
"/Alpha/Center/decibel/2023-01-09T13:13:30.541Z", 0.7196
"/Alpha/Center/decibel/2023-01-09T15:23:00.589Z", 9.7232
"/Alpha/Center/decibel/2023-01-09T15:34:00.520Z", 5.0089
"/Alpha/Center/decibel/2023-01-09T17:04:00.942Z", 9.1861
```

## Complete source code

The complete source code for our IoT backend application is shown below.

```js
// index.js
import { app, Datastore } from 'codehooks-js' // Standard JS lib for express style code

// list a serie of IoT device observations
app.get('/observations/:prefix', async (req, res) => {
    const conn = await Datastore.open();
    const {prefix} = req.params;
    let count = 0;
    res.set('content-type', 'text/csv');
    res.write('Device, Data\n');
    const stream = conn.getAll(`/${prefix}`);
    stream.on('data', (data) => {
        const outString = `"${data.key}", ${data.val}\n`;
        count++
        res.write(outString);
    }).on('end', () => {
        res.write(`Result: ${count}`);
        res.end();        
    })
})

// register a new IoT device observation
app.post('/observation/:region/:location/:device', async (req, res) => {
    const {region, location, device} = req.params;
    const {observation} = req.body;
    // string key e.g. '/North/Plaza/humid03/2023-02-23T05:23:11.306Z'
    const key = `/${region}/${location}/${device}/${new Date().toISOString()}`;
    const conn = await Datastore.open();
    const result = await conn.set(key, observation);
    console.log(result)
    res.status(201).end('Data received ok');
})

export default app.init(); // Bind functions to the serverless cloud
```

This part-4 of the key-value store tutorial has shown how you can combine smart keys with the database streaming capabilities to create kind of IoT time series database with [codehooks.io](https://codehooks.io).

In our next tutorial [Part-5: Managing data with TTL options(https://codehooks.io/docs/tutorials/key-val-store/part-5-managing-data-with-ttl-options), we'll explore how you can use the time-to-live (TTL) feature to delete _old_ data in the key-value database.