Hands On With AWS IoT, Timestream and Grafana
Tags: aws, iot
Published: 2021-03-15 06:44:57


Grafana has long been a favourite of ours for creating stunning visualisations. It's lightweight, easy to get started with and, over time, has added a lot of new features without losing simplicity. 

Yet, we have shied from recommending Grafana to our IoT customers over, say, the [TICK stack from Influx](https://www.influxdata.com/) or [ODFE/Kibana](https://opendistro.github.io/for-elasticsearch/). 

A typical IoT workflow has three main blocks on the cloud and a lot of product builders are thrown off by the complexity of integrating these while maintaining data security and access control.

![Typical Cloud IoT Blocks](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3ex3ciaf0dpzl2t9h9qy.png)

Of these, Grafana only does visualisation and is spectacularly good at it. You need to find your own solutions for data ingress and storage. Sure, Grafana has plugins for most data sources you have heard of and many more you haven't. Yet, most of these require you to set up your own servers. 

![Grafana Data Sources](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9bv4r7s3k7ldho3ngxij.png)


We previously explored integrating [AWS Timestream with Grafana](https://iotready.co/blog/metal-to-alerts-with-aws-iot-timestream-quicksight/#timestream--grafana). Back then, we used the managed service from Grafana.com. With AWS now offering managed services for [IoT ingress](https://aws.amazon.com/iot/), [time series data storage](https://aws.amazon.com/timestream/) and [Grafana](https://aws.amazon.com/grafana/), the integration and security challenges should largely be solved, right? Let's find out.

### Setting up Grafana Instance

Like the best managed offerings from AWS, setting up and configuring Grafana is a breeze. There's a wizard led flow once you click on "Create Workspace". 

![AWS Managed Grafana](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4npoi9oz7d86kohl6bj5.png)

![AWS Grafana Wizard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kkchbdo7bap17m5yxdmf.png)

AWS uses SSO for access control - this is powered by [AWS Organizations](https://aws.amazon.com/organizations/) behind the scenes. More about this in a bit.

![AWS Grafana Auth](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oy5qemd017vt9pis08gf.png)

IAM policies can be enabled with a click or you can configure these yourself. 

![AWS Grafana IAM](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fqmzmu0uuzjnayx20mtb.png)

Once we are done with the wizard, we enable the newly created user for access. In a couple of minutes our brand new Grafana instance is ready for use. We are brought to this friendly and useful summary.

![AWS Grafana Summary](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/auxtu5d56f81jmsutdba.png)

### Accessing & Configuring Grafana

While we were busy clicking Next a few times, AWS created a new Organisation, sent a confirmation email to the logged in IAM user and another invitation email to the SSO user we created. 

![AWS Grafana SSO Email](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i1w98pnzvp3c0jlq0tli.png)

Accepting the invitation allows us to set up a password for this user which we will need when accessing the Grafana instance.

![Grafana AWS SSO](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/539ku11elr9cqwd4qc7a.png)

Once we login, the interface is an AWS white-labeled version of the standard Grafana UI with a section for AWS specific data sources. 

![AWS Grafana Data Sources](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dpizqq1t038urunvejex.png)

As one would expect, finding and setting up our Timestream DB as the default data source is also a matter of a few clicks.

![AWS Grafana Timestream](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7gc7woy8xbxfaqzesnvz.png)

### Setting up Dashboards & Alerts

We have [previously described](https://iotready.co/blog/metal-to-alerts-with-aws-iot-timestream-quicksight/#timestream-db) how to send data to AWS IoT and from there into Timestream. Once that is configured, we send data using our trusty Python script.

![Sending IoT Data](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/laz2ol04kofabkj34o6c.png)

Setting up a dashboard and panel is also exactly the same as in the previous article and we end up with, surprise, a similar panel. 

![Grafana CPU Usage Panel](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pu89vtsk2ev5tqji6p2e.png)

The alerts workflow is the same as usual - with the exception that AWS SNS is the first option in notification channels. Once the alerts are configured, we see them on the Alerts dashboard within Grafana.

![Grafana Alerts Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y49j1ehwh07m35a9r2aq.png)

SNS is really easy to configure from within the [AWS Grafana Console](https://console.aws.amazon.com/grafana/) (where we saw the summary earlier). We will explore this integration in a future post.

## Reflections 

The advent of managed services from AWS (and others) has enabled cloud infrastructure to be configured rather than developed. For most IoT applications, a product builder can probably just pick one of the following configurations.

![IoT Cloud With ODFE](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/483z4oe9a13wka2lw5me.png)

![IoT Cloud With Timestream and Grafana](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3snkwbko8bg8eswyo2oi.png)

There are differences to consider, of course.

### Features

- ODFE has [anomaly detection built-in](https://iotready.co/blog/iot-anomaly-detection-open-distro-elasticsearch/)
- ODFE's authentication system supports multi-tenancy and granular data source level access controls out of the box. 
  - Grafana has viewer and editor roles and it's not obvious how to implement data source level access control.
- Grafana allows you to ingest and visualise data from multiple data sources.
- Grafana has a lot of visualisation plugins with a well documented flow for building new plugins.

### Pricing

- ODFE does not have a pay-as-you-go model - you pay for the hosted ElasticSearch instance(s). Expect to pay $40-$80 per month for <100,000 devices. There are no limits to the number of users you can add to the system.
- AWS Managed Grafana [costs $9/editor](https://aws.amazon.com/grafana/pricing/) and $5/viewer per month. The prices are similar to that for QuickSight.

### Ecosystem

- Timestream is relatively new and the data model takes a little getting used to. Moreover, you have to use official AWS SDKs to access data for custom applications.
- ElasticSearch/ODFE, being a lot more mature, have numerous SDKs to access data and build custom interfaces.


For these reasons, and to avoid vendor lock-in with Timestream, we will continue to recommend ODFE over Grafana. 

_Questions, comments? Connect with us at [hello@iotready.co](hello@iotready.co)._