Analytics using Kinesis Data Firehose 📊
Tags: aws, react, analytics, javascript
Published: 2021-11-23 20:41:45

If you haven't started implementing analytics in your application yet, you're behind the game. The most valuable asset in modern era is; Data, and this will grow in the future as well

So what is Analytics? Through Analytics you can get feedback from your clients without asking them, you can collect data, see how your users are navigating, performing, visiting etc... After collecting this data, you can analyze, target more useful features, and update your UI hence making it more user friendly with your analyzed data

Basically after you implement analytics your application will become more mature, convenient for users, and more successful for your business 👨🏻‍🚀

In this Blog we will create an AWS Kinesis Data Firehose that delivers stream to S3 bucket (using AWS CDK with typescript), and put our records from our React app using Amplify

_Amazon Kinesis Data Firehose is the easiest way to reliably load streaming data into data lakes, data stores, and analytics services_

Let us split the content into two parts, first the Infrastructure then the frontend

### 1- Infrastructure
We are going to use CDK to provision our resources, it helps us to write our code using languages like typescript. I will focus on how to create the S3 bucket & Firehose, and how to give permission to our Unauthenticated users to use it, whereas I highly suggest to go deep in how to create Cognito as well, but since our scope isn't about authentication I wont be focusing on that. You can check the official CDK documentation via [this link](https://docs.aws.amazon.com/cdk/api/latest/)

Resource Diagram:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mfrrm3j02xjt91icc4kt.png) 

CDK stack:
```javascript
const bucket = new s3.Bucket(this, bucketName, {
  bucketName,
});

const result = bucket.addToResourcePolicy(new iam.PolicyStatement({
  actions: ['s3:GetObject'],
  resources: [bucket.arnForObjects('file.txt')],
  principals: [new iam.AccountRootPrincipal()],
}));

const s3Destination = new destinations.S3Bucket(bucket, {
  dataOutputPrefix,
  errorOutputPrefix,
  bufferingInterval: Duration.minutes(bufferingInterval),
  bufferingSize: Size.mebibytes(bufferingSize),
});

const stream = new firehose.DeliveryStream(this, 'Delivery Stream', {
  deliveryStreamName: streamName,
  destinations: [s3Destination],
});
```
```javascript
bucketName: 'your-bucket-name',
streamName: 'your-stream-name',
dataOutputPrefix: 'myFirehose/DeliveredYear=!{timestamp:yyyy}/anyMonth/rand=!{firehose:random-string}',
errorOutputPrefix: 'myFirehoseFailures/!{firehose:error-output-type}/!{timestamp:yyyy}/anyMonth/!{timestamp:dd}',
bufferingInterval: 2,
bufferingSize: 8,
```
- First we create our bucket, then we add the resource policy

Our destination has the following parameters
- **dataOutputPrefix** for files successfully delivered to S3
- **errorOutputPrefix** for failed records before writing them to S3
- By default, the buffer size **(bufferingSize)** is 5 MiB and the buffer interval is 5 minutes **(bufferingInterval)**. But in our example I changed them to 2 minutes and 8 MiB

_Incoming data is buffered before it is delivered to the specified destination. The delivery stream will wait until the amount of incoming data has exceeded some threshold (the "buffer size") or until the time since the last data delivery occurred exceeds some threshold (the "buffer interval"), whichever happens first._

Now regarding our Cognito, and how to be able to put records from frontend, we will add this policy to our Authenticated & Unauthenticated users, this is flexible you can choose which users to be able to send records, in our case I will add for both
```javascript
{
  effect: iam.Effect.ALLOW,
  actions: [
    'firehose:PutRecord',
    'firehose:PutRecordBatch'
  ],
  resources: ['your-firehose-arn']
}
```

### 2- React App
Amplify is a set of purpose-built tools and services that makes it quick and easy to use our AWS resources, I will be using Analytics part here, for official documentation you can visit [this link](https://docs.amplify.aws/)

- First let us see how we can configure our Analytics

App.js
```javascript
import { Analytics, AWSKinesisFirehoseProvider } from 'aws-amplify';

Amplify.configure(awsConfig);
Analytics.addPluggable(new AWSKinesisFirehoseProvider);
```
- **awsConfig** is a json file that holds our configuration, inside it add this object
```javascript
Analytics: {
  AWSKinesisFirehose: {
    region: REGION
  }
}
```

- Now lets add our fancy Button
```javascript
const onClick = async () => {
  const now = new Date;
  const data = {
    id: now.getTime(),
    action: 'Add Button',
    component: 'Button',
    user: 'the username of user',
    source: 'Web',
  };

  try {
    await Analytics.record({
      data: data,
      streamName: FIREHOSE,
    }, 'AWSKinesisFirehose');
  } catch (error) {
    console.log(error);
  }
}

<button
  onClick={onClick}
>Add Button</button>
```

- **FIREHOSE** is our Kinesis firehose name that we created previously from the CDK
- **Analytics.record** is where we put our record to the Firehose, after 2 minutes we can view it from our S3 bucket, our data is simple I added some attributes just to show how it can scale, and you can send very useful data for Analytics or even do ML on it later

Finally our result inside our S3 bucket (the file should be downloaded)
```json
{"id":1637695583710,"action":"Add Button","component":"Button","user":"the username of user","source":"Web"}
```

In this example, I tried to limit the content scope to our topic, I used a simple button component, but this can go beyond your expectations & creativity, you can collect very useful and beneficial data from your users, which would assist in enhancing & improving of your applications 🙂