Collect the log data using Kinesis Agent with delivery stream using Kinesis Data Firehose and Querying the log data using Athena
Tags: awss3, awsathena, aws, awskinesisdatafirehose
Published: 2021-10-22 16:25:09

“Problems faced while collecting the log data in ec2 manually and finding a solution for easily querying the logs of the database in AWS". I have tried to find out certain solutions from a security and cost perspective. Installed and configured kinesis agent for streaming of logs. And also used Amazon athena for querying log data. Created a data delivery stream for source as direct put and destination as Amazon S3. In terms of security perspective, Amazon kinesis data firehose provides an encryption option and s3 bucket also provides encryption. In terms of cost perspective, the logs ingested are having a cost in kinesis data firehose and s3 bucket charges. Athena only needs to pay for queries you run(i.e.; charged for the amount of data scanned).   

Amazon Kinesis Data Firehose is a fully managed service for delivering real-time [streaming data](http://aws.amazon.com/streaming-data/) to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon OpenSearch Service, Splunk and any custom HTTP endpoint or HTTP endpoints owned by supported third-party service providers, including Datadog, Dynatrace, LogicMonitor, MongoDB, New Relic, and Sumo Logic. Kinesis Data Firehose is part of the kinesis streaming data platform, along with [Kinesis Data Streams](https://docs.aws.amazon.com/kinesis/latest/dev/), [Kinesis Video Streams](https://docs.aws.amazon.com/kinesisvideostreams/latest/dg/), and [Amazon Kinesis Data Analytics](https://docs.aws.amazon.com/kinesisanalytics/latest/dev/). With Kinesis Data Firehose, you don’t need to write applications or manage resources. You configure your data producers to send data to Kinesis Data Firehose, and it automatically delivers the data to the destination that you specified. You can also configure Kinesis Data Firehose to transform your data before delivering it. 

Amazon Athena is an interactive query service that makes it easy to analyze data directly in Amazon Simple Storage Service (Amazon S3) using standard [SQL](https://docs.aws.amazon.com/athena/latest/ug/ddl-sql-reference.html). With a few actions in the AWS Management Console, you can point Athena at your data stored in Amazon S3 and begin using standard SQL to run ad-hoc queries and get results in seconds. Athena is serverless, so there is no infrastructure to set up or manage, and you pay only for the queries you run. Athena scales automatically - running queries in parallel - so results are fast, even with large datasets and complex queries.

In this post, you will get to know how to collect the log data using Kinesis Agent and Querying the log data using Athena. Here I have used a ec2 server for collecting the streaming logs data using kinesis agent and used athena service for querying logs.

#Prerequisites
You’ll need an Amazon EC2 Server for this post. [Getting started with amazon EC2](https://aws.amazon.com/ec2/getting-started/) provides instructions on how to launch an EC2 Server.

#Architecture Overview
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0ibbgdcgxsoc3xv3gjg1.png)
The architecture diagram shows the overall deployment architecture with data flow, application server, Amazon Kinesis Data Firehose, Amazon Simple Storage Service and Amazon Athena.

#Solution overview
The blog post consists of the following phases:
1. Create a data delivery stream using Amazon Kinesis service as Kinesis Data Firehose
2. Installing and configuring the Kinesis Agent Application
3. Querying the log data using Amazon Athena

I have a EC2 server as below →
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ydkj55ko5tndylac4rcg.png)

##Phase 1: Create a data delivery stream using Amazon Kinesis service as Kinesis Data Firehose
1. Open the Amazon Kinesis service Console and select a kinesis data firehose.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pw0u7d8m9xex84bks5a4.png)
2. Create a delivery stream, select source as Direct PUT and destination as Amazon S3.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u1fy5h54oidvqhrcb3hd.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0grip6jjkuub8yrd98qq.png)
3. Give a delivery stream name as “log-delivery-stream”. Also create a s3 bucket named as “log-delivery-bucket” with default options. In destination settings, choose the s3 bucket created and give s3 bucket prefix as “logs/” and s3 bucket error output prefix as “errors/”. 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dc1geec0e82co71vp7fd.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d68hzofff6d7m2br4d4j.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x4wawhpes9oo5dzfvmxw.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0aame2d0a71m15k8sdlh.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pvvhk9r1idc13nhwjv5l.png)
4. Set buffer interval as “60 seconds”. And choose create or update IAM role as default defined. Leave all other settings as default only or can change it as per our requirement. And choose a create delivery stream option. Once the stream is created, the status changes to the active state of the delivery stream.   
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8gvazc918jn889o3ys1g.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g9f4c5i2v5250m548ao2.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/out01xp7gob7fs3kk0mq.png)
5. You can view the IAM role policy permissions defined for the creation of the delivery stream.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/47dbgs5yt5e9shuvo00y.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dv6epl9w6qdebb6ilkhv.png)

##Phase 2: Installing and configuring the Kinesis Agent Application.
1. Login the ec2 server and run the command to install the streaming data agent. Create a script for cloudwatch metrics, firehose endpoint and flow including {file pattern and delivery stream}. Run the script and start the aws-kinesis-agent service.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f2l1wvup6g9x4i7wnpl7.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/12sbnlbep0fuu6vzklry.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pnzce5y9md6swvzyypls.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n753dhbm2ht2wv3dmpgm.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/80ewvxzjp6ro94c2lua5.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f4iwvhpk5yrxdklyspif.png)
2. Create another script for adding logs lines in a file named as /tmp/app.log and can check the lines added in the log file. You can check the logs of the kinesis agent using tail command.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9ghqy3da965msatqqyhu.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u6npzw5jo36wixjqjtpq.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4nwtyc5wuja68o5dzdc4.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dbvgwf377ssbua89611z.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nm7lqvy29og4cuos6v8v.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gcllwq8b6zzoqg3rtncl.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uxa6ue3a87cgd9omxuhr.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zemc4k2lyemo8qok323m.png)

##Phase 3: Querying the log data using Amazon Athena 
1. Open the Amazon Athena console and create a folder named as “results” in the bucket. Goto settings option on athena console and select the s3 location of the folder you have created and then save it.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s0bqoqp3x7wbuh5w262w.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ac73ext9bzuhd86nc51s.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j9mw04fxjqnfa4slrvdu.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dz0mvlk67enq2uhwb4md.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/94dj0hnay7ctszbwy2jy.png)
2. Click on connect data source, choose data located as “Query data in Amazon S3” and metadata catalog as “AWS Glue Data Catalog”. In the next page: choose parameter “AWS Glue Data Catalog in this account” and “create a table using the Athena table wizard”. Click on continue to add table...
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6gh6r2wg9gc3n1rdcr8g.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/58k329ddfdo2tdaev19u.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4jfa16aehdo7q8xu1poo.png)
3. Choose to create a new database option.Give the database name and table name. Create a logs folder in s3 bucket and select the bucket location in the input data set option. Then add the data format and regex. Also add the column name and column type and create a table. 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/psje7fd7haipp02o3g3o.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x5ikc277f9fl2mk46vi1.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fa0e63b41uwjs27kf3h9.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lu061oqhdt5kct5jwg82.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3piiovgctu40sh6dg7dy.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mpp7hwt3im6fc8hu4jvb.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hua91a8jsx061i0qvl6r.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zag15p33rrji3idm90nm.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vkqnmtvcz76ngxelj5dn.png)
4. Results of query for database and tables created in athena. And then you are able to save the query on the athena console by “save as” option. You are also able to view the saved queries, history[query submitted time, query engine version and soon..]. You can create a workgroup based on users, teams, workloads and applications… In S3 bucket you can view the logs of the query run. 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/eubbw86m7nlodnqq172t.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ctcz2nhm9cr0q5rw9uvv.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ri6ylmlwldvtkmwngnqm.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ygcza45kwhla3xsa52rs.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tpzd9fyexbtkn99w7ed0.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ojcpw3ano3ba519hrbtk.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/irhdc45zls1m4znr4nfm.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2zx4jl9dvk0uhame041q.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ht84trpelmzkril1ntu6.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h8ea5wepzgwrm3xk1hot.png)

#Clean-up
Terminate EC2 Server.
Delete Amazon Kinesis Data Firehose - Delivery Stream.
Delete the S3 bucket.

#Pricing
I review the pricing and estimated cost of this example. 
Cost of EC2 = $0.0124 per hour = $0.01. 
Cost of Amazon Kinesis Data Firehose = $0.034 per GB of data ingested = $0.31(9.210 GB).
Cost of S3 bucket = $0.0
Cost of Athena = $0.0
Total Cost = $(0.01+0.31+0.0+0.0) = $0.32

#Summary
In this post, I had shown you how to collect the log data using Kinesis Agent and Querying the log data using Athena.
For more details on Amazon Kinesis Data Firehose, Checkout Get started Amazon Kinesis Data Firehose, open the [Amazon Kinesis Data Firehose console](https://ap-south-1.console.aws.amazon.com/kinesis/home?region=ap-south-1#/home). To learn more, read the [Amazon Kinesis Data Firehose documentation](https://docs.aws.amazon.com/kinesis/?id=docs_gateway). For more details on Amazon Athena, Checkout Get started Amazon Athena, open the [Amazon Athena console](https://ap-south-1.console.aws.amazon.com/athena/home?region=ap-south-1#query). To learn more, read the [Amazon Athena documentation](https://docs.aws.amazon.com/athena/?id=docs_gateway).

Thanks for reading!

Connect with me: [Linkedin](https://www.linkedin.com/in/gargee-bhatnagar-6b7223114)