How to Build Log Data Analytics using Apache Spark?
Published: 2020-09-11 11:41:26

---
title: How to Build Log Data Analytics using Apache Spark?
published: true
description: Apache Spark is used to analyze distributed data and big data processing. Here we have explained how to build Log Data Analytics using the Apache Spark framework.
tags: 
//cover_image: ![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/hroipjdg4q3rwpwczfjs.jpg)
---

<p align="justify">We are going to see how to process log data in Spark in this blog. Let us understand the log structure first and write a regular expression to match the log pattern and then extract the value and do <strong><a href="https://www.aegissofttech.com/en-au/big-data-analytics-consulting.html">some data analytics services</a></strong> on the log data.</p>
<p align="justify">First, let us see how the log data we are going to process looks like and understand it.</p>
<h2>Sample log data:</h2>

<div class="highlight">
<pre class="highlight javascript">
<code><span class="kd">
83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET 

/presentations/logstash-monitorama-2013/images/kibana-search.png 

HTTP/1.1" 200 203023 

"http://semicomplete.com/presentations/logstash-monitorama-2013/" 

"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
</code></pre></div>
Now let us break this <a href="https://www.quora.com/What-is-log-data">log data</a> and understand what each of them represents:
<div class="highlight">
<pre class="highlight javascript">
<code><span class="kd">
83.149.9.216 – IPAddress
-            - ClientId
-            - userId
[17/May/2015:10:05:03 +0000] – dateTime
GET – Method
/presentations/logstash-monitorama-2013/images/kibana-search.png 
– endpoint
HTTP/1.1 – protocol
200 – responseCode
203023 – contentSize
"http://semicomplete.com/presentations/logstash-monitorama-2013/" – URL
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 

(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36" – browser
</code></pre></div>
Now let us write code to process this log data using Apache Spark Databricks environment.

I have a log file located in the below location.
<a href='https://postimages.org/' target='_blank'><img src='https://i.postimg.cc/yYqJ4MX5/1.jpg' border='0' alt='1'/></a>
Now let us read few lines of this log data using Spark and see how it looks like.
 <a href='https://postimages.org/' target='_blank'><img src='https://i.postimg.cc/T1JTXDQk/2.jpg' border='0' alt='2'/></a>
<a href='https://postimages.org/' target='_blank'><img src='https://i.postimg.cc/rFhLCcHt/3.jpg' border='0' alt='3'/></a><br /><a href='https://gasstation-nearme.com/'>closest garage to me</a><br />
Now let us write a case class to have all the fields we have in the logs and later we will <a href="http://alvinalexander.com/scala/analyzing-apache-access-logs-files-spark-scala/">create RDD of type Access Log</a>.
 <a href='https://postimages.org/' target='_blank'><img src='https://i.postimg.cc/RVhYmtPN/4.jpg' border='0' alt='4'/></a>
Now lets us create a Pattern that matches the log data using a regular expression like below.
 
<a href='https://postimages.org/' target='_blank'><img src='https://i.postimg.cc/DzT5S4cQ/5.jpg' border='0' alt='5'/></a>
Now let’s write a Scala function that parses the log data and creates an object of case class that we can use to create an RDD.
 
<a href='https://postimages.org/' target='_blank'><img src='https://i.postimg.cc/RZQKbscf/6.jpg' border='0' alt='6'/></a>
Now let us read this log data and parse it to create RDD[AccessLog].
 
<a href='https://postimages.org/' target='_blank'><img src='https://i.postimg.cc/nhYPZt8h/7.jpg' border='0' alt='7'/></a>
Now let us find out the top 10 response codes with the counts in descending order.