Splunk: Building a Secure Monitoring Solution (Part 1)
Tags: splunk, siem, cybersecurity, analyst
Published: 2023-06-08 03:54:06

During the last several weeks of my Cybersecurity boot camp, one of our final projects was to build a secure monitoring environment for a fictitious organization called VSI (Virtual Space Industries) using Splunk Enterprise, which for those who may not know, is a SIEM (Security Information and Event Manager). SIEMs are essential tools that companies can use to detect, analyze, and respond to potential threats against their organization.

Since this was a big project with a lot of steps, I'll be breaking it up into 2 parts:

Part 1:
Creating Reports, Alerts, and Dashboards for Windows server log data as well as Apache webserver log data that can help point out any abnormal activity.

Part 2:
Checking to see if the solutions created in part 1 were effective against a fictitious attack by uploading the attack log data and seeing if our reports, alerts, and dashboards, picked up anything that would have helped the organization take the appropriate action as quickly as possible.

Part 1
I started by launching Splunk, which had been pre-installed in my ubuntu VM. I logged into the application and uploaded the files I would be using to create reports, alerts and Dashboards for.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/efbw8o0lwguxl0tnuovt.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zi8movf9q3mjkpqgat5r.png)

Once the logs were uploaded, I briefly took notice of and analyzed the following fields:
o	signature
o	signature_id
o	user
o	status
o	severity

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gn3b0oenngpi0bt6863r.png)

Apologies that some images might be a bit small and difficult to read. Luckily, I also took some screenshots of the data inside each individual field as well:

signature:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h8bjgkquuxki7f9spr93.png)

signature_id:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mg70ydob3o91x6squkhk.png)

user:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t7jyxitqi6gj9rb2f8uu.png)

status:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9fwmev1eps3kvrmt20cv.png)

severity:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6vcybeta54xl6jqvf0n3.png)

These would be the main points of interest we will be using to create our reports, alerts, and dashboard. Let's start with the reports!

Report 1: A report with a table of signatures and their associated signature_id. This would allow VSI to view reports that show the ID number associated with each specific signature for a Windows activity.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v70j7susmgixuh8tlcwp.png)

Report 2: A report that displays severity levels and the count and percentage of each. This would allow VSI to quickly understand the severity levels of Windows logs being viewed.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rq3doyxpsc5221i4ynii.png)

Report 3: A report that provides a comparison between the success and failure of Windows activities. This would show VSI is there is any suspicious level of failed activities on their Windows server.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2y2l0yhutt1kw2h51kve.png)

Now that those were done, it was time to move onto creating the Alerts!

Alerts would all trigger an email to be sent to the fictitious company at SOC@VSI-company.com

Alert 1: An alert that is triggered when a threshold for hourly failed Windows activities has been reached. This would help VSI see if any failed logins or any other activities occurred an excessive amount of times within an hour, which could be indicative of someone trying to do something they shouldn't be able to do, such as trying to login and failing repeatedly.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c8sj1sq452nm5crlthtu.png)

The threshold for this alert I chose was > 18 per hour.

Alert 2: An alert that is triggered when a threshold has been reached for the amount of successful logins per hour.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hfo0ifuc4csmdz2k6ye8.png)

The threshold I chose for this alert was anything > 26 per hour.

Alert 3: An alert that is triggered when a threshold is met for the signature count when a user's account has been deleted, once again in an hourly window.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wczq9dv0nk96uhzgeu9v.png)

For this alert, I chose anything > 35 within one hour.

Now for the fun part, creating Dashboards to monitor Windows Server Activity at a quick glance!

I always have a lot of fun creating dashboards in Splunk, this project being no exception. I made:

1. A line chart that displays account deletion signatures over time within the span of 1h.
2. A line char that displays the different user field values over time.
3. A pie chart of the different signatures based on windows activities.
4. Another pie chart showing the different users who are active.
5. A final pie chart that tracks the source domain.

Line charts 1 & 2:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/90um9adzdngufsrnufz5.png)

Pie Charts:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7u4hoc3001sgnkivr95q.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/01othm94woqdb55263jr.png)

Full-view Windows Server Monitoring Dashboard:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8kf56rf6gktbc848jxqa.png)

Next, we needed to repeat this process, but for the Apache log data. I went ahead and uploaded the log files and got to work on the reports first.

This time, the important fields we wanted to pay special attention to were:
o	method
o	referrer_domain
o	status
o	clientip
o	useragent

Report 1: A report that shows a table of the different HTTP Methods (GET, POST, HEAD, etc.). This would show VSI the types of HTTP requests being made to the VSI webserver.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3nbwd94mg7dkuujmku61.png)

Report 2: A report that displays the top 10 domains that refer to VSI's website, to help VSI identify any suspicious referrers.

Note: I found it kind of funny that 'referer' was spelled incorrectly (should be 'referrer' in the fields list), but knew it had to match the data, so spelled it incorrectly to match when I had to.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rps75oivmin0vjfq4xx9.png)

Report 3: A report that shows the count of each HTTP response code. This will help VSI to quickly gauge the overall health of their webserver and activities taking place on it.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hlu4wn9v0okszq6y8nr2.png)

With that, it was time to create some Alerts!

Alert 1: The project called for an alert that triggers whenever a connection is made from any IP address outside of the United States. I chose France for this example, which had an IP Address of 176.31.39.30 (Roubaix, France).

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qbjpr880yg6qit8csnyd.png)

Alert 2: An alert that triggers whenever a threshold was met for the count of HTTP POST methods within the span of 1 Hour.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/91mvelmfyt1v0c4vist6.png)

The threshold I chose for HTTP POST requests was anything > 10 within 1 hour.

And now back to the fun of creating visuals for our Dashboard!

HTTP GET Method requests per hour:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/759rltkxf7sxu31vc4gt.png)

HTTP POST Method requests per hour:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8s9she5tza69i8i7g7my.png)

HTTP Methods by type per hour:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hcko34pwrw6sw9xwev87.png)

Top Countries connecting to the server:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qao1l64ngohqwyu2f38e.png)

Top User agents:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m5mmb7zhfd7tvs477mcx.png)

Top URI:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dl6s2zkm3tp1u4thqv3f.png)

Full-view Apache Server Monitoring Dashboard:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iw94emh678vw9vo7a93c.png)

So there you have it! I created Reports, Alerts, and Monitoring Dashboards for VSI's Windows and Apache servers. 

In Part 2 we will see whether or not the solutions I made protected VSI.

Part 2: https://dev.to/r33keeper/splunk-building-a-secure-monitoring-solution-part-2-208m