Setting up a single Splunk Forwarder to send different data to multiple indexes
Tags: splunk, beginners, tutorial, pfsense
Published: 2023-06-08 08:47:46

### Pre-requisites:

1. Have a working Splunk instance (Splunk Enterprise, in my case) to connect to. There are plenty of tuts for this online.
    
2. Have installed a universal forwarder on the endpoint that you want to monitor (see [here](https://hurricanelabs.com/splunk-tutorials/your-all-in-one-guide-to-setting-up-pfsense-and-suricata-in-splunk/), an excellent post which will get you most of the way through setting up Splunk to analyse Suricata & pfSense logs)
    

> **Note:** There are some steps specific to my use case, which are marked accordingly with **<mark>[OPT].</mark>** These can be ignored if just trying to configure a universal forwarder to send data to multiple indexes in Splunk.

### Steps:

1. Create the desired index in Splunk (Settings --&gt; Indexes). I named mine `ids_lan` as I am using an Intrusion Detection System (IDS) to monitor my LAN network on pfSense. You can leave all the index settings as default for now.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fwz5f1mtxvfvj27yyo8p.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nfznr0gbztt5h8c2qz32.png)


2. ***<mark>[OPT]</mark>*** Go to pfsense web UI, and create & configure the instance that you want to monitor. Once up and running, go to 'Logs View' and select the instance to view.
    
![](https://cdn.hashnode.com/res/hashnode/image/upload/v1674202427414/c05ed6d0-aba7-4be3-bd8b-c790429611cc.png align="center")

Note the file path and importantly, the folder name where logs are sent for that instance. Mine path is `/var/log/suricata/suricata_em125470/eve.json` and the folder name is `suricata_em125470`.

1. SSH into your VM/machine with the splunk forwarder installed and modify the `inputs.conf` file. In my case it's found in `/opt/splunkforwarder/etc/apps/TA-Suricata/default`, but that's because I'm using the [TA-Suricata](https://splunkbase.splunk.com/app/2760) app to make my Suricata logs Splunk-readable (matching Splunk's Common Information Model \[CIM\]).


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zm2hy8d2ia6lnzt7gzmk.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9jiok8fj6umryobitz4x.png)

In most cases, you should navigate to `/opt/splunkforwarder/etc/system/local`and create an `inputs.conf` file if there isn't already one. This overrides all of the defaults located in `/opt/splunkforwarder/etc/system/default`, and it's [best practice](https://docs.splunk.com/Documentation/Splunk/9.0.3/Data/Monitorfilesanddirectorieswithinputs.conf#:~:text=The%20inputs.,a%20stanza%20to%20the%20inputs.) to modify files within `/local` instead when you're not using an extra app like I am.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f7453qsy2tmrxahbulgw.png)

1. Within the `inputs.conf` file, create entries like so (as many as needed), making sure to reference the new folder that was created in step 3 (`suricata_em125470`).
Additionally, set `index = ids_lan` , or whatever index you created in Step 1.
    

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2oopvb2hawyezhvddkxi.png)



For a more general installation, here is some sample code (excluding suricata-specific options).

> **Code Dump:**
> 
> `\[monitor://<mark>path_to_your_monitored_file_here</mark>\]`
> 
> `disabled = false`
> 
> `index = <mark>your_index_from_step_1</mark>`
> 
> `host = <mark>your_splunk_instance_name</mark>`

*(Your splunk instance name is found by going to* ***Settings --&gt; Server Settings --&gt; General Settings*** *in the splunk web UI****)***

1. Navigate to the `/opt/splunkforwarder/bin/` directory and run the command `./splunk restart` to restart the splunk forwarder. **A reboot of the system is (likely) not sufficient.**
    

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/64ogn4btvhqukowrwgwf.png)

1. Upon restarting, within the same directory (`/opt/splunkforwarder/bin/`) check that Splunk is running with `./splunk status`. If it isn't, run `./splunk start` . Verify there are logs in the folder you just linked to (in my case, `suricata_em125470`), which can be done either by navigating to the path where they are stored, or in my use case, via *Log Contents* in the pfSense web UI (see step 3).
    
2. Wait a few minutes or so, and then run a splunk search for a timeframe for which you know there are log entries for, with something like `index="your_index_from_step_1"` . Check the time stamp matches when you expect the last log to have been from, and voila!


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z3plyzn1hjmed0g59iw3.png)

    
### 💛 Best of luck, I hope this helps some people and avoids too much keyboard-bashing! 💛