How to classify log status on DataDog
Tags: datadog, logs, status, level
Published: 2020-01-13 20:30:44

DataDog is a great tool for Data Analysis and Log Management. I started using it recently, and the tools they provide really impressed me.

First things first, you will need to collect your logs before monitoring them. There are many ways to do so, through preexisting [integrations](https://docs.datadoghq.com/integrations/) or by collecting them from [files](https://docs.datadoghq.com/agent/logs/?tab=tailexistingfiles).

By default, every log message is classified as `info`, but it can be changed to any [Severity Level](https://en.wikipedia.org/wiki/Syslog#Severity_level) described by syslog.

The possible log status are:
- Emergency
- Alert
- Critical
- Error
- Warning
- Notice
- Informational
- Debug

Once you have collected your logs, it's time to classify them.
To do so, you have to create a pipeline.
## Pipeliiines

Pipelines are workflows that hold one or more processors inside. Processors are predefined data manipulators that can be executed in line. For this post we are going to use 3 Processors:

- Grok Parser
- String Builder
- Status Remapper

### Creating a Pipeline

1. Go to Logs > Configuration on the sidebar menu
![Configuration Sidebar Image](https://thepracticaldev.s3.amazonaws.com/i/95dcur452hp183dnrzx6.png)

2. Click on "New Pipeline" at the upper corner of the page
![New Pipeline Button Image](https://thepracticaldev.s3.amazonaws.com/i/iha387wfba1mbpvv10vq.png)

3. Name your Pipeline
![Creating pipeline](https://thepracticaldev.s3.amazonaws.com/i/59nbc0ygg4qijbiclyuz.png)

4. Once you created the pipeline, you should be able to add processors:
![Create processor image](https://thepracticaldev.s3.amazonaws.com/i/z07h0zcp0qn94420edii.png)

### Parsing the logs
The next step is to parse the logs. For that purpose you can use the Grok Parser and extract information from your text. You can find more information about parsing rules by clicking [here](https://docs.datadoghq.com/logs/processing/parsing/).


Choose the Grok Parser as the processor.
Then, provide some log samples (you can get those on the [Data Dog Logs Page](https://app.datadoghq.com/logs)) and write your own [Parsing Rules](https://docs.datadoghq.com/logs/processing/parsing/?tab=matcher).

A Parsing Rule is defined by the following structure:
```
<Name of the Rule> Pattern %{matcher1:attribute}...

You can also use filters combined with matchers

<Name of the Rule> Pattern %{filter1} Pattern2 %{matcher1:attribute}
```
**[Read more](https://docs.datadoghq.com/logs/processing/parsing/?tab=matcher) about matchers and filters.**

Check out the Grok Parser configuration:
![Grok Parser Configuration](https://thepracticaldev.s3.amazonaws.com/i/766tmbea0420otn64hs8.png)

In most cases, just one rule should be enough. In the example there is just one rule called `ImmaRule` and we collect the attribute `timestamp` as a date and `username` as a word, by using [matchers](https://docs.datadoghq.com/logs/processing/parsing/?tab=matcher).

Once you extracted all the data you needed, a JSON representation of the attributes and values extracted through parsing you specified with the data samples will show up on the bottom of the screen:
![JSON Representation of the extracted data](https://thepracticaldev.s3.amazonaws.com/i/s0jo2s7aqlxvqx3szblh.png).
Name your processor and click save.

### Classifying the logs
The Grok Parser will keep forward only with the logs that matched the pattern you specified earlier. Since it's a pipeline, you can safely classify all logs filtered as errors, warnings, or any other status.

In this case I want to classify them as errors. For this purpose, we're adding a [String Builder](https://docs.datadoghq.com/logs/processing/processors/?tab=ui#string-builder-processor) processor to the pipeline beneath the Grok Parser.
This way we can store the log level in a new attribute and officialize it as the status attribute later by using the [Log Status Remapper](https://docs.datadoghq.com/logs/processing/processors/?tab=ui#log-status-remapper) processor.

So I'll add the String Builder processor and add the attribute `level` to it, with the value `error`:
![String builder processor](https://thepracticaldev.s3.amazonaws.com/i/73uyeu1ookd9aqz67r0w.png)

And finally set the new `level` attribute as the status attribute, by adding the [Log Status Remapper](https://docs.datadoghq.com/logs/processing/processors/?tab=ui#log-status-remapper) processor after the String Builder on the pipeline:
![Status remapper example](https://thepracticaldev.s3.amazonaws.com/i/ddy1uk0kkg408q40yx8i.png)

And finally, our pipeline is done!
![Finished Pipeline](https://thepracticaldev.s3.amazonaws.com/i/11bzb7ya5om3yvozttij.png)

Now you should get your logs on the [Log Explorer](https://app.datadoghq.com/logs) flagged with the `level` you chose previously
![Log explorer](https://thepracticaldev.s3.amazonaws.com/i/a0s7opodou8w8ot1c53k.png)

Thanks for reading.
I hope this tutorial to be somehow useful for you.