How to download large amount of datadog logs in parallel
Published: 2023-06-17 13:12:12

## **How it started**

We had our application logs in datadog and there was a requirement to analyze large amount of logs for some delays in the application

### Issues we faced

1) Downloading logs directly from datadog has the max limit of 100k

2) If we want to calculate delay between 2 logs or from frontend to backend, there is no easy way of doing that

3)  If you want to download large amount of logs say for 1 week which has 1 million records, downloading that by writing simple application takes time

### The approach

When i started i have created a simple utility which downloads the logs synchronously, but the problem is it takes time to download i.e. 15 to 30 min, by downloading 5000k logs per sec. To reduce that time i have created the shell script which runs this above utility in parallel it solved that issue significantly but created another i.e. it created multiple csv files. Combining them is still a tedious task

### The Solution

I decided to write a simple tool to solve the above issues hence *dd-downloader* CLI tool. i have tried to explain the approach in below diagram

 


![datadog-downloader](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hmtsxli7j4tkojvybrf9.png)



### CLI tool


#### Step 1: Clone the repo

```
$ git clone https://github.com/girishg4t/dd-downloader.git

```

#### Step 2: Build binary using make

```
$ make

```

#### Step 3: Run the command

```
$ dd-downloader generate config --name=config.yaml # generates the sample yaml file with date range of 10min
$ dd-downloader validate --config-file=./sample_templates/event_sent.yaml # just validate if the mapping and template is correct
$ dd-downloader run sync --config-file=templates/queued_event.yaml --file=output.csv # will download logs one after the other in chucks of 5000
$ dd-downloader run parallel --config-file=templates/private_event.yaml --file=output.csv  # will run 10 parallel threads to reduce the time of download

```

### Prerequisite

You need to create the yaml config file as per [examples](https://github.com/girishg4t/dd-downloader/tree/master/sample_templates)

#### Things to keep in mind

```
auth:
- dd_api_key => need to specify datadog api key
- dd_app_key => need to specify datadog app key

```

more details are here [datadog](https://docs.datadoghq.com/account_management/api-app-keys/)

```
datadog_filter:
- mode => it can be `synchronous` or `parallel`, in `parallel` mode large time frame more than 10min is converted into 10 parallel chuck to reduce the time of download
- query => logs will be filtered based on this query, verify it in datadog before using
- from => from which date the logs need to be downloaded
- to => to which date

```

more details are here [datadog](https://docs.datadoghq.com/tracing/trace_explorer/query_syntax/)

```
mapping:
- field: This is used for header in csv file
- dd_field: datadog log field need to be mapped to above csv header, (check the logs in datadog and get the fields you want to map)
- inner_field: Since plane data can be mapped easily, however for mapping the Array you need to use this field

eg.
for below yaml mapping
field 'date' is taken from 'log.Attributes.Attributes' same for 'session_id'
for inner object we need to specify '.' and for array we need to specify '-'
in below log from datadog we need to map reqId which is inside the array of data
{
  "data": [
    {
      "event": {
        "snid": "dasgadsgasdgasd",
        "data": {
          "act": "ASDGASDGDDD",
          "srcId": "dsgdsgdgsdg",
          "dstid": "dasgdasdgdgdgg",
          "pid": "adgasdhsdhh",
          "quality": "DAG",
          "sid": "dddadahsdfhdfh"
        },
        "ets": 1686307199869,
        "etyp": "ASDGASD",
        "rqid": "AAAAA"
      },
      "reqId": "AAAAA"
    }
  ]
}

```

is mapped like this

```
- field: "-"
    dd_field: "data"
    inner_field:
    - field: "req_id"
        dd_field: "reqId"
    - field: "event_ts"
        dd_field: "event.ets"
    - field: "event_type"
        dd_field: "event.etyp"
    - field: "dest_id"
        dd_field: "event.data.dstid"
    - field: "source_id"
        dd_field: "event.data.srcId"

```

### Sample YAML file :

```
apiVersion: datadog/v1
kind: DataDog
spec:
  auth:
    dd_site: "datadoghq.com"
    dd_api_key: "xxxxxxxxxx"
    dd_app_key: "xxxxxxxxxx"
  datadog_filter:
    mode: synchronous
    query: 'service:frontend "socket: not able to connect to server" @type:SERVER_EVENT '
    from: 1686306900000
    to: 1686306960000
  mapping:
    - field: "date"
      dd_field: "date"
    - field: "session_id"
      dd_field: "session_id"
    - field: "-"
      dd_field: "data"
      inner_field:
        - field: "req_id"
          dd_field: "reqId"
        - field: "event_ts"
          dd_field: "event.ets"
        - field: "event_type"
          dd_field: "event.etyp"
        - field: "dest_id"
          dd_field: "event.data.dstid"
        - field: "source_id"
          dd_field: "event.data.srcId"

```