How to integrate Datadog Agent in ECS Fargate
Tags: sre, observability, datadog, terraform
Published: 2022-05-26 11:41:00

## Introduction
The other day it was the first time that I set up Datadog Agent in ECS on Fargate. I had a hard time understanding it even after reading [the documentation](https://docs.datadoghq.com/integrations/ecs_fargate/?tab=fluentbitandfirelens), so I'm writing it down here.


### Notes
* As the doc mentioned, if you wanna know how to set up EKS on Fargate, please be careful that the doc is different. Look at this [document](https://docs.datadoghq.com/integrations/eks_fargate/).
* Modify your container to the side-car container, not create another container for the setup.
  * I'd misunderstood the first time that I had to create another container.
* By integrating the Agent, you can get some metrics from ECS Fargate.
  * I'll eventually post about how to get the logs!


### What the problem
First of all, [the document](https://docs.datadoghq.com/integrations/ecs_fargate/?tab=fluentbitandfirelens) only shows how to create it via WebUI, AWS CLI, or CloudFormation, which is confusing because I used Terraform when I created ECS.

I mean, I wanted to install Datadog Agent on a Fargate already built on Terraform, so I had no idea what to do at first.
But in this case, it is actually just a matter of modifying the task definition that already exists.


### How to set up
Here is the my sample task definition, so I mean I only modify this file.

```json
{
    "containerDefinitions": [
        {
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "hoge-hoge-hoge-dev-cwlg-api-server",
                    "awslogs-region": "ap-northeast-1",
                    "awslogs-stream-prefix": "/ecs"
                }
            },
            "portMappings": [
                {
                    "protocol": "tcp",
                    "containerPort": 8080
                }
            ],
            "command": [
                "sh",
                "-c",
                "npm run prisma:migrate:deploy && npm start"
            ],
            "environment": [
                {
                    "name": "SERVER_PORT_NUMBER",
                    "value": "8080"
                },
                {
                    "name": "AWS_S3_REGION",
                    "value": "ap-northeast-1"
                },
                {
                    "name": "TZ",
                    "value": "Asia/Tokyo"
                },
                {
                    "name": "SLACK_TOKEN",
                    "value": "hoge-99999969-60614-hoge-hoge-mEWS1wNZ0E765VkUvkM7"
                }
            ],
            "secrets": [
                {
                    "name": "DATABASE_URL",
                    "valueFrom": "/hoge-study-dev/api-secret-hoge/HOGE_DATABASE_URL"
                },
                {
                    "name": "IMPORT_PROBLEM_END_POINT_SECRET",
                    "valueFrom": "/hoge-study-dev/api-secret-hoge/HOGE_END_POINT_SECRET"
                },
                {
                    "name": "COGNITO_USER_POOL_ID",
                    "valueFrom": "/hoge-study-dev/api-secret-hoge/HOGE_USER_POOL_ID"
                }
            ],
            "essential": true,
            "pseudoTerminal": true,
            "readonlyRootFilesystem": false,
            "privileged": false,
            "name": "app"
        }
    ],
    "memory": "2048",
    "cpu": "1024",
    "executionRoleArn": "arn:aws:iam::99999999999:role/hoge-dev-role-api-server-exec",
    "taskRoleArn": "arn:aws:iam::99999999999:role/hoge-dev-role-api-server-task",
    "family": "hoge-dev-task-api-server",
    "networkMode": "awsvpc",
    "requiresCompatibilities": ["FARGATE"]
}
```


It's a sample task def for Datadog agent. 
```json
{
    "image": "datadog/agent:latest",
    "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
            "awslogs-group": "/hoge/hoge/datadog-agent-dev",
            "awslogs-region": "ap-northeast-1",
            "awslogs-stream-prefix": "ecs"
        }
    },
    "cpu": 10,
    "memory": 256,
    "mountPoints": [],
    "environment": [
        {
            "name": "ECS_FARGATE",
            "value": "true"
        },
        {
            "name": "DD_PROCESS_AGENT_ENABLED",
            "value": "true"
        },
        {
            "name": "DD_DOGSTATSD_NON_LOCAL_TRAFFIC",
            "value": "true"
        }
    ],
    "secrets": [
        {
            "name": "DD_API_KEY",
            "valueFrom": "/hoge-ssm/DATADOG_API_KEY"
        }
    ],
    "name": "api-server-datadog-agent"
}
```

### What Params 
|Key| Values| features| |
|:----|:----|:----|:----|
|image| datadog/agent:latest| Container Image| |
|environment: ECS_FARGATE| true| Set Agent ECS Fargate explicitly| |
|environment: DD_PROCESS_AGENT_ENABLED| true|Enabled live process collect| |
|environment: DD_DOGSTATSD_NON_LOCAL_TRAFFIC| true| Correct Metrics by DogStatsD on UDP8125| |
|secrets: DD_API_KEY | /hoge-ssm/DATADOG_API_KEY | Your Datadog API KEY | |
|name | api-server-datadog-agent | My container name | |


If you don't have the Datadog API key, you can get the organization page, let's please check the [docs](https://docs.datadoghq.com/account_management/api-app-keys/)
![Datadog API key](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i1621oncvethy6l866sw.png)



Also, I put the small technique in that example, so I use SSM parameter store to avoid hard cording DD_API_KEY.
If you are interested in the tech, please look at [the doc](https://docs.aws.amazon.com/AmazonECS/latest/userguide/specifying-sensitive-data-parameters.html).
![SSM param](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e3g3bk9hlmhfrli8st7i.png)
 


Finally, Completed the file💡
```json
{
    "containerDefinitions": [
        {
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "hoge-hoge-hoge-dev-cwlg-api-server",
                    "awslogs-region": "ap-northeast-1",
                    "awslogs-stream-prefix": "/ecs"
                }
            },
            "portMappings": [
                {
                    "protocol": "tcp",
                    "containerPort": 8080
                }
            ],
            "command": [
                "sh",
                "-c",
                "npm run prisma:migrate:deploy && npm start"
            ],
            "environment": [
                {
                    "name": "SERVER_PORT_NUMBER",
                    "value": "8080"
                },
                {
                    "name": "AWS_S3_REGION",
                    "value": "ap-northeast-1"
                },
                {
                    "name": "TZ",
                    "value": "Asia/Tokyo"
                },
                {
                    "name": "SLACK_TOKEN",
                    "value": "hoge-99999969-60614-hoge-hoge-mEWS1wNZ0E765VkUvkM7"
                }
            ],
            "secrets": [
                {
                    "name": "DATABASE_URL",
                    "valueFrom": "/hoge-study-dev/api-secret-hoge/HOGE_DATABASE_URL"
                },
                {
                    "name": "IMPORT_PROBLEM_END_POINT_SECRET",
                    "valueFrom": "/hoge-study-dev/api-secret-hoge/HOGE_END_POINT_SECRET"
                },
                {
                    "name": "COGNITO_USER_POOL_ID",
                    "valueFrom": "/hoge-study-dev/api-secret-hoge/HOGE_USER_POOL_ID"
                }
            ],
            "essential": true,
            "pseudoTerminal": true,
            "readonlyRootFilesystem": false,
            "privileged": false,
            "name": "app"
        },
        ,
        {
            "image": "datadog/agent:latest",
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "/hoge/hoge/datadog-agent-dev",
                    "awslogs-region": "ap-northeast-1",
                    "awslogs-stream-prefix": "ecs"
                }
            },
            "cpu": 10,
            "memory": 256,
            "mountPoints": [],
            "environment": [
                {
                    "name": "ECS_FARGATE",
                    "value": "true"
                },
                {
                    "name": "DD_PROCESS_AGENT_ENABLED",
                    "value": "true"
                },
                {
                    "name": "DD_DOGSTATSD_NON_LOCAL_TRAFFIC",
                    "value": "true"
                }
            ],
            "secrets": [
                {
                    "name": "DD_API_KEY",
                    "valueFrom": "/hoge-dev/api-secret-hoge/DATADOG_API_KEY"
                }
            ],
            "name": "api-server-datadog-agent"
        }
    ],
    "memory": "2048",
    "cpu": "1024",
    "executionRoleArn": "arn:aws:iam::99999999999:role/hoge-dev-role-api-server-exec",
    "taskRoleArn": "arn:aws:iam::99999999999:role/hoge-dev-role-api-server-task",
    "family": "hoge-dev-task-api-server",
    "networkMode": "awsvpc",
    "requiresCompatibilities": ["FARGATE"]
}
```