Parameter Store Organization of Cloud Watch Agent Configs
Tags: aws, cloud, cloudwatch, tutorial
Published: 2023-05-31 22:44:45

One best practice with CloudWatch agent on local instances is to utilize SSM Parameter Store to organize CloudWatch agent configurations. This allows configurations to be more reusable across multiple instances. One consideration is the instance will need access to the SSM service through either an internet gateway, a NAT gateway, or [a VPC Endpoint](https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-create-vpc.html#sysman-setting-up-vpc-create). This guide works through the process on Linux systems but should be easily modified to work on Windows as well.

## IAM Profile Setup

Before CloudWatch agent can work with logs some setup is required. First an EC2 Instance Profile will need to be setup which allows for posting logs to CloudWatch. It will also need to allow reading from the appropriate parameter store location. The IAM managed policy `CloudWatchAgentServerPolicy` can handle this or can be used as a reference to create a more strict policy. As of writing the policy looks like:

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutMetricData",
                "ec2:DescribeVolumes",
                "ec2:DescribeTags",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams",
                "logs:DescribeLogGroups",
                "logs:CreateLogStream",
                "logs:CreateLogGroup"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetParameter"
            ],
            "Resource": "arn:aws:ssm:*:*:parameter/AmazonCloudWatch-*"
        }
    ]
}
```

As the last statement indicates, only SSM parameters that start with "AmazonCloudWatch-" will be readable by the role. Other naming conventions will require a custom policy.If configs are encrypted via SecureString allowing KMS decrypt will also be required:

```json
{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": [
      "kms:Decrypt"
    ],
    "Resource": "arn:aws:kms:*:111122223333:key/MyKeyHere"
  }
}
```

For CloudWatch agents to be managed via SSM, the AWS managed policy ` AmazonSSMManagedInstanceCore` will provide the necessary permissions.

## Cloud Watch Agent Installation

Some instances will also require CloudWatch agent installed. The [AWS documentation has several methods for this](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-on-EC2-Instance.html), including using Systems Manager. Command line installations will also need to have the appropriate [user data](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html) defined or utilize a custom image. If this seems like too much work Amazon 2 AMIs come with the agent installed by default. 

## Cloud Watch Agent Basic Configuration

Depending on installation methods some CloudWatch agents may not have a config file setup already. Assuming a default installation of `/opt/aws/` the CloudWatch agent configuration script can generate one by default:

```bash
# sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
```

Not that as is the system does not have permissions to push the config to parameter store. Permissions can either be added or simply copy the contents of the output file to parameter store manually (making sure that it follows the `AmazonCloudWatch-` prefix convention to work with the managed policy). 

## Working With Multiple Configurations

Normal Linux installations will have a directory `/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d` that can hold multiple configuration files. The `agent` level should generally be part of the main config. Then app specific settings can be declared by declaring the appropriate top level key down to the specific setting. As an example:

```json
{
    "logs": {
        "logs_collected": {
            "files": {
                "collect_list": [
                    {
                        "file_path": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
                        "log_group_name": "LogsTest",
                        "log_stream_name": "amazon-cloudwatch-agent.log",
                        "timezone": "UTC"
                    }
                ]
            }
        }
    }
}
```

This will combine the core configuration with a new substructure of logs which defines an application specific logging format. After a uniquely named file is placed in the `amazon-cloudwatch-agent.d` directory the CloudWatch agent will need to be restarted.

## Working With Parameter Store

Technically you can simply use the AWS CLI to get the parameter store contents and then put it in the file. However, AWS provides a tool to simplify this process: `amazon-cloudwatch-agent-ctl`. To begin, take the JSON above and put it in parameter store. You will need to either create the "LogsTest" log group or update it to another log group of choosing. Be sure the parameter store entry name starts with a `AmazonCloudWatch-` prefix. If you're worried about security use SecureString to encrypt (again, you will need to add the appropriate KMS decrypt permissions if you do). Once this is done, simply run:

```bash
$ /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -s -c ssm:AmazonCloudWatch-CWAgent
```

Be sure to replace `AmazonCloudWatch-CWAgent` with the name of your SSM parameter. Note that despite being in the same product, secrets manager will not work here, only parameter store entries. If the run is successful an entry prefixed with `ssm_` should be in the `/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d`. In the event something goes wrong removal of the file along with another `append-config` should work until the issue is addressed. 

Alternatively if you're using SSM to manage instances the `AmazonCloudWatch-ManageAgent` run command allows you to add configurations as well. This will once again require the appropriate network access to reach the SSM service and `AmazonSSMManagedInstanceCore` or similar policy to work with the document.

## Conclusion

Assuming everything was setup properly the `amazon-cloudwatch-agent.lg` stream should appear under the appropriate log group and log data should be pushed. Be sure to look over the [metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html#CloudWatch-Agent-Configuration-File-Metricssection) and [logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html#CloudWatch-Agent-Configuration-File-Logssection) sections to see how things are laid out when customizing your configurations. I hope this article has helped those looking to centralize their configurations in parameter store to make them more reusable.