Routing container logs to AWS CloudWatch on docker-machine generic servers
Published: 2021-01-09 12:51:31

## Why

My never ending quest is to increase the scale (and therefore profit) of my passive income project, and it is often difficult to determine what is the next greatest obstacle to scale. 

I'd experienced Cloudwatch before briefly and knew that it would be a useful tool to identify areas of the system where performance could be improved.  

## Context

The system is dockerised and runs on multiple machines, including rented dedicated servers in a European datacenter (they were cheap but powerful) and my own physical machines located in my garage in Australia.

I realised that it was difficult to figure out how well the system was actually performing because of a lack of indicators derived from the containers.

Inspecting logs of one container at a single point in time was not very insightful because there was so much noisy detail and not enough context about how other containers were performing and what the performance was like over a larger time period.  

I had experienced the power of CloudWatch logs for the first time in November, when I ran an experiment using AWS ECS to run the containers.  Although the compute cost on ECS proved to be prohibative it was my first exposure to CloudWatch, where I was impressed at the ability to collate the logs across all running instances - something I used mostly for debugging purposes. 

After shutting down the ECS experiment I felt like the system had taken a step backwards in terms of visibility into running services and I knew I had to get a better logging system in place.

## What did I need to do?

I read about the awslogs driver that comes as part of the core docker engine. The docs for the driver gave me hope that I could start piping stdout and stderr to my AWS CloudWatch account without having to make any drastic changes.  In theory all I needed to do was add some configuration to the docker-compose file and make sure the docker engine is able to find some credentials for AWS that have the right permissions for pushing log events and creating log streams.

## How?

My plan was to first update the docker-compose file then try to redeploy the app and see what errors occur.

On the first try docker reported that it couldn't find the AWS credentials. 

I had mistakenly thought that setting the AWS_* environment variables in my laptop's bash profile was the correct way to pass the creds to docker.  Unfortunately I had gotten the wrong `docker` - I didn't need to pass the creds to the docker client app on my laptop, instead I needed to make the creds available to the docker daemon running on the server. 

Unfortunately it took a long time to come to this realisation.

In order to make the AWS creds available to the docker daemon on the servers, I had to update the systemd config on _each of my machines_ with the AWS creds before I was able to successfully start a container using the awslogs log driver.

```
echo "Environment=AWS_ACCESS_KEY_ID=<redacted>" >> /etc/systemd/system/docker.service.d/10-machine.conf && \
echo "Environment=AWS_SECRET_ACCESS_KEY=<redacted>" >> /etc/systemd/system/docker.service.d/10-machine.conf && \
systemctl daemon-reload && \
systemctl restart docker && \
```

After updating each machine I was able to redeploy the containers using docker-compose, and I could see the logs appearing in CloudWatch straight away.

## The result

All 30 running containers now send their stdout and stderr logs to my CloudWatch dashboard, where I can search, filter and aggregate metrics from them.

I can tell already that CloudWatch is going to be a great tool in the quest of increasing scale, helping me to identify and improve the bottlenecks of the system and to generate more profit.