Custom App Logging with Azure Monitor for Containers
Tags: azure, kubernetes, aks, syslog
Published: 2020-06-23 20:47:03

## Table of Contents
- [Introduction](#introduction)
  - [Our Scenario](#our-scenario)
  - [Objective](#objective)
  - [Solution](#solution)
- [Implementation](#implementation)
  - [Prerequisites](#prerequisites)
  - [Step 1 - Invoking syslog in a PHP application](#step-1---invoking-syslog-in-a-php-application)
  - [Step 2 - Dockerfile configurations](#step-2---dockerfile-configurations)
  - [Step 3 - Docker CLI commands](#step-3---docker-cli-commands)
  - [Step 4 - Deploy to AKS](#step-4---deploy-to-aks)
- [In Closing](#in-closing)
- [Resources](#books-resources)
- [Acknowledgements](#clap-acknowledgements)

## Introduction

If you use Azure, you probably spend time in Azure Monitor - Azure's _comprehensive solution for collecting, analyzing, and acting on telemetry from your cloud and on-premises environments_.

And, if you happen to run containerized workloads in Azure - say with Azure Kubernetes Service (AKS) or Azure Container instances (ACI) - you may already be tapping into **Azure Monitor for containers**. If you are new to this feature, in short...

> **Azure Monitor for containers** gives you performance visibility by collecting memory and processor metrics from controllers, nodes, and containers that are available in Kubernetes through the Metrics API. Container logs are also collected.

For emphasis, I'll now repeat that last sentence...

> **Container logs are also collected.**

Azure Monitor for containers collects stdout and stderr from container workloads deployed to AKS (or ACI). 

Knowing this, all one needs to do is route custom application logs to stderr (or stdout) to take advantage of Azure Monitor for containers.

In this post, I will walk you through an example implementation using a simple php application and Rsyslog, the **r**ocket-fast **sys**tem for **log** processing.

### Our Scenario

We have a containerized PHP application running in Azure Kubernetes Service (AKS). In our application, we leverage the `syslog` php function to generate custom log messages, which in turn are consumed by a user defined log handler.

### Objective

Collect the log messages in Azure Monitor for containers for real-time observation and analysis in a Log Analytics workspace.

Below are screen captures from Azure Portal that show the expected custom log output in:
- The Live Data (preview) feature (Figure 1) 
- A Log Analytics workspace (Figure 2)
- The Logs displayed for our container running in Azure Container Instances (Figure 3)

**Figure 1**

![Screen capture of Live Data logging in AKS](https://dev-to-uploads.s3.amazonaws.com/i/c0ppj7lht5r9t577mqhc.png)

**Figure 2**

![Screen capture of custom app log in a Log Analytics workspace](https://dev-to-uploads.s3.amazonaws.com/i/650l73vobj0j3izkjjbg.png)

**Figure 3**

![Screen capture of container logs in Azure Container Instances](https://dev-to-uploads.s3.amazonaws.com/i/e3e1zjdafz4qoy6cdf5o.png)

### Solution

In our container image we install the `rsyslog` library;  configure a user defined `syslog` log handler; and route the messages to `stderr`.

Syslog recognizes the following severity levels for log messages:

| Numerical Code | Severity | Description |
|---|---|---|
| 0 | emerg | system is unusable |
| 1 | alert | action must be taken immediately |
| 2 | crit | critical conditions |
| 3 | error | error conditions |
| 4 | warning | warning conditions |
| 5 | notice | normal but significant condition |
| 6 | info | informational messages |
| 7 | debug | debug-level messages |

In our example implementation, we will evaluate for two severity levels: error (3) and info (6).

The implementation steps below are taken from this GitHub repo to which I contributed {% github snp-technologies/AKS-Custom-App-Log-Demo no-readme %}

## Implementation

### Prerequisites

This article assumes experience with Docker containers, Kubernetes and Azure. This article does not describe the steps to install an AKS cluster. Links to a Quickstart tutorial is provided in [Resources](#books-resources).

This article uses a simple PHP application for the implementation. If you're not a PHP programmer, don't despair - the principals covered here are generally applicable to other languages.

Please note that our image operating system is from the image  `debian:stretch-slim`.
 
### Step 1 - Invoking syslog in a PHP application

The following code example is a simple, single page application I spun up for a "custom application logging with syslog demo". For the demo, 
query strings are used to trigger the log handler, such as:

```shell
?value=0 // Triggers an Error log as a result of division by zero.
?value=2 // Triggers an Information log.
```
The application code:

```php
<?php
echo '<H1>Syslog Demo</H1>';

// Fetch the URL query string and assign to a variable. 
$qstring = $_SERVER['QUERY_STRING'];

parse_str($qstring, $output);

$value = $output['value'];

// For an error evaluation, we divide 1 by $value.
// A value of 0 will throw a division by zero error.
function inverse($x) {
    if (!$x) {
        throw new Exception('Division by zero.');
    }
    return 1/$x;
}

// Get formatted datetime to include in messages.
$access = date("Y/m/d H:i:s");

try {
    echo "<p>Inverse of value " , $value , " is " , inverse($value) , "</p>";
    $message = "Inverse of value succeeded";
    $severity = LOG_INFO;
    $logtext = "INFORMATION: at " . $access . "\n" . $message . "\n" . $_SERVER['REMOTE_ADDR'];
    _log($severity, $logtext);
    
} catch (Exception $e) {
    $message = $e->getMessage();
    echo '<p>Caught exception: ',  $message, "</p>";
    $severity = LOG_ERR;
    $logtext = "ERROR: at " . $access . "\n" . $message . "\n" . $_SERVER['REMOTE_ADDR'];
    _log($severity, $logtext);
}

// Open and close connection of system logger.
function _log($priority, $text) {
  openlog("myApp", LOG_PID | LOG_PERROR, LOG_LOCAL0);  
  syslog($priority, $text);
  closelog();
}

?>
``` 
### Step 2 - Dockerfile configurations

To enable syslog support in our container image, we install the `rsyslog` library:

```Docker
RUN apt-get update; \
	apt-get install -y --no-install-recommends \
	rsyslog \
	;
```
Later in our Dockerfile, we configure a custom log handler for our application logs:

```Docker
RUN echo "local0.* /var/log/apache2/myapp.log" >> /etc/rsyslog.conf
	
```
We then configure the log to go to stderr using a symlink, and we re-set ownership of the `/var/log/apache2/` just as first set in our base image `php:7.3-apache-stretch`:
```Docker
RUN \ 
  ln -sfT /dev/stderr "/var/log/apache2/myapp.log"; \
  chown -R --no-dereference "www-data:www-data" "/var/log/apache2/"
```
### Step 3 - Docker CLI commands

Build a container image:
```shell
docker build -t applogdemo .
```
Test it locally:
```shell
docker run --name applogdemo --rm -i -t applogdemo
```
The log output local test results should look similar to:
```shell
[ ok ] Starting enhanced syslogd: rsyslogd.
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
[Sat Jun 20 20:34:48.662731 2020] [mpm_prefork:notice] [pid 40] AH00163: Apache/2.4.25 (Debian) PHP/7.3.13 configured -- resuming normal operations
[Sat Jun 20 20:34:48.662809 2020] [core:notice] [pid 40] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND'
```
While the container is running, in your browser enter some URL's to invoke the log handler, such as: 
http://localhost/?value=0, http://localhost/?value=2, etc.

You should see additional log output similar to:
```shell
myApp[41]: ERROR: at 2020/06/29 14:18:15
myApp[41]: Division by zero.
myApp[41]: 172.17.0.1
172.17.0.2:80 172.17.0.1 - - [29/Jun/2020:14:18:15 +0000] "GET /?value=0 HTTP/1.1" 200 366 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:7
7.0) Gecko/20100101 Firefox/77.0"
"-" - - [29/Jun/2020:14:18:15 +0000] "GET /?value=0 HTTP/1.1" 200 366 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 F
irefox/77.0"
myApp[42]: INFORMATION: at 2020/06/29 14:18:30
myApp[42]: Inverse of value succeeded
myApp[42]: 172.17.0.1
172.17.0.2:80 172.17.0.1 - - [29/Jun/2020:14:18:30 +0000] "GET /?value=4 HTTP/1.1" 200 268 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:7
7.0) Gecko/20100101 Firefox/77.0"
"-" - - [29/Jun/2020:14:18:30 +0000] "GET /?value=4 HTTP/1.1" 200 268 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0"
```
Now that we have validated the application locally, let's prep the image for publishing to our preferred container registry, Azure Container Registry (Docker Hub or other registry is fine). Let's tag the image:
```shell
docker tag applogdemo myacr.azurecr.io/applogdemo:v1
```
Let's push to our preferred container registry.
```shell
docker push myacr.azurecr.io/applogdemo:v1
```
### Step 4 - Deploy to AKS

To deploy the container, we use the following Kubernetes YAML manifest.

Note the inclusion of a Service resource.

> If you are interested in deploying to Azure Container Instances, please refer to the Quickstart tutorial link provided in [Resources](#books-resources).

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: applogdemo
  labels:
    app: applogdemo
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: applogdemo
  template:
    metadata:
      labels:
        app: applogdemo
    spec:
      containers:
        - image: myacr.azurecr.io/applogdemo:v1 // Placeholder
          name: applogdemo
          ports:
          - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: applogdemo-service
  namespace: default
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
  selector:
    app: applogdemo
  type: LoadBalancer
```
Customize the `image:` value in the `containers:` spec if the `deployment.yml` manifest, e.g.
```YAML
containers:
    -
        image: myacr.azurecr.io/applogdemo:v1
```
Finally, deploy the kubernetes manifests using the commands:
   ```
   kubectl apply -f manifests/deployment.yml
   ```

### Step 5 - Validate the deployment
Validate the deployment by accessing the website via the IP Address exposed by the Kubernetes LoadBalancer service. To identify the IP Address, use the command:
```
$ kubectl get svc drupal-service
NAME             TYPE           CLUSTER-IP   EXTERNAL-IP      PORT(S)        AGE
drupal-service   LoadBalancer   10.2.0.50    52.151.xxx.xxx   80:32758/TCP   10d
```

Test the application just as done for local testing.
Once you have called a sequence of URL's with the `?value=<some integer>` query string go to the Azure Monitor for containers blade in Azure Portal to view your custom application logs.

## In Closing

In this article we have covered the basic configurations necessary to enable custom application logging for Azure Monitor for containers.

From a log analysis standpoint, we've only scratched the  surface in this post. Explore the Kusto query language to craft a wide variety of reports. Create alerts based on log metrics. Pin and share custom log dashboards.

Please feel free to share your questions and comments in the discussion below. Thanks - Mike :smiley:

## :books: Resources

- [Azure Monitor for containers overview](https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-overview)
- [Getting started with Kusto](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/concepts/)
- [Quickstart: Deploy an Azure Kubernetes Service cluster using the Azure CLI](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough)
- [Quickstart: Deploy a container instance in Azure using the Azure CLI](https://docs.microsoft.com/en-us/azure/container-instances/container-instances-quickstart)
- [Rsyslog](https://github.com/rsyslog/rsyslog)
- [PHP syslog function](https://www.php.net/manual/en/function.syslog.php)
- [Log rotation with rsyslog](https://www.rsyslog.com/doc/v8-stable/tutorials/log_rotation_fix_size.html)
- [Docker image code for php:7.3-apache-stretch](https://github.com/docker-library/php/blob/master/7.3/stretch/apache/Dockerfile)

## :clap: Acknowledgements

I would like to thank [Firoz Shaik](https://github.com/firozshk) for his review of this post and insightful recommendations.