How to Monitor your Azure Kubernetes Cluster
Tags: monitoring, azure, kubernetes, cluster
Published: 2021-02-24 16:22:42

<p>Once you have your critical application running on a Kubernetes cluster, it is important to monitor its performance metrics in order to sustain stable operation. Without enough visibility, you will never be able to identify resource bottlenecks or determine the maximum load that the cluster can sustain. This article will only cover monitoring, not alerting. </p>
<p>I looked at three different approaches, specific to monitoring Azure Kubernetes Services workloads:</p>
<ul class="ak-ul" data-indent-level="1">
<li><a class="sc-dqBHgY hiPvaM" title="https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-overview" href="https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-overview" target="_blank" rel="noopener noreferrer">Azure Monitor for Containers</a></li>
<li><a class="sc-dqBHgY hiPvaM" title="https://docs.microsoft.com/en-us/azure/aks/kubernetes-dashboard" href="https://docs.microsoft.com/en-us/azure/aks/kubernetes-dashboard" target="_blank" rel="noopener noreferrer">Kubernetes Dashboard</a></li>
<li><a class="sc-dqBHgY hiPvaM" title="https://github.com/helm/charts/tree/master/stable/prometheus-operator" href="https://github.com/helm/charts/tree/master/stable/prometheus-operator" target="_blank" rel="noopener noreferrer">Prometheus Operator</a></li>
</ul>
<p> </p>
<h1>Enable Azure Monitor for Containers</h1>
<p><a class="sc-dqBHgY hiPvaM" title="https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-overview" href="https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-overview" target="_blank" rel="noopener noreferrer">Azure Monitor for Containers</a> gives you a comprehensive moitoring experience within the Azure ecosystem.</p>
<p>It is based on <a class="sc-dqBHgY hiPvaM" title="https://docs.microsoft.com/en-gb/azure/azure-monitor/log-query/log-query-overview" href="https://docs.microsoft.com/en-gb/azure/azure-monitor/log-query/log-query-overview" target="_blank" rel="noopener noreferrer">Log Analytics Workspace</a>, which integrates with the metrics provided by the Kubernetes cluster. Application and workload metrics are collected from Kubernetes nodes using queries and can be used to create custom alerts, dashboards, and detailed performance analysis. There are several ways to set up monitoring of your Kubernetes workload in Azure. The easiest is go to your cluster and enable Insights.</p>
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5kok3edhb7skcq8y7icn.png)
<p>This starts an onboarding process which will guide you through the necessary steps of creating a <a class="sc-dqBHgY hiPvaM" title="https://docs.microsoft.com/en-gb/azure/azure-monitor/log-query/log-query-overview" href="https://docs.microsoft.com/en-gb/azure/azure-monitor/log-query/log-analytics-overview" target="_blank" rel="noopener noreferrer">Log Analytics Workspace</a> (or choosing an existing one) and then setting up Log Analytics to make additional metrics available to Azure Monitoring. Before any data can be retrieved by Azure Monitoring, a Log Analytics Agent has to be deployed on the targeted cluster. The agent provides performance metrics via a metrics service running on the cluster. The instructions for deploying the agent can be found <a class="sc-dqBHgY hiPvaM" title="https://docs.microsoft.com/en-us/azure/azure-monitor/insights/containers#configure-a-log-analytics-windows-agent-for-kubernetes" href="https://docs.microsoft.com/en-us/azure/azure-monitor/insights/containers#configure-a-log-analytics-windows-agent-for-kubernetes" target="_blank" rel="noopener noreferrer">here</a> for Linux and <a class="sc-dqBHgY hiPvaM" title="https://docs.microsoft.com/en-gb/azure/azure-monitor/log-query/log-query-overview" href="https://docs.microsoft.com/en-gb/azure/azure-monitor/log-query/log-query-overview" target="_blank" rel="noopener noreferrer">here</a> for Windows respectively. If you therefore choose to deploy a Kubernetes DaemonSet without secrets, you will have to provide the Workspace ID and Key of your Log Analytics Workspace. You will find these credentials under <em>Log Analytics Workspace → Agents Management.</em></p>
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i79iy3czx95j3b00ojo8.png)
<p>Once your Log Analytics Agent is up and running, performance metrics are available to your cluster Monitoring service.</p>
<p>Be aware that Azure Monitoring might come with high cost, depending on the amount of data that is transferred. I recommend to check its <a class="sc-dqBHgY hiPvaM" title="https://azure.microsoft.com/en-gb/pricing/details/monitor/" href="https://azure.microsoft.com/en-gb/pricing/details/monitor/" target="_blank" rel="noopener noreferrer">pricing</a> first, before taking on this approach.</p>
<p>See also this <a class="sc-dqBHgY hiPvaM" title="https://sysadminas.eu/Part-4-AKS/" href="https://sysadminas.eu/Part-4-AKS/" target="_blank" rel="noopener noreferrer">blog post</a> for more detailed instructions on how to configure Azure Monitoring.</p>
<h1 id="Use-the-Kubernetes-Dashboard">Use the Kubernetes Dashboard</h1>
<p>If you just need a simple overview dashboard for your Kubernetes cluster Performance, the Kubernetes Dashboard might come in handy. In contrast to Azure Monitoring, it is very convenient to use. If using AKS prior to version 1.18, the Azure dashboard add-on is already installed and enabeld on every Kubernetes Cluster.</p>
<p>With AKS version prior to 1.18 and the (now deprecated) dashboard add-on installed, the dashboard is conveniently accessible by using the Azure-Cli <code>browse</code> command:</p>
<pre class="hljs" style="display: block; overflow-x: auto; padding: 0.5em; background: #f0f0f0; color: #444444;">az aks browse --resource-<span class="hljs-keyword" style="font-weight: bold;">group</span> <span class="hljs-title" style="color: #880000; font-weight: bold;">myResourceGroup</span> --name myCluster</pre>
<div class="code-block sc-jWojfa kiNper"> </div>
<div class="code-block sc-jWojfa kiNper">And your dashboard loads right away inside your default browser.</div>
<p>Note that the add-on is marked as deprecated due to security issues and it is recommended to install the open source version. Instructions for disabling the dashboard add-on can be found in the official <a class="sc-dqBHgY hiPvaM" title="https://docs.microsoft.com/en-us/azure/aks/kubernetes-dashboard" href="https://docs.microsoft.com/en-us/azure/aks/kubernetes-dashboard" target="_blank" rel="noopener noreferrer">AKS dashboard documentation</a>.</p>
<p>The installation process is described <a class="sc-dqBHgY hiPvaM" title="https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/" href="https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/" target="_blank" rel="noopener noreferrer">here</a>. It basically boils down to executing a command like the following:</p>
<pre class="hljs" style="display: block; overflow-x: auto; padding: 0.5em; background: #f0f0f0; color: #444444;">kubectl apply -f <span class="hljs-symbol" style="color: #bc6060;">https:</span>/<span class="hljs-regexp" style="color: #bc6060;">/raw.githubusercontent.com/kubernetes</span><span class="hljs-regexp" style="color: #bc6060;">/dashboard/v</span>2.<span class="hljs-number" style="color: #880000;">0</span>.<span class="hljs-number" style="color: #880000;">0</span>/aio/deploy/recommended.yaml</pre>
<p>After successful installation, you will find the <em>kubernetes-dashboard</em> URL with the <em>cluster-info</em> command, i.e.</p>
<pre class="hljs" style="display: block; overflow-x: auto; padding: 0.5em; background: #f0f0f0; color: #444444;"><span class="hljs-attribute" style="font-weight: bold;">kubectl</span> cluster-<span class="hljs-literal" style="color: #78a960;">info</span></pre>
<p>The dashboard can be made available on your local machine via <a class="sc-dqBHgY hiPvaM" title="https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#proxy" href="https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#proxy" target="_blank" rel="noopener noreferrer">kubectl proxy</a>.</p>
<p>The Kubernetes proxy will establish an authenticated connection to the cluster IP running on the Kubernetes cluster, so that you can access the dashboard with your web browser. The URL is obtained by replacing the cluster host name with the address of your local proxy (localhost:8001 by default).</p>
<p>Note that, by the time of writing, the Kubernetes dashboard supports resource metrics integration only via Heapster, which is being deprecated in favor of metrics server. Support for metrics api is being worked on as part of the next generation dashboard. Therefore the current dashboard does not provide any resource metrics over time. An alternate resource monitoring pipeline for your Kubernetes cluster can be integrated with third party tools like Prometheus and Grafana, which is covered in the next section.</p>
<p> </p>
<h1 id="Install-Prometheus-and-Grafana">Install Prometheus and Grafana</h1>
<p>Prometheus in combination with <a title="Grafana" href="https://grafana.com/" target="_blank" rel="noopener noreferrer">Grafana</a> will get you the most popular open source tool pipeline to monitor your Azure Cubernetes cluster.</p>
<p>To install Prometheus, the most convenient way is using the <a class="sc-dqBHgY hiPvaM" title="https://github.com/helm/charts/tree/master/stable/prometheus-operator" href="https://github.com/helm/charts/tree/master/stable/prometheus-operator" target="_blank" rel="noopener noreferrer">Prometheus Operator Helm Chart</a>. <a class="sc-dqBHgY hiPvaM" title="https://helm.sh/" href="https://helm.sh/">Helm</a> is a package manager for Kubernetes. It simplifies the installation by integrating most of the necessary configuration settings into one package. The following instructions are based on a <a class="sc-dqBHgY hiPvaM" title="https://atouati.com/posts/2019/12/aks-monitoring-with-prometheus/" href="https://atouati.com/posts/2019/12/aks-monitoring-with-prometheus/" target="_blank" rel="noopener noreferrer">blog post</a> which describes the installation process specific for AKS clusters.</p>
<p>First, a local Helm values file is created, which contains several important settings to make Prometheus Operator working with your AKS cluster.</p>
<pre class="hljs" style="display: block; overflow-x: auto; padding: 0.5em; background: #f0f0f0; color: #444444;"><span class="hljs-meta" style="color: #1f7199;">---</span>
<span class="hljs-comment" style="color: #888888;"># Forcing Kubelet metrics scraping on http </span>
<span class="hljs-attr">kubelet:</span>
<span class="hljs-attr">    enabled:</span> <span class="hljs-literal" style="color: #78a960;">true</span>
<span class="hljs-attr">serviceMonitor:</span>
<span class="hljs-attr">    https:</span> <span class="hljs-literal" style="color: #78a960;">false</span>
<span class="hljs-comment" style="color: #888888;"># Disabling scraping of Master Nodes Components</span>
<span class="hljs-attr">kubeControllerManager:</span>
<span class="hljs-attr">    enabled:</span> <span class="hljs-literal" style="color: #78a960;">false</span>
<span class="hljs-attr">kubeScheduler:</span>
<span class="hljs-attr">    enabled:</span> <span class="hljs-literal" style="color: #78a960;">false</span>
<span class="hljs-attr">kubeEtcd:</span>
<span class="hljs-attr">    enabled:</span> <span class="hljs-literal" style="color: #78a960;">false</span>
<span class="hljs-attr">kubeProxy:</span>
<span class="hljs-attr">    enabled:</span> <span class="hljs-literal" style="color: #78a960;">false</span>
<span class="hljs-comment" style="color: #888888;"># Optional: Disable Grafana if you have your own deployment</span>
<span class="hljs-attr">grafana:</span>
<span class="hljs-attr">    enabled:</span> <span class="hljs-literal" style="color: #78a960;">false</span></pre>
<p>Then, the Prometheus Operator is installed in the <code><span class="code">monitoring</span> </code>namespace.</p>
<pre class="hljs" style="display: block; overflow-x: auto; padding: 0.5em; background: #f0f0f0; color: #444444;"># <span class="hljs-keyword" style="font-weight: bold;">If</span> <span class="hljs-keyword" style="font-weight: bold;">using</span> helm <span class="hljs-keyword" style="font-weight: bold;">for</span> the first time, <span class="hljs-keyword" style="font-weight: bold;">add</span> the stable repo
# helm repo <span class="hljs-keyword" style="font-weight: bold;">add</span> stable https:<span class="hljs-comment" style="color: #888888;">//kubernetes-charts.storage.googleapis.com/</span>

kubectl <span class="hljs-keyword" style="font-weight: bold;">create</span> <span class="hljs-keyword" style="font-weight: bold;">namespace</span> monitoring
helm upgrade --install prometheus --<span class="hljs-keyword" style="font-weight: bold;">namespace</span> monitoring stable/prometheus-<span class="hljs-keyword" style="font-weight: bold;">operator</span> --values values.yml </pre>
<p>Check the status of the Prometheus Operator by running:</p>
<pre class="hljs" style="display: block; overflow-x: auto; padding: 0.5em; background: #f0f0f0; color: #444444;">kubectl --<span class="hljs-keyword" style="font-weight: bold;">namespace</span> monitoring <span class="hljs-keyword" style="font-weight: bold;">get</span> pods</pre>
<p>If everything works as expected, the metrics can be viewed with the Grafana dashboard. The Grafana container installed with the <a class="sc-dqBHgY hiPvaM" title="https://github.com/helm/charts/tree/master/stable/prometheus-operator" href="https://github.com/helm/charts/tree/master/stable/prometheus-operator" target="_blank" rel="noopener noreferrer">Prometheus Operator Chart</a> can be accessed with <a class="sc-dqBHgY hiPvaM" title="https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#port-forward" href="https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#port-forward" target="_blank" rel="noopener noreferrer">kubectl port-forward</a>. The default credentials are user <code><span class="code">admin</span></code> with password<code> <span class="code">prom-operator</span></code>. See this <a class="sc-dqBHgY hiPvaM" title="https://atouati.com/posts/2019/12/aks-monitoring-with-prometheus/" href="https://atouati.com/posts/2019/12/aks-monitoring-with-prometheus/" target="_blank" rel="noopener noreferrer">blog post</a> for the details of launching the Grafana dashboards.</p>
<p>To uninstall the Prometheus Operator, carefully follow these instructions:</p>
<pre class="hljs"># Uninstall/<span class="hljs-keyword">delete</span> the prometheus deploymen<span class="hljs-variable">t:</span>
helm <span class="hljs-keyword">delete</span> --namespace monitoring prometheus 

# CRDs created by this chart are not removed by default <span class="hljs-built_in">and</span> should <span class="hljs-keyword">be</span> manually cleaned <span class="hljs-keyword">up</span>
kubectl <span class="hljs-keyword">delete</span> crd prometheuses.monitoring.coreos.<span class="hljs-keyword">com</span>
kubectl <span class="hljs-keyword">delete</span> crd prometheusrules.monitoring.coreos.<span class="hljs-keyword">com</span>
kubectl <span class="hljs-keyword">delete</span> crd servicemonitors.monitoring.coreos.<span class="hljs-keyword">com</span>
kubectl <span class="hljs-keyword">delete</span> crd podmonitors.monitoring.coreos.<span class="hljs-keyword">com</span>
kubectl <span class="hljs-keyword">delete</span> crd alertmanagers.monitoring.coreos.<span class="hljs-keyword">com</span>
kubectl <span class="hljs-keyword">delete</span> crd thanosrulers.monitoring.coreos.<span class="hljs-keyword">com</span>

# Finally, <span class="hljs-keyword">delete</span> the namespace
kubectl <span class="hljs-keyword">delete</span> namespace monitoring --cascade=true</pre>
<p> </p>
<p>If you want to use your existing Grafana AKS deployment, download an existing Kubernetes dashboard like</p>
<ul class="ak-ul" data-indent-level="1">
<li>
<p><a class="sc-dqBHgY hiPvaM" title="https://grafana.com/grafana/dashboards/315" href="https://grafana.com/grafana/dashboards/315" target="_blank" rel="noopener noreferrer">https://grafana.com/grafana/dashboards/315</a></p>
</li>
<li>
<p><a class="sc-dqBHgY hiPvaM" title="https://grafana.com/grafana/dashboards/7249" href="https://grafana.com/grafana/dashboards/7249" target="_blank" rel="noopener noreferrer">https://grafana.com/grafana/dashboards/7249</a></p>
</li>
<li>
<p><a class="sc-dqBHgY hiPvaM" title="https://grafana.com/grafana/dashboards/3119" href="https://grafana.com/grafana/dashboards/3119" target="_blank" rel="noopener noreferrer">https://grafana.com/grafana/dashboards/3119</a></p>
</li>
</ul>
<p>and configure the dashboards with a Prometheus Datasource pointing to <code><span class="code">http://prometheus-prometheus-oper-prometheus.monitoring:9090</span></code>.</p>
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h3mq33wfolsmjcdq7msz.jpg)
<p data-pm-slice="1 1 []">With <a href="https://grafana.com/docs/grafana-cloud/getting-started/gs-metrics/" target="_blank" rel="noopener noreferrer">Grafana Cloud</a> on the horizon, this kind of monitoring stack can be even extended to become a managed, centralized, long-term store for your AKS cluster metrics.</p>
<p data-pm-slice="1 1 []"> </p>
<h1>References</h1>
<ol class="ak-ol">
<li>
<p><a href="https://sysadminas.eu/Part-4-AKS/" target="_blank" rel="noopener noreferrer">Azure Kubernetes Services (AKS). Monitor your AKS cluster.</a></p>
</li>
<li>
<p><a href="https://docs.microsoft.com/en-gb/azure/azure-monitor/log-query/log-analytics-overview" target="_blank" rel="noopener noreferrer">Log Analytics Workspace</a></p>
</li>
<li>
<p><a href="https://docs.microsoft.com/en-us/azure/azure-monitor/insights/containers" target="_blank" rel="noopener noreferrer">Container Monitoring solution in Azure Monitor</a></p>
</li>
<li>
<p><a href="https://docs.microsoft.com/en-us/azure/aks/kubernetes-dashboard" target="_blank" rel="noopener noreferrer">Access the Kubernetes web dashboard in Azure Kubernetes Service (AKS)</a></p>
</li>
<li>
<p><a href="https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/" target="_blank" rel="noopener noreferrer">Kubernetes Web UI (Dashboard)</a></p>
</li>
<li>
<p><a href="https://atouati.com/posts/2019/12/aks-monitoring-with-prometheus/" target="_blank" rel="noopener noreferrer">AKS monitoring with Prometheus</a></p>
</li>
</ol>