Monitoring and Linkerd
Tags: kubernetes, devops, cloud, git
Published: 2023-05-17 10:17:32

Getting a Service Mesh up and running in your environment is the key to network, security and reliability success. It’s the make or break between truly understanding what’s happening at a network layer inside of your cluster and deploying applications while flying blind.

Once the Service Mesh is up, the next Day Two operation that you must focus on is monitoring. Without monitoring, you could have the best Service Meshed network in the world, but it won’t mean much if the network is down.

In this blog post, you’ll learn how to combine monitoring with Linkerd to ensure network and observability success.

## Prerequisites

For the purposes of this blog post, you should have:

- A Kubernetes cluster up and running. It could be a cluster running locally (like Minikube or Kind) or a Managed Kubernetes Service cluster (Azure Kubernetes Service, Elastic Kubernetes Service, Google Kubernetes Engine, etc.)
- Linkerd installed. You can install Linkerd using the documentation here: [https://linkerd.io/2.12/getting-started/](https://linkerd.io/2.12/getting-started/)

## Linkerd and App Deployment

Before having the ability to monitor an application, you first need one deployed. To get a full view and understanding of how monitoring would work with several microservices, you can deploy the popular demo Sockshop app. The Sockshop app has several microservices and various Kubernetes Resources that you can deploy to see a full architecture vs just deploying a Hello World or Nginx web app.

You can find the Sockshop app here: [https://microservices-demo.github.io/docs/quickstart.html](https://microservices-demo.github.io/docs/quickstart.html)

Let’s learn how to deploy it.

First, clone down the Sockshop application repository.

```jsx
git clone https://github.com/microservices-demo/microservices-demo
```

Next, change the directory (`cd`) into the `microservices-demo` directory.

```jsx
cd microservices-demo
```

Change the directory to the Kubernetes deployment directory.

```jsx
cd deploy/kubernetes
```

Create the expected `sock-shop` Kubernetes Namespace that the Sockshop application is expecting the Kubernetes Resources to be deployed into.

```jsx
kubectl create namespace sock-shop
```

Lastly, deploy the Sockshop application.

```jsx
kubectl apply -f complete-demo.yaml
```

You should see an output on the terminal similar to the screenshot below.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/33c2ks8s0h2v3bybxby4.png)


Give the application a few minutes to get up and running.

After a few minutes, you can run the following Kubectl command to see all of the resources up and running.

```jsx
kubectl get all -n sock-shop
```


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u6vwbfuggbwffwntwcyh.png)


Inject Linkerd’s sidecar into the Deployment Pods so they’re fully managed with the Service Mesh.

```jsx
kubectl get deploy -o yaml -n sock-shop | linkerd inject - | kubectl apply -f -
```

After a few minutes, you should see the container count in all of the Pods raise by one container (the Linkerd sidecar).


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8sgthxwo8ngba35ruey1.png)


## Monitoring With Viz

Now that the Sockshop microservice is deployed, you can monitor it using Viz. Viz is a built-in monitoring system for Linkerd. It gives you the ability to see all of the Service Mesh’d resources and gives you various pieces of information about each Pod (which you’ll see in an upcoming section).

In this section, you’ll do the following:

- Install Viz
- View the Sockshop app in Viz

### Installing Viz

to install Viz, you can use the Linkerd CLI with the following command:

```jsx
linkerd viz install | kubectl apply -f -
```

Next, run Viz locally.

```jsx
linkerd viz dashboard &
```

### Monitoring

Once Viz is installed and you run the `dashboard` command above, you can begin to monitor and observe the Service Mesh’d resources.

First, within Linkerd, click the Namespace section and choose the `sock-shop` Namespace.

Under the **HTTP Metrics** section, you’ll see the `sock-shop` Namespace.

Click the Namespace.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5ivuuoqr7yhj86jhvk52.png)


You can now see that all of the Deployments inside of the `sock-shop` Namespace are successfully Mesh’d. Notice how you can see information around the success of the application itself and latency information.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i5ppsbjzbth421u8qlsg.png)


As an example, click on the Carts Deployment.

Within the Deployment, you’re able to see various information around success rates and latency for the application running inside of the Pods.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ftjigdgtxxh6jmo2pxql.png)


Scrolling down a bit on the same page, you can see inbound and outbound routing to the Deployment itself. This is useful if you want a true method of seeing what’s happening inside of the Pod network for your application. Ultimately, the ability to see, from a networking perspective, what’s “talking to” your app is crucial for troubleshooting and understanding Pod communication.

To dive a little bit deeper, click on the Cart Pod.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nzr8weap5la9hiw9h9pk.png)


Once you click on the Pod, you can see what’s accessing your Pod (to the Pod and where the Pod is going) from an ingress and egress perspective. Also notice on the right side that you can see mTLS is active and your connections to the Pod is secure.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zqjocgrpc7mzr73swi07.png)


If you’d like a deeper look into the various routes for your application, you can use the `linkerd viz routes` command.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n4vto0adnzm6fgamcwfc.png)


The Viz dashboard has a lot of great features in terms of Service Mesh, but in a production environment, you’ll most likely want to also use (in addition) a monitoring solution that can monitor your entire environment.

That’s where Prometheus and Grafana come into play.

## Prometheus and Viz

When Viz gets installed, Prometheus automatically gets installed as well. Prometheus is a great way to visualize the Metrics portion of observability practices.

To view the Prometheus dashboard, you can use the following command for port forwarding.

```jsx
kubectl port-forward -n linkerd-viz service/prometheus 68484:9090
```

Open up a web browser and go to the URL that displays when you run the above command.

Once in the Prometheus instance, go to Status → Targets.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7rhicbxhelhvi3kcqnb3.png)


You’ll see within the targets all of the Linkerd resources are automatically observed via Prometheus. It’ll show you the endpoint for the metrics, labels, and the status of the endpoint to ensure that it’s up and operational.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l8d0tkz8hixlvig3u142.png)


## Grafana and Linkerd

The final piece to monitoring Linkerd is by using one of the most popular monitoring UIs in the Kubernetes space, Grafana.

However, Grafana isn’t installed yet. To get Grafana up and running, you’ll have to ensure that Grafana is installed. You can install Grafana by using the official Helm Chart.

```jsx
helm repo add grafana https://grafana.github.io/helm-charts

helm install grafana -n grafana --create-namespace grafana/grafana
```

Once you run the above installation, you’ll see an output similar to the screenshot below.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ai4rwffuxj8qvgiw2jl6.png)


Wait a few minutes for the full Grafana installation to be complete.

To confirm that the installation was a success, check that all of the resources in the Namespace were created successfully.

```jsx
kubectl get all -n grafana
```

To reach the Grafana UI, run the following:

```jsx
kubectl port-forward -n grafana service/grafana :80
```

The UI should now be available locally and you can reach it via output from the above command.

Once the UI is available, you’ll need the password. The Grafana password, when installed via Helm, is saved as a Kubernetes Secret.

To retrieve the password for the login, run the following command (the username is admin):

```jsx
kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```

Once you’re logged into Grafana, go to the Dashboards section and click the blue **New** button.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b5hsfvnj7jgiwnep8tvw.png)

Choose the **Import** option


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sy07e4jbz0hk2j9kehw4.png)


At this point, you can import a Grafana dashboard.

For Linkerd, there’s already a dashboard that you can use. You can download the JSON for the dashboard here: [https://grafana.com/api/dashboards/3986/revisions/1/download](https://grafana.com/api/dashboards/3986/revisions/1/download)

Once downloaded, upload the JSON dashboard and you should see a similar screen as shown in the screenshot below and click the blue **Import** button.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c6fucnfng1avhfdzzs32.png)


Once uploaded, you’ll see a Linkerd dashboard showing statistics about the Service Mesh.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4231myop2q03tf1wurbw.png)


## A Quick Note On Buoyant Cloud

Buoyant Cloud is fully automated Linkerd and is the enterprise/paid version of the platform. The UI gives you the ability to see:

- The health of Linkerd itself
- The health and behavior of your application.

It’s multi-user, fully hosted, cross-cluster, and gives you the enterprise feel of a Service Mesh if multiple members of your team need to access it.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nzsc7kpogvt20w9co2mz.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gdyqq9w9l45lspa4xxnl.png)