Knative monitoring with Grafana, Zipkin, Weavescope & other plugins..
Tags: prometheus, grafana, kubernetes, cloud
Published: 2019-03-21 09:21:57

---
title: Knative monitoring with Grafana, Zipkin, Weavescope & other plugins..
published: true
tags: prometheus,grafana,kubernetes,cloud
canonical_url: https://medium.com/vmacwrites/knative-monitoring-with-grafana-zipkin-weavescope-other-plugins-30a2d8d20344
---

In this post, you will see the telemetry side of Knative and Istio for a nodejs app named Knative-node-app published on IBM Cloud in the previous post [_Install Knative with Istio and deploy an app on IBM Cloud_](https://dev.to/vidyasagarmsc/install-knative-with-istio-on-ibm-cloud-2fd9).

As per the Monitoring, Logging and Tracing Installation documentation of Knative,

> Knative Serving offers two different monitoring setups: [Elasticsearch, Kibana, Prometheus and Grafana](https://github.com/knative/docs/blob/master/serving/installing-logging-metrics-traces.md#elasticsearch-kibana-prometheus--grafana-setup) or [Stackdriver, Prometheus and Grafana](https://github.com/knative/docs/blob/master/serving/installing-logging-metrics-traces.md#stackdriver-prometheus--grafana-setup) You can install only one of these two setups and side-by-side installation of these two are not supported.

We will stick to Elasticsearch, Kibana, Prometheus and Grafana stack and will additionally use Weavescope for in-depth visualization of containers, pods etc.,

If you have installed serving component while setting up Knative, you should have the monitoring component is already installed. To confirm Knative serving component installation, run the below command

```
$ kubectl describe deploy controller --namespace knative-serving
```

![](https://cdn-images-1.medium.com/max/1024/1*dMS_nwLCgnPozse2FRUpyg.png)<figcaption>Knative Serving details</figcaption>

To check the installation of monitoring component, run the below command

```
kubectl get pods --namespace knative-monitoring
```

![](https://cdn-images-1.medium.com/max/1024/1*w9v0J3mPNewj16F2DihEtg.png)

> If you don’t see anything running, follow the [steps here](https://github.com/knative/docs/blob/master/serving/installing-logging-metrics-traces.md#before-you-begin) to setup.

### Grafana

You can access metrics through the [Grafana](https://grafana.com/) UI. Grafana is the visualization tool for [Prometheus](https://prometheus.io/).

To open Grafana, enter the following command:

```
kubectl port-forward --namespace monitoring $(kubectl get pods --namespace monitoring --selector=app=grafana --output=jsonpath="{.items..metadata.name}") 3000
```

**Note:** This starts a local proxy of Grafana on port 3000. For security reasons, the Grafana UI is exposed only within the cluster.

Navigate to the Grafana UI at [http://localhost:3000](http://localhost:3000/).

![](https://cdn-images-1.medium.com/max/1024/1*GSpmymhThc2bJeauSltYhA.png)

![](https://cdn-images-1.medium.com/max/1024/1*yeAs4iWTkNXLtxB-8IBPTg.png)<figcaption>Grafana UI — knative-node-app</figcaption>

You can also check the metrics for Knative Serving- scaling, deployments, pods etc.,

![](https://cdn-images-1.medium.com/max/1024/1*C59TynesT42mG-cUJv4kcQ.png)<figcaption>Knative Serving — scaling metrics</figcaption>

The following dashboards are pre-installed with Knative Serving:

- Revision HTTP Requests: HTTP request count, latency, and size metrics per revision and per configuration
- Nodes: CPU, memory, network, and disk metrics at node level
- Pods: CPU, memory, and network metrics at pod level
- Deployment: CPU, memory, and network metrics aggregated at deployment level
- Istio, Mixer and Pilot: Detailed Istio mesh, Mixer, and Pilot metrics
- Kubernetes: Dashboards giving insights into cluster health, deployments, and capacity usage

### Zipkin

In order to access request traces, you use the Zipkin visualization tool.

To open the Zipkin UI, enter the following command:

```
kubectl proxy
```

This command starts a local proxy of Zipkin on port 8001. For security reasons, the Zipkin UI is exposed only within the cluster.

Navigate to the [Zipkin UI](http://localhost:8001/api/v1/namespaces/istio-system/services/zipkin:9411/proxy/zipkin/) and Click “Find Traces” to see the latest traces. You can search for a trace ID or look at traces of a specific application. Click on a trace to see a detailed view of a specific call.

![](https://cdn-images-1.medium.com/max/1024/1*J4ZCAZX5c0rD_n97yTaMZA.png)

![](https://cdn-images-1.medium.com/max/1024/1*OX1kUKfK5-yNKwsOry43Pg.png)<figcaption>Zipkin Traces — Knative-node-app-0001</figcaption>

### Weavescope

While [obtaining and visualising uniform metrics, logs, traces across microservices using Istio ](https://vmacwrites.wordpress.com/2018/06/28/obtain-and-visualise-uniform-metrics-logs-traces-across-microservices-using-istio/), I fell in love with [Weavescope](https://www.weave.works/oss/scope/). So thought of playing with it to understand the processes, containers, hosts and others involved in my application.

Scope is deployed onto a Kubernetes cluster with the command

```
kubectl apply -f “[https://cloud.weave.works/k8s/scope.yaml?k8s-version=$(kubectl](https://cloud.weave.works/k8s/scope.yaml?k8s-version=%24(kubectl) version | base64 | tr -d ‘\n’)”
```

To open Weavescope, run the command and open [http://localhost:4040/](http://localhost:4040/)

```
kubectl port-forward -n weave “$(kubectl get -n weave pod — selector=weave-scope-component=app -o jsonpath=’{.items..metadata.name}’)” 4040
```

![](https://cdn-images-1.medium.com/max/1024/1*6azgps1dDbeLcNHqepNH7A.png)

![](https://cdn-images-1.medium.com/max/1024/1*hsXkyaiJNbtGR4UcMd73-Q.png)

### Kibana + ElasticSearch

I tried to visualise the logs using Kibana UI (the visualization tool for [Elasticsearch](https://info.elastic.co/)), but struck with the following error while configuring an index pattern — _“Unable to fetch mapping. Do you have indices matching the pattern?”_

![](https://cdn-images-1.medium.com/max/1024/1*bOQ9W4aXu3vb5iMiQt1oMA.png)

As there will be a revise of “logging and monitoring” related topics as per [this issue](https://github.com/knative/docs/issues/447)on knative GitHub repo. I will be revisiting logs in future for sure.

**Update:** Found a workaround for this issue following the answers in this [stackoverflow question](https://stackoverflow.com/questions/53512182/knative-logging-kibana-unable-to-fetch-mapping).

Here are the steps,

Run the below command to apply a patch to fix the fluentd-ds pods not showing issue

```
kubectl apply -f [https://raw.githubusercontent.com/gevou/knative-blueprint/master/knative-serving-release-0.2.2-patched.yaml](https://raw.githubusercontent.com/gevou/knative-blueprint/master/knative-serving-release-0.2.2-patched.yaml)
```

Verify that each of your nodes have the beta.kubernetes.io/fluentd-ds-ready=true label:

```
kubectl get nodes --selector beta.kubernetes.io/fluentd-ds-ready=true
```

If you receive the No Resources Found response:

- Run the following command to ensure that the Fluentd DaemonSet runs on all your nodes:

```
kubectl label nodes — all beta.kubernetes.io/fluentd-ds-ready=”true”
```

- Run the following command to ensure that the fluentd-ds daemonset is ready on at least one node:

```
kubectl get daemonset fluentd-ds --namespace knative-monitoring
```

![](https://cdn-images-1.medium.com/max/1024/1*pZSzO_sKo4q9yjNs8EWj5Q.png)

Wait for a while and run this command

```
kubectl proxy
```

- Navigate to the [Kibana UI](http://localhost:8001/api/v1/namespaces/knative-monitoring/services/kibana-logging/proxy/app/kibana). _It might take a couple of minutes for the proxy to work_.
- Within the “Configure an index pattern” page, enter logstash-\* to Index pattern and select @timestamp from Time Filter field name and click on Create button.
- To create the second index, select Create Index Pattern button on top left of the page. Enter zipkin\* to Index pattern and select timestamp\_millis from Time Filter field name and click on Create button.

![](https://cdn-images-1.medium.com/max/1024/1*bN4O7ds87NnpaXmQGBLhEw.png)

If you still see the issue, try clicking **Dev Tools** on the **Kibana** page andrun this command

```
GET \_cat/indices?v
```

![](https://cdn-images-1.medium.com/max/1024/1*kk7Gvn-q84O6nEf8aWB6ow.png)

### Further reading

- **Knative** [Monitoring with Sysdig](https://medium.com/vmacwrites/knative-monitoring-with-sysdig-on-ibm-cloud-dc206c3f413d)
- **Knative 2/2**  — [https://medium.com/google-cloud/knative-2-2-e542d71d531d](https://medium.com/google-cloud/knative-2-2-e542d71d531d)
- **Obtain and visualise uniform metrics, logs, traces across different services using Istio**  — [https://github.com/IBM-Cloud/jpetstore-kubernetes/tree/master/istio](https://github.com/IBM-Cloud/jpetstore-kubernetes/tree/master/istio)

* * *