Tracing Ingresses with Nginx and Linkerd
Tags: kubernetes, nginx, linkerd
Published: 2021-01-10 23:19:08

Recently, I had posted [about application observability with Grafana and Loki](https://dev.to/mikeyglitz/kubernetes-service-monitoring-and-observability-3g67). The set up discussed in the previous post enables Kubernetes administrators and application support teams to troubleshoot application issues through log inspection.

Today, I'll continue the discussion of the topics by introducing ingress tracing. Kubernetes resources are accessed from outside the Kubernetes cluster using an [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/). An Ingress proxies inbound requests into Kubernetes and routes the requests to internal Kubernetes resources (i.e. Services and Pods). Ingresses are managed using an ingress controller which provides a proxy interface, a set of cluster-resource-definitions (usually in the form of specialized annotations which are attached to the Ingress resource), and a web-hook which responds to ingress requests.

![Ingress Network Diagram](https://github.com/mikeyGlitz/dev-to-blog/raw/master/Images/ingress.png)

Using [ingress-nginx](https://kubernetes.github.io/ingress-nginx/) and [Jaeger](https://www.jaegertracing.io/) traffic in the ingress network can be monitored in the form of traces. The example discussed with use ingress-nginx and Jager on top of the [Linkerd](https://linkerd.io) service mesh.

## Setting Up Linkerd

Since Jaeger and ingress-nginx will rely on Linkerd to stream metrics from nginx to Jager, Linkerd will have to be set up first.

Create a file, `config.yaml`, which will be used to customize the Linkerd install.

```yaml
tracing:
  enabled: true
```

Install Linkerd using the command:

```bash
linkerd install --config config.yaml | kubectl apply -f -
```

If you have an existing installation of Linkerd, the Linkerd installation can be updated using the `linkerd upgrade` command:

```bash
linkerd upgrade --config config.yaml | kubectl apply -f -
```

Once Linkerd is updated or fully initialized, ingress-nginx will need to be updated or deployed. Jaeger is deployed as part of the linkerd deployment.

## Ingress-Nginx Helm Deployment

The easiest way to deploy ingress-nginx is by utilizing the [helm chart](https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx). Create a file, `values.yml` which will contain customization values for the helm install. ingress-nginx will need to be configured to stream traces to Jaeger.

```yaml
controller:
  config:
    enable-opentracing: "true"
    zipkin-collector-hosts: linkerd-collector.linkerd:55678
  podAnnotations:
    linkerd.io/inject: enabled
    config.linkerd.io/trace-collector: linkerd-collector.linkerd:55678
    config.alpha.linkerd.io/trace-collector-service-account: linkerd-collector
```

Deploy the ingress-nginx helm chart with the following:
```bash
# Update your helm repos with the ingress-nginx repo
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

# Install the ingress-nginx chart
helm install ingress-nginx ingress-nginx/ingress-nginx --namespace=ingress-nginx --values=values.yaml
```

## Upgrading an Existing ingress-nginx configuration

An existing installation of ingress-nginx can be updated to trace to Jaeger in a manner similar to the Helm Deployment mentioned earlier.

```bash
kubectl edit -n ingress-nginx configmap ingress-nginx-controller
```

Update the `data` section with the following yaml body

```yaml
data:
    enable-opentracing: "true"
    zipkin-collector-hosts: linkerd-collector.linkerd:55678
```

Update the `ingress-nginx-controller` deployment with the following pod annotations

```bash
kubectl edit -n ingress-nginx deployment ingress-nginx-controller
```

```yaml
spec:
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
        config.linkerd.io/trace-collector: linkerd-collector.linkerd:55678
        config.alpha.linkerd.io/trace-collector-service-account: linkerd-collector
```

Once the ingress-controller is re-deployed, traces will be visible on Jaeger

![Linkerd Dashboard](https://github.com/mikeyGlitz/dev-to-blog/raw/master/Images/linkerd-dashboard-ingress.png)
![Jaeger](https://github.com/mikeyGlitz/dev-to-blog/raw/master/Images/linkerd-dashboard.png)

# References
- https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/opentracing/
- https://linkerd.io/2/tasks/distributed-tracing/
- https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx