From Istio 1.3.x to 1.4.x after a memory leak 🚀
Tags: kubernetes, istio, grafana, pingdom
Published: 2020-07-08 12:27:28

# Context

Recently I designed the new Ornikar platform using [Kubernetes](https://kubernetes.io/) and [Istio](https://istio.io/) using [Google Cloud](https://cloud.google.com/).

[Kubernetes](https://kubernetes.io/), it's good, everyone knows and it's fantastic. OK.
But [Kubernetes](https://kubernetes.io/) without service-mesh and without the super powers of [Istio](https://istio.io/), it's a bit like Superman can only fly two hours a day, it's cool, but it's very limited.

Joking aside, so I set up [Istio](https://istio.io/) in version `1.3.x`. The troubles started when I detected a memory leak on one of our gateways.

## First memory leak detection

The first detection was really made thanks to external observability.
In fact, I use [Pingdom](https://www.pingdom.com/) to measure the uptime, availability and response time of my external endpoints.

![Pingdom problem](https://dev-to-uploads.s3.amazonaws.com/i/kh1czk6c1mqvvdlvw9cb.png)

Naturally, I first try to see if there were no false positives aside [Pingdom](https://www.pingdom.com/).
But that was not it.

## Memory leak detection confirmation

Then I easily found the information with our amazing stack [Prometheus](https://prometheus.io/), [Thanos](https://thanos.io/) and [Grafana](https://grafana.com/grafana/).

![Grafana problem](https://dev-to-uploads.s3.amazonaws.com/i/a8j5qpauu8nfqks6yfkc.png)

This graph is to be correlated with the uptime graph.
We can clearly see the memory leak.
At this moment, I must act!


## Problem solve

To correct the problem, a simple upgrade to the higher version of [Istio](https://istio.io/) was enough.

![Github PR](https://dev-to-uploads.s3.amazonaws.com/i/i572dfn282vmon8qn593.png)

Very simple with us because the whole infrastructure is "as-code" and with use [Github](https://github.com/) and [Codefresh](https://codefresh.io/) to deploy.

---

To finish, I just had to restart each deployment to update [Envoy](https://www.envoyproxy.io/) in sidecar.

![Grafana fix graph daily](https://dev-to-uploads.s3.amazonaws.com/i/6ercq2r1c7uclucchzje.png)

We can see on the graph above that the upgraded version of Istio did fix the memory leaks.

![Grafana fix graph](https://dev-to-uploads.s3.amazonaws.com/i/4yhv5h49pqwtcxey6hr5.png)

We also see that this solution is stable over time.

## Some conclusions

### First conclusion

When we talk to you about "observability", that should be taken seriously.
Many people tend to forget that observability is above all to have eyes on what we do.
In this case, if I had not set up [Pingdom](https://www.pingdom.com/) and [Grafana](https://grafana.com/grafana/) I would never have been able to detect the failure.
And in a non-proactive way, it's the platform users who have reported the disturbances.


### Second conclusion

Each time a component is installed, whether infrastructure or software.
Add the right metrics that let you know if the component is healthy or not.


### Last conclusion

I like Istio, I like service mesh and I love Open Source. 🚀


### Cover credit

[julienberthier.org](https://julienberthier.org/love-love.html)