Running a modern infrastructure on Kubernetes
Tags: devops, containers, docker, kubernetes
Published: 2019-10-26 16:59:43

---
title: Running a modern infrastructure on Kubernetes
published: true
date: 2018-02-09 01:57:58 UTC
tags: devops,containers,docker,kubernetes
canonical_url: 
---

At StashAway, we have been running Docker containers from the very first day. Initially, we were using Rancher as the container orchestrator, but as we grew, we decided to switch to Kubernetes (k8s) — mainly because of its rapidly growing ecosystem and wide adoption.

This post describes how we use k8s and its tooling stack to run our application in a production-grade environment.

![](https://cdn-images-1.medium.com/max/1024/1*G9OVxt8KHclvgfedgmyV-Q.png)<figcaption>Google trends graph showing how the interest on various container schedulers changed over time.</figcaption>

All applications whether stateless or stateful needs an environment with these fundamental necessities built-in:

### Service Discovery

Containers are elastic in nature, they can come up and go down anytime. Since each container gets a dynamic IP address, registration of each container instance is a must, so that others can communicate with it. Kubernetes supports two modes of discovery: Environment variables and DNS-based.

If you are for example running Cassandra inside a container, its IP address will be available both as an environment variable _CASSANDRA\_HOST_, as well as a domain name _cassandra.default.svc.cluster.local_.

DNS-based service discovery is more popular among the two but special care needs to be taken since some DNS client libraries set high DNS cache TTL values. Eg. [JVM caches domain names forever by default](https://docs.oracle.com/javase/8/docs/technotes/guides/net/properties.html).

### Service Addressing

DNS-based names for discovery as shown above are only resolvable from inside the Kubernetes cluster. In order to address a service from outside the cluster, it needs an automatic DNS registration to a third party DNS provider such as AWS Route53, Google CloudDNS, AzureDNS, CloudFlare. To mitigate against the dependency on a single DNS provider, you should consider hosting your zones on multiple providers.

In k8s world, this can be achieved easily via [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) using [ExternalDNS](https://github.com/kubernetes-incubator/external-dns). This incubator project takes care of registering a new (sub-)domain as soon as any new k8s service or ingress controller is created. It is also aware of the records it manages via an extra TXT record along with the primary A record, hence preventing any accidental overwriting of existing records.

### Routing

With lots of containers and services popping in and out of existence, routing external traffic to healthy containers is challenging. [Kubernetes Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) is the saviour. It provides load balancing, SSL termination and even name based routing. Ingress is just an abstraction layer which can use any software load-balancer as its implementation.

Current Ingress controller implementations include [Nginx Ingress](https://github.com/kubernetes/ingress-nginx) (Nginx based), [Voyager](https://github.com/appscode/voyager) (HAProxy based) and [Contour](https://github.com/heptio/contour) (EnvoyProxy based). The first one is the most matured which we are using (along with ELB) for all our traffic routing — but it provides only HTTP based routing. For TCP based routing, you’ll need to use Voyager. Contour is very interesting since it comes along with all the benefits of Envoy which is a service proxy designed specifically for modern cloud native applications. It has first class support for gRPC and provides features like circuit breaking which are not available in standard load-balancers.

### Monitoring

Many Kubernetes objects like pods, services and ingresses together define the application, hence it is important to monitor the state of each one of them.

[Prometheus](https://prometheus.io/) is definitely the right choice available in open-source to monitor your Kubernetes apps and cluster. It has an inbuilt [discovery](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#<kubernetes_sd_config>) for these k8s objects. Since monitoring without alerting is useless, [Alertmanager](https://github.com/prometheus/alertmanager) perfectly fills the gap by providing nice integrations like Slack notifications.

Most people use Prometheus along with [Heapster](https://github.com/kubernetes/heapster) which can be integrated with many open-source monitoring solutions like InfluxDB and Riemann. Those who want to get fine container level metrics can add [cAdvisor](https://github.com/google/cadvisor) to their monitoring stack, too.

### Logging

While you are running multiple instances of the same image, you can’t afford to login into each container and tail the logs. Each k8s node needs to run an agent to push these container logs. Surprisingly, in k8s world, the EFK stack is more popular than the ELK stack.

[Fluent-bit](http://fluentbit.io/) is a lightweight (alternative to [Fluentd](https://www.fluentd.org/)) and is a fully Docker- and Kubernetes-aware agent which can be used to push these logs directly to Elasticsearch. It automatically adds kubernetes labels and annotations in each log line. You can also integrate it with Slack for sending notifications in case of any error/exception.

### Deploying

We bundle all our applications as Docker images along with its dependencies. To keep the deployment pipeline clean, its very important to templatize these manifests. [Helm](https://helm.sh/), the package manager for k8s, is a great way to deploy apps on Kubernetes. There are many community-managed helm [charts](https://github.com/kubernetes/charts) which are stable and ready to be used in production.

Since Helm doesn’t provide a neat way to store secrets, we use [Ansible Vault](http://docs.ansible.com/ansible/2.4/vault.html) as their source of truth. We trigger the helm command-line via Ansible using the [ansible-helm](https://gist.github.com/chhetripradeep/12cd9f0b94572cede89e18502b84ced1) module.

One of the pain-points of helm is that someone needs to write these charts by first understanding each of the YAML fields. [Ksonnet](https://ksonnet.io/) is going to [remove it](https://blog.heptio.com/the-next-chapter-for-ksonnet-1dcbbad30cb) by dynamically generating helm charts on demand.

### SSL Certificates

Almost 90% of modern applications expose an HTTP endpoint. To secure these HTTP services, the basic requirement is to install an SSL certificate to enable encrypted communication.

Provisioning, installing & updating these certificates can become cumbersome, if it is not automated properly.

Automatic provisioning of [Let’s Encrypt](https://letsencrypt.org/) certificates for k8s ingresses can be done via [kube-cert-manager](https://github.com/PalmStoneGames/kube-cert-manager). We chose this over [kube-lego](https://github.com/jetstack/kube-lego) since it has support for Let’s encrypt DNS based validation challenges. Hence it can be used for issuing certs for applications which are hosted in a private network. It also takes care of renewing these certificates.

JetStack folks are developing another tool named [cert-manager](https://github.com/jetstack/cert-manager) which is pretty interesting since it will soon be able to use [Hashicorp Vault](https://www.vaultproject.io/) as a CA authority.

### Conclusion

In this post, we talked about why you should choose Kubernetes for your next project. We will go in-depth into few of these topics in our upcoming blog posts.

_We are constantly on the lookout for great tech talent to join our team — _[_visit our website_](https://www.stashaway.sg) _to learn more and feel free to reach out to us!_

* * *