What’s inside etcd? A deep dive into the Kubernetes world
Tags: kubernetes, devops, docker, go
Published: 2019-12-03 01:37:26

{% twitter 1201843718585823233 %}

> # Having been away from working on K8s for a while, I decided to cut open K8s to see how it works to try falling in love with it again. This is going to be a series of blogs, with the first surgery being trying to figure out what’s stored inside the etcd cluster.

This is a story of finding the **etcd** pod inside the kubernetes cluster without diving deep into how etcd works.

[Etcd](https://etcd.io/) is kubernetes vault. It stores the entire state of the cluster: its configuration, specifications, and the statuses of the running workloads. In this blog,

## What is etcd?
> # Etcd is defined as a distributed, reliable key-value store for the most critical data of a distributed system. Etcd is written in [Go](https://golang.org/).

## How does etcd work inside kubernetes?
> # Our friends at Heptio have a great blog where they have shown the components involved during a simple Pod creation process. It’s a great illustration of the API Server and etcd interaction.

![](https://cdn-images-1.medium.com/max/2408/1*QLGlVwQ1wmm6c6TrPg4s-w.png)

## Installing etcd

**Download Binary**
```sh
    curl -L [https://github.com/etcd-io/etcd/releases/download/v3.3.11/etcdv3.3.11-linux-amd64.tar.gz](https://github.com/etcd-io/etcd/releases/download/v3.3.11/etcdv3.3.11-linux-amd64.tar.gz) -o etcd-v3.3.11-linux-amd64.tar.gz
```



**Extract**
```sh

    tar xzvf etcd-v3.3.11-linux-amd64.tar.gz
```

## Operate ETCD

**Run server**
```sh

    ./etcd
```
**Run client (on a different terminal)**
```sh

    // Set a key
    ./etcdctl set key1 value1

    // Get a key
    ./etcdctl get key1
```
One can also use [etcd labs](http://play.etcd.io/install) to play around with etcd deployment. It is similar to [docker playground](https://labs.play-with-docker.com/).

## The Kubernetes Test Cluster

![](https://cdn-images-1.medium.com/max/2490/1*nw6vZG9aooP_-9Jggc7vqw.png)

I use minikube along with a virtualboxdriver for starting a docker container, so the next task would be to setup up a one node K8s cluster on my local machine to interact with the etcd store.

**Start the cluster**
```sh
    minikube start — vm-driver=virtualbox
```
**Finding the etcd pod**
```sh

    $ kubectl get pods -n kube-system

    NAME                               READY     STATUS    RESTARTS   AGE
    coredns-5644d7b6d9-2qg54           1/1       Running   0          2d
    coredns-5644d7b6d9-6mbqk           1/1       Running   0          2d
    etcd-minikube                      1/1       Running   0          2d
    kube-addon-manager-minikube        1/1       Running   0          2d
    kube-apiserver-minikube            1/1       Running   0          2d
    kube-controller-manager-minikube   1/1       Running   0          20s
    kube-proxy-kgt6n                   1/1       Running   0          2d
    kube-scheduler-minikube            1/1       Running   0          2d
    storage-provisioner                1/1       Running   0          2d
```
**Connecting to the etcd pod**
```sh

    kubectl exec -it etcd-minikube -n kube-system sh
```
After connecting to the shell, I tried running the **etcdctl** command to find all the keys.
```sh

    etcdctl get /
```
Turns out, it wasn’t as simple as this. Etcd uses an ADVERTISE_URL over which it exposes the process. I had to figure out all the flags which etcd uses at startup. I used the following command to get all the necessary flags.
```sh

    $ echo “$(ps aux)”

    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    root         1  2.9  2.3 10611260 46916 ?      Ssl  10:24   7:04

    etcd --advertise-client-urls=[https://192.168.99.100:2379](https://192.168.99.100:2379) --cert-file=/var/lib/minikube/certs/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/minikube/etcd --initial-advertise-peer-urls=[https://192.168.99.100:2380](https://192.168.99.100:2380) --initial-cluster=minikube=[https://192.168.99.100:2380](https://192.168.99.100:2380) --key-file=/var/lib/minikube/certs/etcd/server.key --listen-client-urls=[https://127.0.0.1:2379,https://192.168.99.100:2379](https://127.0.0.1:2379,https://192.168.99.100:2379) --listen-metrics-urls=[http://127.0.0.1:2381](http://127.0.0.1:2381) --listen-peer-urls=[https://192.168.99.100:2380](https://192.168.99.100:2380) --name=minikube --peer-cert-file=/var/lib/minikube/certs/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/var/lib/minikube/certs/etcd/peer.key --peer-trusted-ca-file=/var/lib/minikube/certs/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/var/lib/minikube/certs/etcd/ca.crt
```
To get **etcdctl** to talk to the **etcd** server, it needs the above flags since **K8s** sets up **etcd** along with authentication inside the pod. We first need to figure out the ADVERTISE_URL, which can be found using the --advertise-client-urls flag.

**Getting all the keys**
```sh

    $ ADVERTISE_URL=”[https://192.168.99.100:2379](https://192.168.99.100:2379)"

    $ ETCDCTL_API=3 etcdctl --endpoints $ADVERTISE_URL \
    --cacert /var/lib/minikube/certs/etcd/ca.crt \
    --cert /var/lib/minikube/certs/etcd/server.crt \ 
    --key /var/lib/minikube/certs/etcd/server.key \ 
    get / — prefix — keys-only
```
**Funny output**
```sh
    /registry/apiregistration.k8s.io/apiservices/v1.
    /registry/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io
    /registry/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io
    /registry/apiregistration.k8s.io/apiservices/v1.apps
    /registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io
    /registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io
    /registry/apiregistration.k8s.io/apiservices/v1.autoscaling
    /registry/apiregistration.k8s.io/apiservices/v1.batch
    /registry/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io
    /registry/apiregistration.k8s.io/apiservices/v1.networking.k8s.io
    /registry/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io
```
Let’s make sense of this by storing the output it in a json file.
```sh

    $ ETCDCTL_API=3 etcdctl --endpoints $ADVERTISE_URL \
    --cacert /var/lib/minikube/certs/etcd/ca.crt \
    --cert /var/lib/minikube/certs/etcd/server.crt \ 
    --key /var/lib/minikube/certs/etcd/server.key \ 
    get / — prefix — keys-only -w json > out.json
```
It outputs all the KV pairs in a json format where they keys are encoded.
```sh

    {
     "header": {
      "cluster_id": 12197035334886545600,
      "member_id": 9217530203749069991,
      "revision": 19300,
      "raft_term": 2
     },
     "kvs": [{
      "key": "L3JlZ2lzdHJ5L2FwaXJlZ2lzdHJhdGlvbi5rOHMuaW8vYXBpc2VydmljZXMvdjEu",
      "create_revision": 12,
      "mod_revision": 12,
      "version": 1
     }, {
      "key": "L3JlZ2lzdHJ5L2FwaXJlZ2lzdHJhdGlvbi5rOHMuaW8vYXBpc2VydmljZXMvdjEuYWRtaXNzaW9ucmVnaXN0cmF0aW9uLms4cy5pbw==",
      "create_revision": 9,
      "mod_revision": 9,
      "version": 1
     }, {
      "key": "L3JlZ2lzdHJ5L2FwaXJlZ2lzdHJhdGlvbi5rOHMuaW8vYXBpc2VydmljZXMvdjEuYXBpZXh0ZW5zaW9ucy5rOHMuaW8=",
      "create_revision": 10,
      "mod_revision": 10,
      "version": 1
     }]
    }
```
**Output for one key**
```sh

    $ ETCDCTL_API=3 etcdctl — endpoints $ADVERTISE_URL — cacert /var/lib/minikube/certs/etcd/ca.crt — cert /var/lib/minikube/certs/etcd/server.crt — key /var/lib/minikube/certs/etcd/server.key get /registry/apiregistration.k8s.io/apiservices/v1.

    {
     "kind": "APIService",
     "apiVersion": "apiregistration.k8s.io/v1beta1",
     "metadata": {
      "name": "v1.",
      "uid": "fd15144e-ef8c-4a02–87fc-0fc72c178118",
      "creationTimestamp": "2019–11–22T15:19:43Z",
      "labels": {
       "kube-aggregator.kubernetes.io/automanaged": "onstart"
      }
     },
     "spec": {
      "service": null,
      "version": "v1",
      "groupPriorityMinimum": 18000,
      "versionPriority": 1
     },
     "status": {
      "conditions": [{
       "type": "Available",
       "status": "True",
       "lastTransitionTime": "2019–11–22T15:19:43Z",
       "reason": "Local",
       "message": "Local APIServices are always available"
      }]
     }
    }
```
The above keys are the configuration and status of all the resources within the cluster like:

* Nodes

* Namespaces

* ClusterRoles

* ClusterRoleBindings

* ConfigMaps

* Secrets

* Workloads: Deployments, DaemonSets, Pods

## Summary

This is my personal log on how to connect to the etcd pod inside a kubernetes cluster for debugging purpose in the future. It is interesting to see what information a K8s cluster stores in etcd when running. Helps in learning how to design complex systems in the future.


Please follow [TechLog](https://medium.com/techlog) for more.