Debugging K8s infrastructure locally with Telepresence
Published: 2020-04-11 20:18:07

#What is Telepresence?#

With so many moving parts and services in the mix, it's no wonder that any engineer who has had to debug a Kubernetes micro-service can be a major pain.  That said, what if it didn't have to be? I would like to introduce you to a CNCF Incubation product created by [Datawire](https://www.datawire.io/) that has caught my interest recently and has really become a handy tool in my cloud-native toolbox.

Until now, Telepresence was a word to describe a sensation of being elsewhere, created by the use of virtual reality technology. This is why I feel like this is such a perfect name for a tool that makes you feel like you are "inside" your Kubernetes cluster while working locally. 

Telepresence is an open-source tool that lets you run a single service locally while connecting that service to a remote Kubernetes cluster. This lets developers working on multi-service applications to:

1. Do fast local development of a single service, even if that service depends on other services in your cluster. Make a change to your service, save, and you can immediately see the new service in action.
2. Use any tool installed locally to test/debug/edit your service. For example, you can use a debugger or IDE!
3. Make your local development machine operate as if it's part of your Kubernetes cluster. If you've got an application on your machine that you want to run against a service in the cluster -- it's easy to do.

#How it works#

Telepresence deploys a two-way network proxy in a pod running in your Kubernetes cluster. This pod proxies data from your Kubernetes environment (e.g., TCP connections, environment variables, volumes) to the local process. The local process has its networking transparently overridden so that DNS calls and TCP connections are routed over the proxy to the remote Kubernetes cluster.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/8no42hpyriqe2wza17nv.png)

This approach gives:

* your local service full access to other services in the remote cluster
* your local service full access to Kubernetes environment variables, secrets, and ConfigMap
* your remote services full access to your local service

#Installing Telepresence#

##OS X##

On OS X you can install Telepresence by running the following:

```
brew cask install osxfuse
brew install datawire/blackbird/telepresence
```

##Ubuntu 16.04 or later##

On Ubuntu 16.04 and up, you can run the following to install Telepresence:

```
curl -s https://packagecloud.io/install/repositories/datawireio/telepresence/script.deb.sh | sudo bash
sudo apt install --no-install-recommends telepresence
```

f you are running another Debian-based distribution that has Python 3.5 installable as `python3`, you may be able to use the Ubuntu 16.04 (Xenial) packages. The following works on Linux Mint 18.2 (Sonya) and Debian 9 (Stretch) by forcing the PackageCloud installer to access Xenial packages:

```
curl -sO https://packagecloud.io/install/repositories/datawireio/telepresence/script.deb.sh
sudo env os=ubuntu dist=xenial bash script.deb.sh
sudo apt install --no-install-recommends telepresence
rm script.deb.sh
```

A similar approach may work on Debian-based distributions with Python 3.6 by using the Ubuntu 17.10 (Artful) packages.

##Fedora 26 or later##

Run the following:
```
curl -s https://packagecloud.io/install/repositories/datawireio/telepresence/script.rpm.sh | sudo bash
sudo dnf install telepresence
```

If you are running a Fedora-based distribution that has Python 3.6 installable as `python3`, you may be able to use Fedora packages. See the Ubuntu section above for information on how to invoke the PackageCloud installer script to force OS and distribution.

##Install from source##

On systems with Python 3.5 or newer, install into `/usr/local/share/telepresence` and `/usr/local/bin` by running:

```
sudo env PREFIX=/usr/local ./install.sh
```

Install the software from the list of dependencies to finish. Install into arbitrary locations by setting other environment variables before calling the install script. After installation, you can safely delete the source code.

##Dependencies##

If you install Telepresence using a pre-built package, dependencies other than `kubectl` are handled by the package manager. If you install from source, you will also need to install the following software.

* `kubectl` (OpenShift users can use `oc`)
* Python 3.5 or newer
* OpenSSH (the `ssh` command)
* `sshfs` to mount the pod's filesystem
* `conntrack` and `iptables` on Linux for the vpn-tcp method
* `torsocks` for the inject-tcp method
* Docker for the container method
* `sudo` to allow Telepresence to
* modify the local network (via `sshuttle` and `pf`/`iptables`) for the vpn-tcp method
* run the `docker` command in some configurations on Linux
* mount the remote filesystem for access in a Docker container

#Getting Started#

Telepresence offers a broad set of [proxying options](https://www.telepresence.io/reference/methods) that have different strengths and weaknesses, but for the sake of this tutorial, we are going to start with the recommended container method, which provides the most consistent environment for your code. 

Now let's learn how to use Telepresence to connect and debug out Kubernetes clusters locally by getting hands-on!

Telepresence allows you to get transparent access to a remote cluster from a local process. This allows you to use your local tools on your laptop to communicate with processes inside the cluster.

##Connecting to the remote cluster##

You should start by running a demo service in your cluster:

```
$ kubectl run myservice --image=datawire/hello-world --port=8000 --expose
$ kubectl get service myservice
NAME        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
myservice   10.0.0.12    <none>        8000/TCP   1m
```

**If your cluster is in the cloud**, you can find the address of the resulting `Service` like this:

```
$ kubectl get service hello-world
NAME          CLUSTER-IP     EXTERNAL-IP       PORT(S)          AGE
hello-world   10.3.242.226   104.197.103.123   8000:30022/TCP   5d
```

If you see `<pending>` under EXTERNAL-IP wait a few seconds and try again. In this case the `Service` is exposed at `http://104.197.103.123:8000/`.

It may take a minute or two for the pod running the server to be up and running, depending on how fast your cluster is.

Once you know the address you can store its value (don't forget to replace this with the real address!):

```
$ export HELLOWORLD=http://104.197.103.13:8000
```

You can now run a local process using Telepresence that can access that service, even though the process is local but the service is running in the Kubernetes cluster:

```
$ telepresence --run curl http://myservice:8000/
Hello, world!
```

(This will not work if the *hello world* pod hasn't started yet... if so, try again.)

What's going on:

1. Telepresence creates a new `Deployment`, which runs a proxy.
2. Telepresence runs `curl` locally in a way that proxies networking through that `Deployment`.
3. The DNS lookup and HTTP request done by `curl` get routed through the proxy and transparently access the cluster... even though `curl` is running locally.
4. When `curl` exits the new `Deployment` will be cleaned up.

**Setting up the proxy**

To use Telepresence with a cluster (Kubernetes or OpenShift, local or remote) you need to run a proxy inside the cluster. There are three ways of doing so.

**Creating a new deployment**

By using the `--new-deployment` option `telepresence` can create a new deployment for you. It will be deleted when the local `telepresence` process exits. This is the default if no deployment option is specified.

For example, this creates a `Deployment` called `myserver`:

```
telepresence --new-deployment myserver --run-shell
```

This will create two Kubernetes objects, a `Deployment` and a `Service`, both named `myserver`. (On OpenShift a `DeploymentConfig` will be used instead of `Deployment`.) Or, if you don't care what your new `Deployment` is called, you can do:

```
telepresence --run-shell
```

**Running Telepresence manually**

You can also choose to run Telepresence manually by starting a `Deployment` that runs the proxy in a pod.

The `Deployment` should only have 1 replica, and use the different Telepresence image:

```
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: myservice
spec:
  replicas: 1  # only one replica
  template:
    metadata:
      labels:
        name: myservice
    spec:
      containers:
      - name: myservice
        image: datawire/telepresence-k8s:0.103  # new image
```

You should apply this file to your cluster:

```
kubectl apply -f telepresence-deployment.yaml
```

Next, you need to run the local Telepresence client on your machine, using `--deployment` to indicate the name of the `Deployment` object whose pod is running `telepresence/datawire-k8s`:

```
telepresence --deployment myservice --run-shell
```

Telepresence will leave the deployment untouched when it exits. When you finish, you should see an output that looks like this: 


![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ei08jo31pv3waicqkzev.png)

And that's it! You're now ready to debug and work with your cluster locally with Telepresence! I hope you enjoyed this introduction and if you want to dive into the product more, check out the docs [here](https://www.telepresence.io/reference.)