Accessing Kubernetes cluster using SSH tunnel
Tags: kubernetes, ssh, gcp, devops
Published: 2022-12-14 10:19:56

---
title: Accessing Kubernetes cluster using SSH tunnel
published: true
date: 2022-12-14 10:20:00 UTC
tags: kubernetes,SSH,GCP,devops
canonical_url: https://banach.net.pl/posts/2022/accessing-kubernetes-cluster-using-ssh-tunnel/
---

Whether we want it or not adoption of Kubernetes is growing. It can be set up as a managed solution (all major cloud providers provide such products) or we can set it up by yourselves. No matter if we select the former or the latter - we would like to make it as secure as it can be. One of the solutions to make the Kubernetes cluster more secure is to hide a control plane (to be more specific - `kube-apiserver`) behind a firewall. That means cluster management is not available from the Internet.

That creates problems with accessing it. We can SSH to a server that is in the same network and run `kubectl` commands from there, but this is a nuisance which we want to avoid. Fortunately, SSH tunnels came with help in that case ([not for the first time!](https://banach.net.pl/posts/2022/how-to-expose-local-service-using-nginx-and-ssh/)) - we can create a tunnel to the server in the same network and pass all traffic to the cluster through it!

To do it we can run the command:

```sh
$ ssh our-gate.example.com -L 16443:10.0.10.2:443
```

In the command, we’re assuming that:

- our server is available with the domain `our-gate.example.com`,
- the cluster is in the same local network as the server and its control plane is available at `10.0.10.2` IP address,
- we want the tunnel to be available at port `16443`.

Now we need to replace IP address / hostname of cluster in `~/.kube/config` to point to local tunnel. To do this we will open aforementioned file and replace put `https://127.0.0.1:16443` in `clusters.cluster.server` property:

```yaml
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: <long string with CA data>
# line below was something like https://our-public-address-of-cluster.example.com or just local IP like https://10.0.10.2 
server: https://127.0.0.1:16443 
name: my-k8s-cluster
```

After that we can try to run a command like `kubectl cluster-info`:

```sh
~ > kubectl cluster-info
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Unable to connect to the server: x509: certificate is valid for our-public-address-of-cluster.example.com, 10.0.10.2, not 127.0.0.1
```

It fails, but why? It is because we want to connect to `127.0.0.1` but API (securely server over https) is responding with a certificate issued for different domain / IP address. Fortunately solution for that is simple which is to put `clusters.cluster.tls-server-name` property with one of the valid values from the error message - in our case it can be `10.0.10.2`.

Our final configuration will look like this:

```yaml
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: <long string with CA data>
server: https://127.0.0.1:16443
tls-server-name: 10.0.10.2
name: my-k8s-cluster
contexts:
- context:
cluster: my-k8s-cluster
user: admin
name: my-k8s-cluster
current-context: my-k8s-cluster
kind: Config
preferences: {}
users:
- name: admin
user:
token: <some-fancy-token>
```

And now running `cluster-info` command will gives us what we expect:

```yaml
~ > kubectl cluster-info
Kubernetes control plane is running at https://10.0.10.2:16443
GLBCDefaultBackend is running at https://10.0.10.2:16443/api/v1/namespaces/kube-system/services/default-http-backend:http/proxy
KubeDNS is running at https://10.0.10.2:16443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://10.0.10.2:16443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
```

Yay! Now we can use `kubectl`, `helm` and other software that uses configuration from `~/.kube/config` from our local machine!

And a small “bonus” (and also describing it to the future myself) - accessing Kubernetes in GCP (GKE) through Identity Aware Proxy:

```sh
$ gcloud compute ssh "some-instance" --zone "$ZONE" --project "$PROJECT" --tunnel-through-iap --ssh-flag="-L 16443:10.0.10.2:443"

```

That is it - the SSH tunnel again saves the day! :-)