How we connect to Kubernetes Pods from GitHub Actions
Tags: kubernetes, github, githubactions
Published: 2020-04-06 18:15:51

---
title: How we connect to Kubernetes Pods from GitHub Actions
tags: kubernetes,github,githubactions
published: true
---

Ever wanted to connect to some Pod or Service running a Kubernetes cluster, but don't want or can't setup an Ingress or Load Balancer to connect to it? 

Only want to connect to it for some local workflow like GitHub Actions instead of opening it wider internet using a tool like [inlets](https://inlets.dev/)?

If you answered yes to both of these questions, then you've got a few options available to you:

* Setup a secure VPN within your network. If you're using a hosted service, there's likely a VPN option available. If not, there's a new and growing ecosystem of VPN and VPN-like options that may suit your purposes, enabled by advances like WireGuard. However this option still has a lot of friction and might be overkill.
* Create something like a [jump server](https://en.wikipedia.org/wiki/Jump_server) to access your internal cluster services.
* Finally, if your Kubernetes API is accessible (which is many managed provider's default), you already have access to a great proxy tool in the form of `kubectl port-forward`.

This article is all about setting up that third option.

## Creating a Service Account

If you're connecting in a non-interactive fashion, through something like a CI/CD workflow, you'll likely need to setup a way for you workflow to authentication with your cluster. A "service account" if you will. Luckily, Kubernetes has the built-in concept of a `ServiceAccount` that is normally used to identify Pods and other cluster-native entities. These `ServiceAccount` resources can be used to authenticate external tools too, as I'll show you.

Creating a `ServiceAccount` is pretty simple, but there are a few things you should consider when setting one up. Firstly, are you trying to only access a specific pod? If so, you'll want to create a `ServiceAccount` in that pod's namespace. Additionally, you'll want to make sure that account only has the permissions necessary to create a port-forward on that pod unless you're planning to use this `ServiceAccount` for more than just port-forwarding.

In this example, we'll say you're trying to access the `Pod` `my-pod` on namespace `default`.

First, let's create the `ServiceAccount` we'll be using:

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-pod-port-forward
  namespace: default
```

This one's pretty straight forward. We want to make make a `ServiceAccount` with the name `my-pod-port-forward` in the namespace `default`. Note you can name it whatever you want, but it should descriptive and make it clear what it's used for or with.

Next, let's create the `Role` with the appropriate permissions that we'll use with this `ServiceAccount`:

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-pod-port-forward
  namespace: default
rules:
- apiGroups: [""]
  resources: ["pods"]
  resourceNames: ["my-pod"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods/portforward"]
  resourceNames: ["my-pod"]
  verbs: ["create"]
```

There's quite a bit here, so I'll break it down without getting too in the nitty-gritty. If you're interested in a deeper dive, I'd check out the [official docs](https://kubernetes.io/docs/reference/access-authn-authz/rbac/).

First, we're creating an RBAC `Role` with the name `my-pod-port-forward` in the namespace `default`. The name can again be anything you like, but it should be descriptive. Then, we're giving this `Role` three sets of permission:

1. The ability to access get our specify pod.
2. The ability to create a `port-forward` for that specific pod.

This is the barest set of permissions necessary to port-forward to a pod.

_Note: If you want to connect to a `Service` or `Deployment`, it gets a bit more complicated. I'll add an example of what's necessary at the end of the article._

Finally, we need to bind our new `Role` to the `ServiceAccount`:

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-pod-port-forward
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: my-pod-port-forward
subjects:
- kind: ServiceAccount
  name: my-pod-port-forward
  namespace: default
```

This one should be pretty straight forward and follows the same naming theme as before.

With all those applied, you now have a `ServiceAccount` you can use! But...how do you go from `ServiceAccount` to `kubectl`?

## Using the Service Account

Whenever you create a service account, it's automatically issued a `Secret` that's used to authenticate with the cluster's API.

You can get the `Secret` associated with a `ServiceAccount` by running:

```bash
TOKENNAME=`kubectl -n default get serviceaccount/my-pod-port-forward -o jsonpath='{.secrets[0].name}'`
```

In that `Secret` is a lot of important information, but the thing we want is the `token`. To get that, you can run:

```bash
kubectl -n default get secret/$TOKENNAME -o jsonpath='{.data.token}' | base64 -d
```

If you've ever worked with JWTs before, you'll likely recognize this. So how do we use this token? To use it, we need to create a kube config file with the appropriate context and user.

Our template:

```yaml
apiVersion: v1
clusters:
- cluster:
    server: <api-server-url>
    certificate-authority-data: <api-server-ca-data>
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: service-account
  name: target-cluster
current-context: target-cluster
kind: Config
preferences: {}
users:
- name: service-account
  user:
    token: <token>
```

You can use your normal means of getting the cluster's API URL and certificate authority, but chances are you're already connected to the cluster you want to target. As such, you can pull these values from your own kube config! For example, to get the various clusters in you config, you can run:

```bash
grep '\- cluster:' -A 3 ${HOME}/.kube/config
```

Set those values and replace `<token>` with the token you got from those previous files, and you've got your self a service account kube config! To use it, let's say you saved it to something like `~/.kube/my-pod-port-forward.yaml`.

The magic command to create the port-forward is:

```bash
kubectl port-forward pod/my-pod 80:80 --kubeconfig ~/.kube/my-pod-port-forward.yaml
```

With that, you should get a message like this:

```text
Forwarding from 127.0.0.1:80 -> 80
Forwarding from [::1]:80 -> 80
```

And that's it! You're now able to connect to your pod locally!

## GitHub Actions (and more)

Now that we've got a service account setup and a kube config made, we can get up and running in Actions! This is pretty simple, but there are a few quirks I'll call out.

First, we want to drop our kube config into our action so that we can authenticate `kubectl`. What I did was base64 encoded the kube config we created before, and then dropped into GitHub Secrets as `KubeConfig`. We're base64'ing it to make it easier to consume in the action as you'll see.

```bash
cat ~/.kube/my-pod-port-forward.yaml | base64 -w 0 # No wrap so it's all one line
```

Now that we have it stored in secret, we can access it from our workflow and use it for kubectl.

```yaml
name: CI
on:
  push:
    branches: [ master ]
jobs:
  build:
    env:
        # Here, we're choosing to place it in the workspace folder
        # in a folder called `.kube` to keep things organized.
        # Note: You could just do KUBECONFIG if you're planning to use
        # this for all other `kubectl` actions in this workflow
        POD_KUBECONFIG: '${{ github.workspace }}/.kube/pod-kubeconfig'
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2

    # Here, we're creating the parent directory and writing out our decoded
    # kubeconfig to the location we stated above.
    - run: |
        mkdir -p '${{ github.workspace }}/.kube' \
          && echo '${{ secrets.KubeConfig}}' | base64 -d > $POD_KUBECONFIG

    # Finally, let's try using it. If you used `KUBECONFIG`, can can remove
    # the `--kubeconfig $POD_KUBECONFIG` as kubectl will automatically use it.
    - run: 'kubectl version --kubeconfig $POD_KUBECONFIG'
```

If you drop that in you `.github/workflows` folder and push it, should see the action run and final step print out both the client `kubectl` and the target cluster's version information!

Now, we need to start the port-forward. Here, we're going to use bash's built-in support for jobs to run the port-forward in the background:

```yaml
    - run: 'kubectl port-forward pod/my-pod 80:80 --kubeconfig $POD_KUBECONFIG &'
```

With that, it should be accessible locally! If you want to try it out, you can add a step like this to see it in action:

```yaml
    - run: curl http://localhost:80
```

And that's it! Hopefully, this works well for you, and just leave a comment below if you have any questions or run into any issues.

### Credits

- [Alex Ellis](https://twitter.com/alexellisuk?s=20) who helped give me through the initial idea of connecting to a cluster through port-forwarding and got me thinking about the options enabled by that. (Also the man behind OpenFaaS and Inlets which you should checkout!)
- [Daniel Albuschat](https://dev.to/danielkun) who has an excellent deeper dive into k8s auth and service accounts, including how to re-issue tokens: {% post https://dev.to/danielkun/kubernetes-certificates-tokens-authentication-and-service-accounts-4fj7 %}
- All the Kubernetes Docs contributors and poor, fellow lost souls on Stack Overflow who helped me generally understand many of the concepts in this article.

### Additional Notes

If you want to access a `Service` or `Deployment` instead of a specific `Pod`, you'll need a slightly different `Role` configuration:

1. Permissions for a `Deployment`.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-pod-port-forward
  namespace: default
rules:
- apiGroups: ["apps/v1"]
  resources: ["deployments"]
  resourceNames: ["<deployment name>"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/portforward"]
  verbs: ["create"]
```

This will be the same `Role` as earlier, but with the addition of `get` for deployments, and the removal of `resourceName` from the `pods` and `pods/portforward` since you wouldn't know the pod name ahead of time.


2. Permissions for a `Service`.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-pod-port-forward
  namespace: default
rules:
- apiGroups: [""]
  resources: ["services"]
  resourceNames: ["<service name>"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/portforward"]
  verbs: ["create"]
```

This will be the same `Role` as earlier, but with the addition of `get` for services, and the removal of `resourceName` from the `pods` and `pods/portforward` since you wouldn't know the pod name ahead of time.