Kubernetes: Helm  - "x509: certificate signed by unknown authority", and ServiceAccount for Pod
Tags: security, kubernetes, devops
Published: 2021-10-01 07:53:19

---
title: Kubernetes: Helm  - "x509: certificate signed by unknown authority", and ServiceAccount for Pod
published: true
date: 2021-09-29 06:46:09 UTC
tags: security,kubernetes,devops
canonical_url: https://rtfm.co.ua/en/kubernetes-helm-x509-certificate-signed-by-unknown-authority-and-serviceaccount-for-pod/
---

### Kubernetes: Helm — “x509: certificate signed by unknown authority”, and ServiceAccount for Pod

![](https://cdn-images-1.medium.com/max/1024/1*5-LJw8RG96qMMxPZeKcV5w.png)

We have Github runners in our AWS Elastic Kubernetes service cluster, that are used to build Docker images and deploy them with Helm or ArgoCD.

On the first helm install run in a Github runner's Pod, we are getting the " **_x509: certificate signed by unknown authority_**" error:

```
$ helm --kube-apiserver=https://kubernetes.default.svc.cluster.local list
Error: Kubernetes cluster unreachable: Get “https://kubernetes.default.svc.cluster.local/version?timeout=32s": x509: certificate signed by unknown authority
```

Or, if not set an API URL, then we can see a permissions error:

```
$ helm list
Error: list: failed to list: secrets is forbidden: User “system:serviceaccount:dev-1–18-backend-github-runners-helm-ns:default” cannot list resource “secrets” in API group “” in the namespace “dev-1–18-backend-github-runners-helm-ns”
```

So, the cause is obvious: Helm in the Pod is trying to access the Kubernetes API server using the _default_ ServiceAccount, that was created during Github runner deployment.

### Checking a ServiceAccount’s permissions

As already discussed in the [Kubernetes: ServiceAccounts, JWT-tokens, authentication, and RBAC authorization](https://rtfm.co.ua/en/kubernetes-serviceaccounts-jwt-tokens-authentication-and-rbac-authorization/) post, to authenticate on an API server we need to have its Certificate Authority key and a token.

Connect to the pod:

```
$ kubectl -n dev-1–18-backend-github-runners-helm-ns exec -ti actions-runner-deployment-7f78968949-tmrtt bash
Defaulting container name to runner.
Use ‘kubectl describe pod/actions-runner-deployment-7f78968949-tmrtt -n dev-1–18-backend-github-runners-helm-ns’ to see all of the containers in this pod.
root@actions-runner-deployment-7f78968949-tmrtt:/actions-runner#
```

Set variables:

```
root@actions-runner-deployment-7f78968949-tmrtt:/actions-runner# CA_CERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

root@actions-runner-deployment-7f78968949-tmrtt:/actions-runner# TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

root@actions-runner-deployment-7f78968949-tmrtt:/actions-runner# NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)
```

And try to access API now:

```
root@actions-runner-deployment-7f78968949-tmrtt:/actions-runner# curl -s — cacert $CA_CERT -H “Authorization: Bearer $TOKEN” “https://kubernetes.default.svc.cluster.local/api/v1/namespaces/$NAMESPACE/pods" | jq ‘{message, code}’
{
“message”: “pods is forbidden: User \”system:serviceaccount:dev-1–18-backend-github-runners-helm-ns:default\” cannot list resource \”pods\” in API group \”\” in the namespace \”dev-1–18-backend-github-runners-helm-ns\””,
“code”: 403
}
```

A similar error you’ll get if try to run the `kubectl get pod`:

```
root@actions-runner-deployment-7f78968949-tmrtt:/actions-runner# kubectl get pod
Error from server (Forbidden): pods is forbidden: User “system:serviceaccount:dev-1–18-backend-github-runners-helm-ns:default” cannot list resource “pods” in API group “” in the namespace “dev-1–18-backend-github-runners-helm-ns”
```

Okay, and what we can do here?

### Creating a Kubernetes ServiceAccount for a Kubernetes Pod

Actually, instead of mounting the default ServiceAccount to the pod, we can create our own with permissions set with Kubernetes RBAC Role amd Kubernetes RBAC RoleBinding.

#### Creating an RBAC Role

We can use a pre-defined role from the list here — [User-facing roles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles), for example, _cluster-admin_:

```
$ kubectl get clusterrole cluster-admin
NAME CREATED AT
cluster-admin 2020–11–27T14:44:52Z
```

Or, we can write our own with strict limitations. For example, let’s grant access to the _verbs_ `get` and `list` for the _resources_ `pods`:

```
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
   name: "github-runner-deployer-role"
rules:
 - apiGroups: [""]
   resources: ["pods"]
   verbs: ["get", "list"]
```

Create that role in a necessary Kubernetes Namespace:

```
$ kubectl -n dev-1–18-backend-github-runners-helm-ns apply -f github-runner-deployer-role.yaml
role.rbac.authorization.k8s.io/github-runner-deployer-role created
```

Check it:

```
$ kubectl -n dev-1–18-backend-github-runners-helm-ns get role
NAME CREATED AT
github-runner-deployer-role 2021–09–28T08:05:20Z
```

#### Creating a ServiceAccount

Next, create a new ServiceAccount in the same namespace:

```
apiVersion: v1
kind: ServiceAccount
metadata:
  name: "github-runner-deployer-sa"
```

Apply it:

```
$ kubectl -n dev-1–18-backend-github-runners-helm-ns apply -f github-runner-deployer-sa.yaml
```

#### Creating a RoleBinding

Now, we need to create a _binding _— a connection between this ServiceAccount and the role, which was created above.

Create a RoleBinding:

```
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
   name: "github-runner-deployer-rolebinding"
subjects:
 - kind: ServiceAccount
   name: "github-runner-deployer-sa"
   namespace: "dev-1-18-backend-github-runners-helm-ns"
roleRef:
   kind: Role
   name: "github-runner-deployer-role"
   apiGroup: rbac.authorization.k8s.io
```

Apply it:

```
$ kubectl -n dev-1–18-backend-github-runners-helm-ns apply -f github-runner-deployer-rolebinding.yaml
rolebinding.rbac.authorization.k8s.io/github-runner-deployer-rolebinding created
```

#### Mounting the ServiceAccount to a Kubernetes Pod

And the final thing is to attach this ServiceAccount to our pods.

First, let’s do it manually to check, without a Deployment:

```
apiVersion: v1
kind: Pod
metadata:
  name: "github-runners-deployer-pod"
spec:
  containers:
    - name: "github-runners-deployer"
      image: "nginx"
      ports:
        - name: "web"
          containerPort: 80
          protocol: TCP
  serviceAccountName: "github-runner-deployer-sa"
```

Create this Pod:

```
$ kubectl -n dev-1–18-backend-github-runners-helm-ns apply -f github-runner-deployer-pod.yaml
pod/github-runners-deployer-pod created
```

Connect to it:

```
$ kubectl -n dev-1–18-backend-github-runners-helm-ns exec -ti github-runners-deployer-pod bash
root@github-runners-deployer-pod:/#
```

And check API access:

```
root@github-runners-deployer-pod:/# CA_CERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

root@github-runners-deployer-pod:/# TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

root@github-runners-deployer-pod:/# NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)


root@github-runners-deployer-pod:/# curl -s --cacert $CA_CERT -H “Authorization: Bearer $TOKEN” “https://kubernetes.default.svc.cluster.local/api/v1/namespaces/$NAMESPACE/pods"
{
“kind”: “PodList”,
“apiVersion”: “v1”,
“metadata”: {
“selfLink”: “/api/v1/namespaces/dev-1–18-backend-github-runners-helm-ns/pods”,
“resourceVersion”: “251020450”
},
“items”: [
{
“metadata”: {
“name”: “actions-runner-deployment-7f78968949-jsh6l”,
“generateName”: “actions-runner-deployment-7f78968949-”,
“namespace”: “dev-1–18-backend-github-runners-helm-ns”,
…
```

Now we are able to use the `get` and `list` for Pods, but not for the Secrets, as we didn't add this permission:

```
root@github-runners-deployer-pod:/# curl -s — cacert $CA_CERT -H “Authorization: Bearer $TOKEN” “https://kubernetes.default.svc.cluster.local/api/v1/namespaces/$NAMESPACE/secrets"
{
“kind”: “Status”,
“apiVersion”: “v1”,
“metadata”: {
},
“status”: “Failure”,
“message”: “secrets is forbidden: User \”system:serviceaccount:dev-1–18-backend-github-runners-helm-ns:github-runner-deployer-sa\” cannot list resource \”secrets\” in API group \”\” in the namespace \”dev-1–18-backend-github-runners-helm-ns\””,
…
```

### Helm ServiceAccount

Let’s go back to our Github Runners and helm in there.

So, as this Helm will be used to deploy anything and everywhere in the EKS cluster, we can give it admin access by using the _cluster-admin_ ClusterRole.

Delete RoleBinding created before:

```
$ kubectl -n dev-1–18-backend-github-runners-helm-ns delete rolebinding github-runner-deployer-rolebinding
rolebinding.rbac.authorization.k8s.io “github-runner-deployer-rolebinding” deleted
```

Create a new one, but at this time by using the ClusterRoleBinding type to give access to the whole cluster:

```
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
   name: "github-runner-deployer-cluster-rolebinding"
subjects:
 - kind: ServiceAccount
   name: "github-runner-deployer-sa"
   namespace: "dev-1-18-backend-github-runners-helm-ns"
roleRef:
   kind: ClusterRole
   name: "cluster-admin"
   apiGroup: rbac.authorization.k8s.io
```

Attach this ServiceAccoun to this ClusterRole:

```
$ kubectl -n dev-1–18-backend-github-runners-helm-ns apply -f github-runner-deployer-clusterrolebinding.yaml
clusterrolebinding.rbac.authorization.k8s.io/github-runner-deployer-rolebinding created
```

Go back to the Pod, and try to access a Kubernetes Secrets now:

```
root@github-runners-deployer-pod:/# curl -s --cacert $CA_CERT -H “Authorization: Bearer $TOKEN” “https://kubernetes.default.svc.cluster.local/api/v1/namespaces/$NAMESPACE/secrets"
{
“kind”: “SecretList”,
“apiVersion”: “v1”,
“metadata”: {
“selfLink”: “/api/v1/namespaces/dev-1–18-backend-github-runners-helm-ns/secrets”,
“resourceVersion”: “251027845”
},
“items”: [
{
“metadata”: {
“name”: “bttrm-docker-secret”,
“namespace”: “dev-1–18-backend-github-runners-helm-ns”,
“selfLink”: “/api/v1/namespaces/dev-1–18-backend-github-runners-helm-ns/secrets/bttrm-docker-secret”,
…
```

Nice — now our Helm has full access to the cluster!

And let’s update the Github Runners Deployment to use this ServiceAccount.

Edit it:

```
$ kubectl -n dev-1–18-backend-github-runners-helm-ns edit deploy actions-runner-deployment
```

Set a new ServiceAccount by adding the serviceAccount:

![](https://cdn-images-1.medium.com/max/459/0*cOK55jKXRj3VQN7s.png)

Wait for new pods will be created (check the [Kubernetes: ConfigMap and Secrets — data auto-reload in pods](https://rtfm.co.ua/en/kubernetes-configmap-and-secrets-data-auto-reload-in-pods/)), connect and check the Pod’s permissions now:

```
root@actions-runner-deployment-6dfc9b457f-mc7rt:/actions-runner# kubectl auth can-i list pods
yes
```

And check access to another namespace, as we’ve created the ClusterRoleBinding, which is applied to the whole cluster:

```
root@actions-runner-deployment-6dfc9b457f-mc7rt:/actions-runner# kubectl auth can-i list pods — namespace istio-system
yes
```

Good — and we have access to the _istio-system_ Namespace too.

And check the Helm:

```
root@actions-runner-deployment-6dfc9b457f-mc7rt:/actions-runner# helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
github-runners dev-1–18-backend-github-runners-helm-ns 2 2021–09–22 19:50:18.828686642 +0300 +0300 deployed github-runners-1632329415 v1.0.0
```

Done.

_Originally published at_ [_RTFM: Linux, DevOps, and system administration_](https://rtfm.co.ua/en/kubernetes-helm-x509-certificate-signed-by-unknown-authority-and-serviceaccount-for-pod/)_._

* * *