Using OpenPolicyAgent to Inject Proxy Configuration into Kubernetes Workloads
Tags: kubernetes, networking, openpolicyagent, k3s
Published: 2021-12-18 06:49:46

---
title: Using OpenPolicyAgent to Inject Proxy Configuration into Kubernetes Workloads
published: true
description: 
tags: kubernetes, networking, openpolicyagent, k3s
//cover_image: https://direct_url_to_image.jpg
---

It's not unusual to find enterprise services behind a corporate proxy, and in order to get out, must have this proxy access configured to access out through it. The same applies to services running o Kubernetes. 

One way to do this is manually, by creating a `Secret` resource containing a proxy string like the following:

```bash
kubectl create secret generic corporate-proxy --from-literal=authentication-string="http://{USERNAME}:{PASSWORD}@{PROXY_ADDRESS}:{PROXY_PORT}" --from-literal=no-proxy-string="localhost,127.0.0.1,0.0.0.0,10.0.0.0/8,cattle-system.svc,10.42.0.0/24,.svc,.cluster.local,example.com"
```
and, mount it to the environment of a Kubernetes application resource:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: appWithProxy
  namespace: default
spec:
  containers:
  - name: appWithProxy
    image: yourApplication
    env:
    - name: HTTP_PROXY
      valueFrom:
        secretKeyRef:
          name: corporate-proxy
          key: authentication-string
    - name: NO_PROXY
      valueFrom:
        secretKeyRef:
          name: corporate-proxy
          key: no-proxy-string
```

and when you `exec` into that container, you can use that proxy to access public resources:

```bash
curl --proxy $HTTP_PROXY http://ipinfo.io/json
```

However, this is a tedious, manual process that you would, then, need to repeat for every resource you'd like to have proxy access.

Among the many things OpenPolicyAgent Gatekeeper does, is allow you to mutate Kubernetes API requests based on a variety of scoping rules, and inject things like environment variables into resources that meet said scope. 

This is done using a Mutating Admission Webhook. Using [Mutating Admission Webhook to inject environment variables like these proxy settings](https://slack.engineering/simple-kubernetes-webhook/), for example, to modify requests to create a resource like a `Pod` or `Deployment`, etc. to inject the proxy configuration data. There are other types of Webhooks, such as (in the linked example) Validating Webhooks, which do things like confirm a workload has a label, or an appropriate name, etc. to dictate some other action by the Admission Webhook.

[OPA Gatekeeper](https://rancher.com/docs/rancher/v2.0-v2.4/en/cluster-admin/tools/opa-gatekeeper/), in this scenario, will do much of this for you, and instead, mutation and scoping is dictated by policies. In our case, we want to create a namespace that will contain resources that we wish to mutate, upon request, to include mounting the above `corporate-proxy` resource `Secret` as an environment variable.

OPA also provides [facilities for testing these policies](https://www.openpolicyagent.org/docs/latest/kubernetes-primer/#testing-policies).

After [installing OPA Gatekeeper](https://open-policy-agent.github.io/gatekeeper/website/docs/install/#using-prebuilt-image):

```bash
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/release-3.7/deploy/gatekeeper.yaml
```
Or if you're a [Rancher user, Gatekeeper can be installed and enabled from the UI](https://rancher.com/docs/rancher/v2.0-v2.4/en/cluster-admin/tools/opa-gatekeeper/).

We can then define a `ModifySet` resource:

```yaml
apiVersion: mutations.gatekeeper.sh/v1beta1
kind: ModifySet
metadata:
  name: demo-annotation-owner
  namespace: proxy-test
spec:
    applyTo:
      - groups: [""]
        versions: ["v1"]
        kinds: ["Pod", "Deployment"]
    match:
      scope: Namespaced
      kinds:
        - apiGroups: ["*"]
          kinds: ["Pod"]
    location: "spec.containers[name:*].env"
    parameters:
      values:
        fromList:
          - name: HTTP_PROXY
            valueFrom:
              secretKeyRef:
                name: corporate-proxy
                key: authentication-string
          - name: NO_PROXY
            valueFrom:
              secretKeyRef:
                name: corporate-proxy
                key: authentication-string
```

which is deployed to the `proxy-test` namespace, and a couple of lines down, the `scope` is set to `Namespaced`, meaning that it will only apply the following to matching resources in that namespace. 

If the API server passes a request that is that namespace, and matches any apiGroup ("*"), and is of `kind: Pod`, it will read a spec like the following:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: shell-demo
  namespace: proxy-test
spec:
  volumes:
  - name: shared-data
    emptyDir: {}
  containers:
  - name: nginx
    image: nginx
    env:
    - name: SOME_EXISTING_VAR
      value: "Present"
    volumeMounts:
    - name: shared-data
      mountPath: /usr/share/nginx/html
  dnsPolicy: Default
```
where it will look in `spec.containers`, and you'll see since this request does not contain an `env` block, it will inject our `Secret` from above (`secret/corporate-proxy`) into this spec appending `env` represented above in Yaml, but can also be inserted as JSON:

```json
[{"name": "HTTP_PROXY", "valueFrom": {"secretKeyRef": {"name": "corporate-proxy", "key": "authentication-string"}}}]
```

and create the `Pod`. 

You can test this out by creating two `Pods` using the above spec, one in the `proxy-test` namespace and one in another:

```bash
cat << EOF >> proxy.yaml
apiVersion: v1
kind: Pod
metadata:
  name: shell-demo
  namespace: proxy-test
spec:
  volumes:
  - name: shared-data
    emptyDir: {}
  containers:
  - name: nginx
    image: nginx
    env:
    - name: SOME_EXISTING_VAR
      value: "Present"
    volumeMounts:
    - name: shared-data
      mountPath: /usr/share/nginx/html
  dnsPolicy: Default
EOF

kubectl apply -f proxy.yaml
```
and one in another:
```bash
cat << EOF >> no-proxy.yaml
apiVersion: v1
kind: Pod
metadata:
  name: shell-demo
  namespace: default
spec:
  volumes:
  - name: shared-data
    emptyDir: {}
  containers:
  - name: nginx
    image: nginx
    env:
    - name: SOME_EXISTING_VAR
      value: "Present"
    volumeMounts:
    - name: shared-data
      mountPath: /usr/share/nginx/html
  dnsPolicy: Default
EOF

kubectl apply -f no-proxy.yaml
```

and when you run `kubectl describe` on the namespaced `Pod`, you'll see the following:

```bash
root@ubuntu-proxy-k3s:~# kubectl describe pod/shell-demo -n proxy-test
Name:         shell-demo
Namespace:    proxy-test
...
Containers:
  nginx:
    Image:      nginx
...
    Environment:
      SOME_EXISTING_VAR:     "Present"
      HTTP_PROXY:  <set to the key 'authentication-string' in secret 'corporate-proxy'>  Optional: false
      NO_PROXY:    <set to the key 'authentication-string' in secret 'corporate-proxy'>  Optional: false
```
where the one in the `default` namespace will not have an Envrionment set. 

These `ModifySet` resources can be written to mutate based on a number of other scopes as well, and any part of a manifest that you specify in the `location` key, and then assign with a `value` under parameters as we did above. Other examples of using OPA Gatekeeper resources to mutate resources can be found [here](https://open-policy-agent.github.io/gatekeeper/website/docs/mutation/#!).