Keycloak on Distroless into Kubernetes
Tags: keycloak, kubernetes, security, docker
Published: 2021-06-01 09:14:27

---
title: Keycloak on Distroless into Kubernetes
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c3klk25l26f7s9mke1xb.jpeg
tags: keycloak, kubernetes, security, docker
series: Keycloak on Distroless
published: true
---

In the previous article, we've seen how to **build** and **run** a **Keycloak** application based on **Distroless** base image. However, this article only used **Docker** to launch containers, and you mostly use an orchestrator to do that. In this article, we will see how to run our previously created image in a **Kubernetes** cluster. 

# Challenges

![lots of challenges](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/olt1rwsnvjbs440k3b80.jpeg)

Using a **Keycloak** image based on **Distroless** requires some adaptation:

* Launching the application require a specific command
* Stopping the application require a specific command
* Probes should be defined 
* Database connection should be provided via env variables
* Configuration file should be mounted by the orchestrator

## Database

Before doing some configuration, we will "install" a PostgreSQL instance inside our cluster. This will be done with this *manifest*:

```yaml
# database.yaml
apiVersion: v1
kind: Service
metadata:
  name: database
  labels:
    app: database
spec:
  ports:
    - name: pg-port
      port: 5432
      targetPort: 5432
      protocol: TCP
  selector:
    app: database
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: database
spec:
  selector:
    matchLabels:
      app: database
  template:
    metadata:
      labels:
        app: database
    spec:
      containers:
        - image: postgres:13.3-alpine
          imagePullPolicy: IfNotPresent
          name: database
          env:
            - name: POSTGRES_USER
              valueFrom:
                configMapKeyRef:
                  name: database
                  key: user
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: database
                  key: password
            - name: POSTGRES_DB
              valueFrom:
                configMapKeyRef:
                  name: database
                  key: name
          lifecycle:
            preStop:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - su - postgres -c "pg_ctl stop -m fast"
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - exec pg_isready -U $POSTGRES_USER -d $POSTGRES_DB -h 127.1 -p 5432
          readinessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - -e
                - exec pg_isready -U $POSTGRES_USER -d $POSTGRES_DB -h 127.1 -p 5432
          ports:
            - name: pg-port
              containerPort: 5432
              protocol: TCP
```

**NOTE**: This manifest is some kind of `hello-world` at PostgreSQL level and **SHOULD NOT** be used for production… because it doesn't manage any state. Every data will be lost if the pod restart. This is just for this example.

## Launching Keycloak

In the previous step, we fetched the `java` command used to launched **Keycloak**. We will reuse it, in the **Kubernetes** manifest. For information, the command was: 

```
java '-D[Standalone]' -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Dorg.jboss.boot.log.file=/opt/jboss/keycloak/standalone/log/server.log -Dlogging.configuration=file:/opt/jboss/keycloak/standalone/configuration/logging.properties -jar /opt/jboss/keycloak/jboss-modules.jar -mp /opt/jboss/keycloak/modules org.jboss.as.standalone -Djboss.home.dir=/opt/jboss/keycloak -Djboss.server.base.dir=/opt/jboss/keycloak/standalone -Djboss.bind.address=172.17.0.2 -Djboss.bind.address.private=172.17.0.2 -c=standalone-ha.xml
```

We will set `$.spec.template.spec.containers[0].command` and `$.spec.template.spec.containers[0].args` to the previously used `ENTRYPOINT` in the `Dockerfile`.

```yaml
# keycloak.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  labels:
    app: keycloak
spec:
  template:
    spec:
      containers:
        - name: keycloak
          # We define Java as the main command of the image.
          command: [ "java" ]
          # We put back all the parameters from the previous step.
          args:
            - "-D[Standalone]"
            - "-server"
            - "-Xms64m"
            - "-Xmx512m"
            - "-XX:MetaspaceSize=96M"
            - "-XX:MaxMetaspaceSize=256m"
            - "-Djava.net.preferIPv4Stack=true"
            - "-Djboss.modules.system.pkgs=org.jboss.byteman"
            - "-Djava.awt.headless=true"
            - "--add-exports=java.base/sun.nio.ch=ALL-UNNAMED"
            - "--add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED"
            - "--add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED"
            - "-Dorg.jboss.boot.log.file=/opt/jboss/keycloak/standalone/log/server.log"
            - "-Dlogging.configuration=file:/opt/jboss/keycloak/standalone/configuration/logging.properties"
            - "-jar"
            - "/opt/jboss/keycloak/jboss-modules.jar"
            - "-mp"
            - "/opt/jboss/keycloak/modules"
            - "org.jboss.as.standalone"
            - "-Djboss.home.dir=/opt/jboss/keycloak"
            - "-Djboss.server.base.dir=/opt/jboss/keycloak/standalone"
            - "-c=standalone.xml"
            - "-b=0.0.0.0"
            - "-bprivate=0.0.0.0"
            - "-bmanagement=0.0.0.0"
```

**NOTE**: Like previously, the parameter **can** and **should** be adapted to your needs, for performance and security reasons. 

## Providing configuration to Keycloak

Another part of the Keycloak configuration is the `standalone.xml`. Our goal here will be to mount the file inside the running container, at the right place, to be used as source of configuration at boot time. 

```yaml
# keycloak.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  labels:
    app: keycloak
spec:
  template:
    spec:
      # We declare a volume containing the configuration of keycloak
      volumes:
        - name: keycloak-config
          # The configMap targeted here will be declared in the next step of the article
          configMap:
            name: keycloak
      containers:
        - name: keycloak
          # And then, we mount the file from the previously seen volume at the 
          # location where keycloak want to find it…  
          volumeMounts:
            - name: keycloak-config
              mountPath: /opt/jboss/keycloak/standalone/configuration/standalone.xml
              subPath: standalone.xml
```

## Probes

In **Kubernetes** world, the orchestrator needs to know when the system is **live** and **ready**. For this, we have to define some **liveness** and **readyness** probes allowing **Kubernetes** to act and react to problem with **Keycloak** if required. Again, this is just some values in our manifest: 

```yaml
# keycloak.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  labels:
    app: keycloak
spec:
  template:
    spec:
      containers:
        - name: keycloak
          # Liveness, if in error, keycloak will be restarted
          livenessProbe:
            httpGet:
                path: /auth/
                port: http
            initialDelaySeconds: 30
            timeoutSeconds: 5
          # Readiness, if in success, traffic will be routed to this pod
          readinessProbe:
            httpGet:
                path: /auth/realms/master
                port: http
            initialDelaySeconds: 30
            timeoutSeconds: 1
```

**NOTE**: RedHat team will introduce more advanced and precise `liveness` and `readyness` probes in future version of Keycloak 😇.

## Lifecycle Pre-Stop Command

To gracefully shutdown **Keycloak**, we usually have to execute a specific shell script. In our case, without any shell, this will be problematic… like for `ENTRYPOINT`, we will extract the `java` command from the shell script and use it in our manifest. 

```shell
# We launch the keycloak original image and launch a bash inside it
$ docker run -it --rm --entrypoint=bash jboss/keycloak:13.0.1
# We use our magic trick to see commands executed by the jboss-cli.sh we will use
bash-4.4$ awk -i inplace 'NR==2 {print "set -x"} 1' /opt/jboss/keycloak/bin/jboss-cli.sh
# Finally, we launch the script with parameters we want to execute in our pre-hook
bash-4.4$ /opt/jboss/keycloak/bin/jboss-cli.sh --connect command=:shutdown --timeout=20
++ dirname /opt/jboss/keycloak/bin/jboss-cli.sh
+ DIRNAME=/opt/jboss/keycloak/bin
+ GREP=grep
+ . /opt/jboss/keycloak/bin/common.sh
++ '[' x = x ']'
++ COMMON_CONF=/opt/jboss/keycloak/bin/common.conf
++ '[' -r /opt/jboss/keycloak/bin/common.conf ']'
+ cygwin=false
+ darwin=false
+ case "`uname`" in
++ uname
+ false
++ cd /opt/jboss/keycloak/bin/..
++ pwd
+ RESOLVED_JBOSS_HOME=/opt/jboss/keycloak
+ '[' x/opt/jboss/keycloak = x ']'
++ cd /opt/jboss/keycloak
++ pwd
+ SANITIZED_JBOSS_HOME=/opt/jboss/keycloak
+ '[' /opt/jboss/keycloak '!=' /opt/jboss/keycloak ']'
+ export JBOSS_HOME
+ '[' x = x ']'
+ JBOSS_MODULEPATH=/opt/jboss/keycloak/modules
+ '[' x = x ']'
+ '[' x '!=' x ']'
+ JAVA=java
+ setDefaultModularJvmOptions
+ setModularJdk
+ java --add-modules=java.se -version
+ MODULAR_JDK=true
+ '[' true = true ']'
++ echo
++ grep '\-\-add\-modules'
+ DEFAULT_MODULAR_JVM_OPTIONS=
+ '[' x = x ']'
+ DEFAULT_MODULAR_JVM_OPTIONS=' --add-exports=java.base/sun.nio.ch=ALL-UNNAMED'
+ DEFAULT_MODULAR_JVM_OPTIONS=' --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED'
+ DEFAULT_MODULAR_JVM_OPTIONS=' --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED'
+ JAVA_OPTS='  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED'
+ false
+ false
+ JAVA_OPTS='  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Djboss.modules.system.pkgs=com.sun.java.swing'
+ JAVA_OPTS='  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Djboss.modules.system.pkgs=com.sun.java.swing -Dcom.ibm.jsse2.overrideDefaultTLS=true'
++ eval echo '"/opt/jboss/keycloak/modules"'
+++ echo /opt/jboss/keycloak/modules
+ JBOSS_MODULEPATH=/opt/jboss/keycloak/modules
++ echo --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Djboss.modules.system.pkgs=com.sun.java.swing -Dcom.ibm.jsse2.overrideDefaultTLS=true
++ grep logging.configuration
+ LOG_CONF=
+ '[' x = x ']'
+ exec java --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Djboss.modules.system.pkgs=com.sun.java.swing -Dcom.ibm.jsse2.overrideDefaultTLS=true -Dlogging.configuration=file:/opt/jboss/keycloak/bin/jboss-cli-logging.properties -jar /opt/jboss/keycloak/jboss-modules.jar -mp /opt/jboss/keycloak/modules org.jboss.as.cli --connect command=:shutdown --timeout=20
Failed to connect to the controller: The controller is not available at localhost:9990: java.net.ConnectException: WFLYPRT0053: Could not connect to remote+http://localhost:9990. The connection failed: WFLYPRT0053: Could not connect to remote+http://localhost:9990. The connection failed: Connection refused
```

The execution ends in error, which is normal because, in this case, no Keycloak instance are running. The `java` command is displayed, starting with `+ exec java`. All of this can be moved into our **yaml** manifest:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  labels:
    app: keycloak
spec:
  template:
    spec:
      containers:
        - name: keycloak
          lifecycle:
            preStop:
              exec:
                command:
                  - "java"
                  - '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED'
                  - '--add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED'
                  - '--add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED'
                  - '-Djboss.modules.system.pkgs=com.sun.java.swing'
                  - '-Dcom.ibm.jsse2.overrideDefaultTLS=true'
                  - '-Dlogging.configuration=file:/opt/jboss/keycloak/bin/jboss-cli-logging.properties'
                  - "-jar"
                  - "/opt/jboss/keycloak/jboss-modules.jar"
                  - "-mp"
                  - "/opt/jboss/keycloak/modules"
                  - "org.jboss.as.cli"
                  - "--connect"
                  - '--commands=shutdown --timeout=20'
```

## Environment Configuration

Finally, we have to provide all the configuration to **Keycloak**. Until now, we only define **how** the value will be linked but not **which** value at all. To do that, we will first include some environment values into the Keycloak manifest:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  labels:
    app: keycloak
spec:
  template:
    spec:
      containers:
        - name: keycloak
          env:
            - name: DB_ADDR
              value: database
            - name: DB_DATABASE
              valueFrom:
                configMapKeyRef:
                  name: database
                  key: name
            - name: DB_USER
              valueFrom:
                configMapKeyRef:
                  name: database
                  key: user
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: database
                  key: password
```

Here, we are mainly targeting values required for database connection. To provide it to the deployment, we will use a `kustomization.yaml` file. It will gather every manifests presented until now and add concret values:

```yaml
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Install everything in the kubernetes namespace named Keycloak
namespace: keycloak

# Load both keycloak.yaml and database.yaml
resources:
  - keycloak.yaml
  - database.yaml

# Generate two ConfigMap
configMapGenerator:
  # One for Keycloak, containing the standalone.xml
  - name: keycloak
    files:
      - standalone.xml
  # One for database, containing user and name
  - name: database
    literals:
      - user=keycloak
      - name=keycloak

# Generate a Secret
secretGenerator:
  # For the database, containing just the password of the database
  - name: database
    literals:
      - password=sPCwZjuq8CMvrBn7
```

**NOTE**: The password is here stored in clear, you can choose to use some tooling to encrypt the secret at git level (with [`SOPS`](https://dev.to/stack-labs/manage-your-secrets-in-git-with-sops-g0a)) or cluster level (with [`SealedSecret`](https://dev.to/stack-labs/store-your-kubernetes-secrets-in-git-thanks-to-kubeseal-hello-sealedsecret-2i6h)).

## Deploying Keycloak

In this step, we will deploy our version of **Keycloak** in a **Kubernetes** Cluster. For this example, I choose to use `docker-for-mac` and its **Kubernetes** integration. You can, of course, use any **Kubernetes** distribution (Google Kubernetes Engine, Azure Kubernetes Service, Amazon Elastic Kubernetes Service…). 

```shell
$ ls
database.yaml  keycloak.yaml  kustomization.yaml  standalone.xml
# We deploy all the manifests
$ kubectl apply -k .
configmap/database-56h9f7gfdh created
configmap/keycloak-96k2tfg747 created
secret/database-8g8gk22d26 created
service/database created
service/keycloak created
deployment.apps/database created
deployment.apps/keycloak created
# We create a port-forward to the keycloak pod
$ kubectl -n keycloak  port-forward pod/keycloak-567797b6bb-6vqvz 8080:8080
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
```

To access it, we create a `port-forward` between my computer and the cluster. Thanks to that, I'm able to access the UI:

![creation-user](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2y5wuyg2708ea054zczy.png)
![creation-user-in-progress](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4ildfow8j87bsmrqplpz.png)
![login](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/92fj38bjluyd2ghirtrj.png)
![keycloak ui](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2vomubefzab1darab7ny.png)

And **Voila**!

# Conclusion

In this article, we've seen how to deploy our custom **Keycloak** based on **Distroless Java**. This is a challenging setup, but at the end we can use a more secure version of **Keycloak** without any `shell`, which prevent any attack using `shell` as a vector of code execution for example!

I hope you liked it, you can find all the sample files from this article in this **GitLab** repository: [davinkevin/keycloak-distroless](https://gitlab.com/davinkevin/keycloak-distroless).