Continuous Development with Java and Kubernetes
Tags: kubernetes, java, jib, skaffold
Published: 2019-03-01 12:26:04

---
title: Continuous Development with Java and Kubernetes
published: true
description: Continuous Development with Java and Kubernetes with the help of Jib and Skaffold
tags: kubernetes, java, jib, skaffold
cover_image: https://thepracticaldev.s3.amazonaws.com/i/g396wvqmv4erccd4m0p8.png
---
If you are developing multiple applications in the same time for Kubernetes, you will realise that running and debugging these during the daily work is very sophisticated and time consuming, since we have to repeat the following steps:

 - Building the application
 - Building the docker image
 - Pushing the container image to a docker registry
 - Creating/Refreshing the Kubernetes' objects

It's enough to have a database and an MQ connection in addition, and you are in a serious trouble. I'm not saying that It's impossible to test everything together, but I'm absolutely sure that if you need to do the steps above by hand, it will break your productive flow.

Fortunately Google [recently announced Jib 1.0.0](https://cloud.google.com/blog/products/application-development/jib-1-0-0-is-ga-building-java-docker-images-has-never-been-easier) which combined with [Skaffold](https://github.com/GoogleContainerTools/skaffold) able to solve this problem. In this article I'm going to show you a workflow with these tools alongside Spring Boot and [Helm](https://helm.sh/).

### A Spring Boot application

First we need to have a basic Spring Boot application. I suggest to use [Spring Initiallizr](https://start.spring.io/) which can help you bootstrap a Spring Boot application in a few seconds. In this article I'm using Spring Boot 2.1.3 with gradle, however you will find the maven related files [in the repository](https://github.com/Pozo/continuous-java-kubernetes) too.

Just download the generated project and let's create a `RestController`

```java
@RestController
public class TestController {

    @GetMapping(value = "/echo/{text}")
    public ResponseEntity test(@NotNull @PathVariable String text) {
        return ResponseEntity.ok(text);
    }

    @GetMapping
    public ResponseEntity hello() {
        return ResponseEntity.ok("HELLO");
    }
}

```
During the article I want to demonstrate Jib's customizability, so let's change the default port number to `55000`. Create an `application.properties` and put this line into the file

```properties
server.port=55000
```
Open your favorite shell and run `./gradlew bootRun`. If both `http://localhost:55000` and `http://localhost:55000/echo/test` seems to work, let's continue with the containerisation phase.

### Using Jib for docker image creation

Jib is [implemented in Java](https://cloudplatform.googleblog.com/2018/07/introducing-jib-build-java-docker-images-better.html) and runs as part of your Maven or Gradle build. You do not need to maintain a Dockerfile, run a Docker daemon. You just need to open your `build.gradle` file, append the plugins section with `id 'com.google.cloud.tools.jib' version '1.0.0'` and add the following piece of code to the end 
```groovy
jib {
    to {
        image = 'com.github.pozo/spring-boot-jib'
    }
    container {
        ports = ['55000']
    }
}
```
It should be noted that the configuration above is not mandatory. However, without these lines Jib would produce an image named `spring-boot-jib:0.0.1-SNAPSHOT` where the first part is the value of the `rootProject.name` from `settings.gradle` and the second one is provided by the `version` variable from `build.gradle`. In order to build a more advanced image, I suggest to look over the [available options](https://github.com/GoogleContainerTools/jib/tree/master/jib-gradle-plugin#extended-usage).

The above configuration’ equal would look like [this](https://github.com/GoogleContainerTools/jib/blob/master/docs/faq.md#what-would-a-dockerfile-for-a-jib-built-image-look-like) with Dockerfile
```docker
FROM gcr.io/distroless/java:latest

COPY dependencyJars /app/libs
COPY snapshotDependencyJars /app/libs
COPY resources /app/resources
COPY classFiles /app/classes

COPY src/main/jib /

ENTRYPOINT ["java", jib.container.jvmFlags, "-cp", "/app/resources:/app/classes:/app/libs/*", jib.container.mainClass]
CMD [jib.container.args]
```
Jib uses [distroless](https://github.com/GoogleContainerTools/distroless) java as the default base image, which seems to use [container related JVM flags by default](https://github.com/GoogleContainerTools/distroless/pull/295). The multiple copy statements are used to break the app into layers, allowing for faster rebuilds after small changes. 

If you want to specify a different base image, just add a [`from`](https://github.com/GoogleContainerTools/jib/tree/master/jib-gradle-plugin#extended-usage) section
```groovy
jib {
    from {
        image = 'java:8-jre-alpine'
    }
    to {
        image = 'com.github.pozo/spring-boot-jib'
    }
    container {
        ports = ['55000']
    }
}
```

If we want to build the container image we have two options. Using [`jib`](https://github.com/GoogleContainerTools/jib/tree/master/jib-gradle-plugin#build-your-image) task which pushes a container image for your application to a container registry, or [`jibDockerBuild`](https://github.com/GoogleContainerTools/jib/tree/master/jib-gradle-plugin#build-to-docker-daemon) which uses the docker command line tool and requires that you have docker available on your PATH or you can set the executable location via the [`dockerClient`](https://github.com/GoogleContainerTools/jib/tree/master/jib-gradle-plugin#extended-usage) object.

Open a terminal and execute
```bash
./gradlew jibDockerBuild
```
I'm going to explain why we need to use this task instead of`jib` in the next paragraph.

Check the output of `docker images` command. If you wonder why your image created by ~49 years ago, It's [for reproducibility purposes](https://github.com/GoogleContainerTools/jib/blob/master/docs/faq.md#why-is-my-image-created-48-years-ago), Jib sets the creation time of the container images to 0. In order to use the current time just add `useCurrentTimestamp = true` inside of the `jib.container`. For more advanced questions check out the [Jib's FAQ](https://github.com/GoogleContainerTools/jib/blob/master/docs/faq.md).

After the build phase we can run our image with
```bash
docker run -p 8080:55000 com.github.pozo/spring-boot-jib
```
If both `http://localhost:8080` and `http://localhost:8080/echo/test` seems to work, we can continue with our Kubernetes objects.

### Running the application image inside of Kubernetes

If you are not familiar with Kubernetes I recommend to start with the [official documentation](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/).

First of all we need an up and running Kubernetes cluster. Fortunately we have several options nowadays. For instance we can use [Minikube](https://kubernetes.io/docs/setup/minikube/) which runs a [single-node Kubernetes cluster](https://kubernetes.io/docs/setup/minikube/) inside a VM on your laptop.

An another option is [Docker for Desktop Kubernetes](https://blog.docker.com/2018/07/kubernetes-is-now-available-in-docker-desktop-stable-channel/). It runs a [single-node cluster](https://docs.docker.com/v17.09/docker-for-mac/kubernetes/) locally **within** your Docker instance. I believe this one is the most comfortable way to hack around with Kubernetes, so I suggest to use this one. 

We also need [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) which is a command line interface for [running commands](https://kubernetes.io/docs/reference/kubectl/overview/) against Kubernetes clusters.

If you decided to use Minikube then start the cluster with
```
minikube start
```
This command [creates and configures](https://kubernetes.io/docs/setup/minikube/#starting-a-cluster) a Virtual Machine that runs a single-node cluster. This command also configures your kubectl installation to communicate with this cluster. You also need to run 
```
eval $(minikube docker-env)
```
The command `minikube docker-env` returns a set of Bash environment variable exports to configure your local environment to re-use the Docker daemon inside the Minikube instance. [(source)](https://stackoverflow.com/a/52310892)

This means you [don't have to build on your host machine](https://github.com/kubernetes/minikube/blob/0c616a6b42b28a1aab8397f5a9061f8ebbd9f3d9/README.md#reusing-the-docker-daemon) and push the image into a docker registry, you can just build inside the same docker daemon as Minikube. So after
```
./gradlew jibDockerBuild
```
Minikube will be able to reach out the image `com.github.pozo/spring-boot-jib`. Don't forget to run `eval $(minikube docker-env)` every time when you prompt a new terminal.

To enable the Kubernetes cluster in case of Docker for Desktop just follow the [platform specific instructions](https://docs.docker.com/docker-for-mac/#kubernetes).

To run our image in the cluster we need to define a [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) first. Deployment represent a set of multiple, identical [Pods](https://kubernetes.io/docs/concepts/workloads/pods/pod/) with no unique identities. A Deployment runs multiple replicas of your application and automatically replaces any instances that fail or become unresponsive. In addition we need an externally exposed [Service](https://kubernetes.io/docs/concepts/services-networking/service/) whereby we can reach our application from outside of the cluster. Here is a diagram about what we want to achieve
![A Kubernetes Service and Deployment](https://thepracticaldev.s3.amazonaws.com/i/65s6556zsypdlwx39dxa.png)

Let's Create a `kubernetes` directory under the project's root and create a file `spring-boot-jib.yaml` with the following content under the freshly created directory

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: spring-boot-jib
  template:
    metadata:
      labels:
        app: spring-boot-jib
    spec:
      containers:
        - name: spring-boot-jib-pod
          image: com.github.pozo/spring-boot-jib
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 55000
---
apiVersion: v1
kind: Service
metadata:
  name: spring-boot-jib-service
spec:
  type: LoadBalancer
  ports:
    - port: 8080
      targetPort: 55000
      protocol: TCP
      name: http
  selector:
    app: spring-boot-jib

```
A few important things to mention here. 
 - The Deployment's `spec.selector.matchLabels.app` value must be the same as the `spec.template.metadata.labels.app` value
 - Service's `spec.selector.app` value must be the same as the Deployment's `spec.selector.matchLabels.app` value. So the service can find our Pod, and hand over every request to it.
 - The Service's `spec.ports.port` value should be what we want to expose to the outside world, in our case `8080`
 - The Service's `spec.ports.targetPort` must be the same as the Deployment's `spec.template.spec.containers.ports.containerPort` so the Service will redirect everything to the container's port number `55000`.
 - Finally the Pod's `spec.containers.imagePullPolicy` **must be `IfNotPresent` or `Never`**. The default value `Always` would produce an error, since there is no such `com.github.pozo`repository exist. 

As we previously used `jibDockerBuild` we have our image locally, and because of Docker for Desktop's cluster uses our host's docker instance, it will able reach the image by default. In case of Minikube, due to `eval $(minikube docker-env)` the image built by Minikube's docker daemon, and it will able reach the image too.

Open a terminal and run
```bash
kubectl create -f kubernetes/
```
The [create](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#create) command iterates over the `kubernetes` directory and creates Kubernetes resources from it's content.

Open a browser and try to reach `http://localhost:8080` and `http://localhost:8080/echo/test` again. In case of a Minikube execute `minikube service list` to find out the deployed service address. If we are getting status code `200` then we did a great job, and we have a running application in our cluster.

### Put everything together with Skaffold

We almost crossed the finish line! Currently we have a containerised application and we can deploy it anytime into the cluster _by hand_. At this point we must repeat the building and deploying steps after every changes. 

Skaffold going to help us to eliminate this handwork. Go to [their website](https://skaffold.dev/docs/getting-started/#installing-skaffold) and follow the installation instructions. If everything set, create a `skaffold.yaml` into the project's root directory with the following content

```yaml
apiVersion: skaffold/v1beta4
kind: Config
build:
  local:
    push: false
  artifacts:
    - image: com.github.pozo/spring-boot-jib
      jibGradle: {}
deploy:
  kubectl:
    manifests:
      - kubernetes/*.yaml
```
This is our brand new [Skaffold pipeline file](https://skaffold.dev/docs/concepts/#configuration-of-the-skaffold-pipeline-skaffold-yaml).
 - The `build.local.push: false` [enforces Skaffold](https://skaffold.dev/docs/how-tos/builders/) to use `jibDockerBuild`.
 - The [`build.artifacts`](https://skaffold.dev/docs/how-tos/builders/#dockerfile-locally-with-docker) is a list of the actual images we're going to be build. 
 - The `build.artifacts.jibGradle` configures Skaffold to use Jib during the image building phase. 
 - The [`deploy.kubectl.manifests`](https://skaffold.dev/docs/how-tos/deployers/#deploying-with-kubectl) value set the folder name where we already have our kubernetes manifest files. If we skip this the default directory name would be [`k8s`](https://github.com/GoogleContainerTools/skaffold/blob/cff3918df8c165f2c4bb17b07f3ed50222f6a63c/pkg/skaffold/constants/constants.go#L79). 

If you are looking for more advanced pipeline configuration I recommend to check out [this well annotated example](https://github.com/GoogleContainerTools/skaffold/blob/b9c979d7b59a15bba9e4a9c7977ece407b21d0c4/integration/examples/annotated-skaffold.yaml).

Open a terminal and run
```bash
skaffold dev --trigger notify
```
This command runs our pipeline file in development mode, which means every change in our codebase will trigger Skaffold to call Jib to build the image, and kubectl to deploy it. Sounds cool right ?

Change the return value of `TestController`'s `hello` method to `"GOODBYE"` and see what happens in the terminal. Refresh the browser after a few seconds, and you will see `"GOODBYE"` instead of `"HELLO"`.
 
### Using helm

If you are not familiar with helm I suggest to take a look at the [official quick starter guide](https://docs.helm.sh/using_helm/#quickstart) first.

I must admit It's absolutely not mandatory to use [Helm](https://docs.helm.sh/using_helm/#quickstart) for development, and someone suggest you to [think twice before using it](https://medium.com/virtuslab/think-twice-before-using-helm-25fbb18bc822), however according to my experience helm making the application deployment easy, standardised and reusable, especially when you have to work with several applications.

Create a [helm chart](https://helm.sh/docs/developing_charts/) for our application with
```bash
helm create spring-boot-jib
```
The helm's [create command](https://docs.helm.sh/helm/#helm-create) will generate a directory structure with several files. In order to make it cleaner what do we have in this folder, rename it to `helm`. The most important files are in the templates directory and the `values.yaml` itself. 

Change the generated `service.yaml`'s `spec.ports.targetPort` to
```yaml
targetPort: {{ .Values.service.containerPort }}
```
The `deployment.yaml`'s `spec.template.spec.containers.image` value to 
```yaml
image: "{{ .Values.image.repository }}{{ if .Values.image.tag }}:{{ .Values.image.tag }}{{ end }}"
```
And the `deployment.yaml`'s `spec.template.spec.containers.ports` value to 
```yaml
ports:
  {{- toYaml .Values.container.ports | nindent 12 }}
```

Everything between `{{ }}` came from the `values.yaml` or `_helper.tpl` files. In fact we are using [Go templating](https://helm.sh/docs/chart_template_guide/). And change the `values.yaml` file like this

```yaml
replicaCount: 1

image:
  repository: com.github.pozo/spring-boot-jib
  tag: latest
  pullPolicy: IfNotPresent

nameOverride: ""
fullnameOverride: "spring-boot-jib"

service:
  type: LoadBalancer
  port: 8080
  containerPort: 55000

container:
  ports:
    - name: http
      containerPort: 55000
      protocol: TCP
```
As we want to use Helm instead of kubectl, we need to adjust the Skaffold pipeline accordingly
```yaml
apiVersion: skaffold/v1beta4
kind: Config
build:
  local:
    push: false
  artifacts:
    - image: com.github.pozo/spring-boot-jib
      jibGradle: {}
deploy:
  helm:
    releases:
      - name: spring-boot-jib
        chartPath: helm
        values:
          image.repository: com.github.pozo/spring-boot-jib
        setValues:
          image.tag: ""

```
We must configure the `chartPath`, the `image.repository`, and we must [set `image.tag` value to empty string](https://stackoverflow.com/a/50936738) for Helm, so Skaffold will be able to manage custom tags on the deployment.

If everything set, run 
```bash
skaffold dev --trigger notify
```
If we are using Minikube we don't need to execute `eval $(minikube docker-env)` anymore, since Skaffold will take care of it. If you want to see what happens under the hood, just add the `-v debug` switch.
```bash
skaffold dev --trigger notify -v debug
```
###  Live debugging

During the development It's a natural demand to setup a breakpoint and check the application's state while It's running. Locally It's a very easy process but what if we have everything in the cluster ? Actually It's easier as you might think, we need to adjust just a few things.

 - Add `5005` to the list of port under the jib configuration in the `build.gradle`
 - Add the `agentlib` related configuration inside of `jvmFlags` property
```groovy
jib {
    to {
        image = 'com.github.pozo/spring-boot-jib'
    }
    container {
        useCurrentTimestamp = true
        ports = ['55000', '5005']
        jvmFlags = [ '-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005' ]
    }
}
```
 - Extend `values.yaml`'s `container.ports` with 
```yaml
- name: debug
  containerPort: 5005
  protocol: TCP
```
 - Create a [remote configuration](https://stackoverflow.com/a/30793101) for the debugger

Then run the pipeline. Skaffold will automatically forward every listed ports, however be careful, after a change Skaffold might [pick a random port if the requested one isn't available](https://skaffold.dev/docs/how-tos/portforward/). Set a breakpoint, and run the previously created Remote configuration, call the corresponding endpoint and voilà.

I hope you enjoyed reading the article as much as I enjoyed writing it. I'm not a proficient writer, so if you have a comment, remark feel free to share it in the comments section. The source code of this article is available [on GitHub](https://github.com/Pozo/continuous-java-kubernetes).

I want to thank [Gergő Szabó](https://dev.to/@szgergo) and [Dániel Szabó](https://dev.to/@szada92) for all the help.

If you are interested in this topic I suggest to look over these articles as well
 - [Dockerize Spring Boot Application With Jib](http://mrhaki.blogspot.com/2018/09/spring-sweets-dockerize-spring-boot.html)
 - [Dockerizing Java Apps using Jib](https://www.baeldung.com/jib-dockerizing)
 - [Continuous Development for Kubernetes (Slides)](https://speakerdeck.com/coollog/skaffold-for-java-continuous-development-for-kubernetes)
 - [Introducing Jib — build Java Docker images better](https://cloudplatform.googleblog.com/2018/07/introducing-jib-build-java-docker-images-better.html)
 - [awesome-kubernetes - A curated list for awesome kubernetes sources](https://github.com/ramitsurana/awesome-kubernetes)
 - [Kubernetes Helm: Why It Matters](https://platform9.com/blog/kubernetes-helm-why-it-matters/)
 - [Draft vs Gitkube vs Helm vs Ksonnet vs Metaparticle vs Skaffold](https://blog.hasura.io/draft-vs-gitkube-vs-helm-vs-ksonnet-vs-metaparticle-vs-skaffold-f5aa9561f948/)