Database (Schema) migration to Kubernetes - initContainers vs k8s jobs -
Tags: kubernetes, database, devops, laravel
Published: 2021-04-12 11:45:31

# Introducton

We in [trackxy](https://trackxy.com/) were migrating from microservices managed each by its own team to our [Kubernetes](https://kubernetes.io/) cluster, where we have our services managed.

# Problem
- First, we need to know what schema migration is

>schema migration (also database migration, database change management) refers to the management of incremental, reversible changes and version control to relational database schemas. A schema migration is performed on a database whenever it is necessary to update or revert that database's schema to some newer or older version. 
Migrations are performed programmatically by using a schema migration tool. When invoked with a specified desired schema version, the tool automates the successive application or reversal of an appropriate sequence of schema changes until it is brought to the desired state.
[source](https://en.wikipedia.org/wiki/Schema_migration#:~:text=In%20software%20engineering%2C%20schema%20migration,control%20to%20relational%20database%20schemas.)


- So, as mentioned above schema migration is good right? yes it's When we have one instance of the program which runs the schema migration we're free to migrate it anytime we need and we're not worried about syncing or which instance will migrate first but when instances are more than 1 we face race condition and problem with which instance should run the migration 


# Brainstorming
In this section, we'll try to engage you in our thinking process and the pros and cons of each solution and why we chose the solution we did.

### Encaspulating Migration inside image

- we first thought about encapsulating the **migration process** inside the `image` itself but there are 3 reasons why we didn't go for this option
    1. SoC (*Separation of Concerns*), because it's not the image duty to migrate the database 
    2.  Race condition when we have several replicas of the image need to migrate at the same time and  *Kubernetes* pods creation time can't be controlled
    3.  Ambiguous behaviors, multiple replicas trying to migrate at the same time leads to ambiguous behaviors which we couldn't track and we chose not to go forward with this because the deployment component  in Kubernetes  creates pods in arbitrary order and all pods can start the migration process and make inconsistency in the database
- due to the reasons we mentioned above we decided not to go forward with this solution
![Schema migration in multiple replicas reaching same database](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4zfrmgkgg6mdldpk56ob.png)
This diagram shows the race condition happens when 3 replicas tries to migrate schema at the same time

---

### InitContainers


- It's nearly similar to the previous solution and has the same cons

### Kubernetes job

> A Job creates one or more Pods and will continue to retry execution of the Pods until a specified number of them successfully terminate. As pods successfully completed, the Job tracks the successful completions. When a specified number of successful completions is reached, the task (ie, Job) is complete. Deleting a Job will clean up the Pods it created.

**[source](https://kubernetes.io/docs/concepts/workloads/controllers/job/)** 

So as mentioned above the Kubernetes job functionality is to create one pod which do a specific job then dies without restarting, and that's exactly what we needed to run our migration once and in one pod only
# Our Solution

## prerequisites

- This solution assumes you have a good background about ***docker***, ***Kubernetes*** and ***helm***
- Having `kubectl` on your local machine and configured to your cluster
- You have a Kubernetes up and running -we're not discussing configuring k8s cluster here-
    - if not you can use *[Minikube](https://kubernetes.io/docs/tutorials/hello-minikube/)*
- We will use PHP/Laravel here to demonstrate our solution
- we have a PHP/Laravel pod
    - we configured our Database with PHP
- Our Database up and running -either in the cluster or outside the cluster-

## Use case

- We need to add a new table to our database

```bash
php artisan make:migration create_flights_table
```

- Migration should be done and the PHP/Laravel image redeployed, this where our solution comes to solve this problem

---

## Configuration

- we eventually went with the Kubernetes job solution
- This is an example of what we have done to migrate the database
- you have to have a `values.yaml` file to refer to the values we referred to here using **HELM**

```yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "APPLICATION.fullname" . }}-database-migration-job
spec:
  template:
    spec:
      containers:
        - name: {{ include "APPLICATION.fullname" . }}-database-migration-job
          image: "{{ .Values.databaseMigration.image  }}:{{.Values.databaseMigration.imageTag}}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          args:
            - php
            - artisan
            - migrate
          env:
            - name: ENV_VARIABLE
              value: "false"
          envFrom:
            - configMapRef:
                name:  {{ include "APPLICATION.fullname" . }}
            - secretRef:
                name: {{ .Values.laravel.secrets_names.laravel_secrets}}
      restartPolicy: Never
```

### Run

- to run the job and deploy using **HELM** we run in the root directory

```yaml
helm  upgrade --install -n [NAMESPACE] api  . 
```

- to run the job using `kubectl` , you have to fill the variables instead of using `values.yaml`

```yaml
kubectl apply -f job.yaml
```

### Testing

- You can see the job and the pod it looks after like so:

```bash
kubectl get jobs
```

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v7sqtl3tmq7uejnf629v.png)

```bash
kubectl get pods
```

- we can see that the pod for the job is created, ran and stopped after completing

    ![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0k0eklgrbqs4jehbqgxj.png)

- we can see the logs of the pod to make sure that our migration is done well

    ```bash
    kubectl logs [pod-name]
    ```

    ![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/358s5lvytvgqhkvhhxcz.png)


### invoking job
First we need to know that Kubernetes jobs don't rerun by default, we found that we have to deploy the job every single time you deploy so we decided to redeploy every time using our CI/CD pipeline - outside this article scope -, you can contact me to help you setup your pipeline
###author
written by me  [Ahmed Hanafy](https://github.com/ahmeddrawy) feel free to contact me if you have any questions