How to build a CI/CD workflow with Skaffold for your application (Part II)
Tags: php, gitops, kubernetes, skaffold
Published: 2022-11-16 11:24:15

---
title: How to build a CI/CD workflow with Skaffold for your application (Part II)
published: true
date: 2022-11-07 08:45:42 UTC
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oyxqet24zkc82cc2mjwh.jpg
canonical_url: https://blog.equationlabs.io/how-to-build-a-cicd-workflow-with-skaffold-for-your-application-part-ii
tags: #php #gitops #kubernetes #skaffold
series: "Full CI/CD workflow with Skaffold for your application"
---

## Lets recap the `Workflow`

As you remind - and if not, you can read the first release of this tutorial 😅 - my main idea is to implement one tool - `skaffold` - as building block for my `CI/CD workflow`, who should be managed from a single `makefile` as entrypoint for local development and pipelines - on `gitlab` - and all this should be deployed in a `K8s` cluster in `GCP`

And also, for simplicity of this tutorial, we use the `image` and `artefact` repository in the same `gitlab` SAAS, but you can use whatever you want for this task (`Amazon S3`, `Docker` Registry, `Private Registries`, `Azure` Object Storage, etc)

This is the `workflow` so far:

_📣 The tool setup and local workflow was covered in the first delivery of this tutorial._

![2fl6qCIhG.png.jpeg](https://cdn.hashnode.com/res/hashnode/image/upload/v1667561874111/Qhz2KHP07.jpeg)

## The Makefile

As said, the `makefile`, is, the main entrypoint for commands to be executed by `developers` when they do development work locally, and for `gitlab pipelines` in their different stages (must vary on your implementation), and that `makefile` commands mostly are wraps for `skaffold` ones, with the difference, that i need to pass dynamic values to those stages to work I expect to, so it's better to wrap then in a makefile target that received that dynamic params and then run the 'skaffold' command itself. (later you will now why)

As for the time I was writing this article, my targets are:

![Screenshot 2022-11-06 at 20.33.21.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1667763214992/_MmsTbgzE.png)

Hopefully for local development we have an unique command since `skaffold run` do the full pipeline cycle `(build, test, deploy, hot reload)` for development , so, why then i need to wrap this single command in a `makefile target`?, mostly because I need that this work no matter what type/kind of local cluster technology the developer use (docker-desktop, Minikube, etc) and no matter what OS developer machine was (MacOS, \*Nix) and for that, I need to pass the `kube-context` parameter to `skaffold` that in my case is `docker-desktop` (since docker desktop already bring me a pre installed `k8s cluster for local development` and that free me to the need to install manually a cluster in my machine - a win for docker-desktop here -.

So you will encounter that the majority of the `makefile` wraps, come in the form of rehuse the same command in multiple stages (pipelines) based on the received parameter in the make execution, and also, generating random seeds to prefix namespaces (because you will have more than one developer working in the same code base at once) and I want to avoid collision between `gitlab pipelines` and deploys in lower environments when N developers are working in the same code base.

We must focused in `pipeline` target ones (all of then with `skaffold` tool), since the infrastructure ones, related to install prerequisites in local cluster to comply with the architecture design, isn't cover in this series, but in the another series that 'll wrote in next weeks)

Here is how my `makefile` looks at the time of writing this article, and for those stages (because in technology everything evolves quickly I always remind this 😅)

```
#------ Development and Pipeline targets ----------#
run: ## DEVELOPMENT[skaffold]: Up and running stack in development mode with hot reloading in the local machine
    @$(MAKE) _requirements
    @skaffold dev -f $(DEPLOY_DIR)/skaffold.yaml -p development -n $(PROJECT_NAME) --no-prune=false --cache-artifacts=false
unit: ## DEVELOPMENT[skaffold]: build, deploy and run unit tests => FOR PIPELINE: `make unit profile=pipeline kube_context=cluster-gitlab-context`
    @$(MAKE) _run_test_suite SUITE=unit PROFILE=$(PROFILE) NAMESPACE=$(DYNAMIC_NAMESPACE) KUBE_CONTEXT=$(KUBE_CONTEXT) || $(MAKE) _cleanup KUBE_CONTEXT=$(KUBE_CONTEXT) PROFILE=$(profile) NAMESPACE=$(DYNAMIC_NAMESPACE)
integration: ## DEVELOPMENT[skaffold]: build, deploy and run integration tests => FOR PIPELINE: `make integration profile=pipeline kube_context=cluster-gitlab-context`
    @$(MAKE) _run_test_suite SUITE=integration PROFILE=$(PROFILE) NAMESPACE=$(DYNAMIC_NAMESPACE) KUBE_CONTEXT=$(KUBE_CONTEXT) || $(MAKE) _cleanup KUBE_CONTEXT=$(KUBE_CONTEXT) PROFILE=$(profile) NAMESPACE=$(DYNAMIC_NAMESPACE)
functional: ## DEVELOPMENT[skaffold]: build, deploy and run functional tests => FOR PIPELINE: `make functional profile=pipeline kube_context=cluster-gitlab-context`
    @$(MAKE) _run_test_suite SUITE=functional PROFILE=$(PROFILE) NAMESPACE=$(DYNAMIC_NAMESPACE) KUBE_CONTEXT=$(KUBE_CONTEXT) || $(MAKE) _cleanup KUBE_CONTEXT=$(KUBE_CONTEXT) PROFILE=$(profile) NAMESPACE=$(DYNAMIC_NAMESPACE)
build: ## PIPELINE[skaffold]: build and push images to registry => `make build tag=1.0.0|71dcab00 kube_context=docker-desktop`
    @skaffold build -f $(DEPLOY_DIR)/skaffold.yaml -p production -t $(tag) --kube-context=$(kube_context) --file-output=pipeline-artifacts.json
render: ## PIPELINE[skaffold]: render manifests and push to artifact registry => `make render namespace=availability tag=1.0.0|71dcab00`
    @skaffold render -f $(DEPLOY_DIR)/skaffold.yaml -p production -n $(PROJECT_NAME) -a pipeline-artifacts.json -o $(PROJECT_NAME)-api-$(tag)-production.yaml
deploy: ## PIPELINE[skaffold]: apply hydrated manifests to desired namespace on cluster `make deploy tag=1.0.0|71dcab00 profile=production namespace=availability kube_context=docker-desktop`
    @kubectl create namespace $(namespace) --context=$(kube_context)
    @skaffold apply -f $(DEPLOY_DIR)/skaffold.yaml -p $(profile) -n $(namespace) --kube-context=$(kube_context) --status-check=true $(PROJECT_NAME)-api-$(tag)-production.yaml || $(MAKE) _cleanup KUBE_CONTEXT=$(kube_context) PROFILE=$(profile) NAMESPACE=$(namespace)

```

As you can see, all targets are warps for `skaffold`, but, allowing me to pass dynamic data, context and profiles (we'll look this in the `skaffold` file explanation below), the complete file is more larger, but with this snippet you can get an idea on how you can build something like that.

## The `skaffold` file

The main `workflow orchestrator` has a main config file, when we can define, how the application should be builded, tested, render their manifest and deployed, so it's a vertebral part of this strategy, and now I'll explain you mine.

I have two `skaffold` profiles: one called `development` and the other `production, and a common part share between them like tag strategy, deploy strategy, and both of them use the same`Dockerfile`to build their images pointing to the correct`target`. (You can use separate Dockerfiles, but in my case I want to maintain as simple as possible because the difference between 2 images dependencies are minimum)

the skaffold fiel lives inside my deploy folder (do you remember my file organisation? You can re-checked in the first delivery of this series), and all the deployment related files are store there.

![Screenshot 2022-11-04 at 14.28.59.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1667568630626/BK1BkpCMP.png)

As you can see the `skaffold.yaml` file is in the root of the deploy folder since is my main `workflow orchestrator`, in the other folder we have the main `k8s manifests` and inside overlays we have the yaml patch's for every profile (most of the cases the same as an environment) that we want to declare.

## The `.gitlab-ci.yml` Pipeline

Since I use trunk base development strategy in micro services, we need to design 2 flows of pipelines, one for features branches and one for main branch, additionally to that, to be able to reach production we first need to tag a commit, so that's the real trigger for Go-To-Prod Pipeline.

### Feature Branches Workflow (trigger by merge\_request commit):

In this stage, the developer needs to be able to:

- Run all tests suites
- Build and Image with production dependencies
- Deploy to a dynamic namespace in the same cluster where pipeline runs (a Dynamic QA to say so, allowing to every developer to have "their own" QA server while their are working on a feature)
- Destroy the dynamic deployment (remove review app and namespace) 

Looks like this:

![Screenshot 2022-11-06 at 19.27.58.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1667759285752/uJZIk6dpo.png)

![Screenshot 2022-11-06 at 15.46.45.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1667759296006/aVxZ8ellf.png)

Also we use gitlab environments to see in the Gitlab UI "what's is deployed in what environment", and also to visualise production and stages latest deployment status and artefacts.

So, when a new "dynamic QA" environment is deployed, you'll see this in environment page of your project:

![Screenshot 2022-11-06 at 19.52.14.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1667760771094/pZsqLNroN.png)

And also on the MR View on Gitlab you'll see in what "review" environment is deployed that MR.

![Screenshot 2022-11-06 at 19.52.36.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1667760778935/DCwFN5n6x.png)

### Main Branch Workflow (trigger by a TAG):

In this stage, the developer needs to be able to:

- Run all tests suites
- Build and Image with production dependencies
- Create a Release Package (this a gitlab feature like Github to display release contents in a special page on gitlab)
- Deploy to Production

Looks like this:

![Screenshot 2022-11-06 at 16.57.35.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1667759456750/KMqd5KDZa.png)

![Screenshot 2022-11-06 at 19.31.20.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1667759490932/OG7wFHc0A.png)

The main workflow has 3 stages more, including deploying to staging cluster/namespace, but more important has the creation of the release package and the deployment to production.

The only manual job is deploy to production.

Gitlab Release

![Screenshot 2022-11-06 at 20.00.37.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1667761248338/FXomvjxOy.png)

Production Environment after Deployment

![Screenshot 2022-11-06 at 20.00.08.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1667761214382/VQzJafEOg.png)

All the best of all, all this is DONE automatically via the automated Pipeline on Gitlab. You can view the skeleton of the pipeline in this Snippet:

https://gitlab.com/playground-arena/api-symfony-roadrunner-cqrs/-/snippets/2448780

## BONUS: `Kaniko` image Builder

Since version 1.24, [`Kubernetes` was moving away from `dockershim`](https://kubernetes.io/blog/2022/01/07/kubernetes-is-moving-on-from-dockershim/) as container runtime, so I want a solution that:

- Allow me to build container inside a K8s cluster
- Continue to writing Dockerfiles as today
- Not need to share or mount a socket into a pod (that's the main security reason to not use docker)

So I discover the last year `Kaniko`, this is another tool on the `Google Container Tools` on `Github`, that allows us to build an container image from a Dockerfile without Docker. Marvellous.

It has som benefits:

\*`Kaniko` doesn't depend on a `Docker daemon` and executes each command within a Dockerfile completely in userspace. (no need to mount a docker socket anymore)

- Provides a handy docker image to use in pipelines (a very light weight one, so your jobs doesn't take much time to run)
- Provides a Caching system in the same repository where the images are stored, so in every job that tries to build the image, `Kaniko` will check first the repo cache layers and downloaded speeding up the build job.

When a build a image I check my repo and I can see the caching layers separated from the final images (see image below)

![Screenshot 2022-11-06 at 20.18.09.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1667762313501/PjAyGdJLh.png)

Cache Layers

![Screenshot 2022-11-06 at 20.18.13.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1667762327292/bBwJblXs_.png)

Builded and Tagged Images

![Screenshot 2022-11-06 at 20.18.21.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1667762342390/LI9SwKP31.png)

## Deployment safety considerations

All in life has it's tradeoff, and this isn't the exception, however, is possible to joint team autonomy with security and governance, no only with this example, but in general in software development.

Some of my personal recommendations on this are:

- Secrets: Always use a secret vault (in my case i use Hashicorp one) 
- Fine-grained permissions control with the CI/CD tunnel via impersonation and k8s agent:
  - Allows you to leverage your K8s authorization capabilities to limit the permissions of what can be done with the CI/CD tunnel on your running cluster
  - Lowers the risk of providing unlimited access to your K8s cluster with the CI/CD tunnel
  - Segments fine-grained permissions with the CI/CD tunnel at the project or group level
  - Controls permissions with the CI/CD tunnel at the username or service account
- Make your high environment namespaces inmutable (on K8s namespace creation time)
- Fine Grained RBAC on Gitlab Roles (I think this isn't available on Free and Self-Managed version)

## Next Chapter

In the next chapter I'll wrap up all things that we cover in this first 2 releases of the series in a full functional pipeline from local to production with a small k8s cluster and i expect, that you can see all the work in action and some final remarks to allow you to test in your own projects.

Thanks for reading!

## Support me

If you like what you just read and you find it valuable, then you can buy me a coffee by clicking the link in the image below, it would be appreciated.

<a href="https://www.buymeacoffee.com/rcastellanosm" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" style="height: 30px !important;width: 117px !important;" ></a>