Executing Nextflow pipelines with K3d
Tags: nextflow, kubernetes, k3s, k3d
Published: 2024-05-02 20:09:44

---
title: Executing Nextflow pipelines with K3d
published: true
date: 2024-05-01 00:00:00 UTC
tags: nextflow,kubernetes,k3s,k3d
canonical_url: https://jorge.aguilera.soy/blog/2024/k3s-nextflow.html
---

_Nextflow enables scalable and reproducible scientific workflows using software containers. It allows the adaptation of pipelines written in the most common scripting languages._

In this post I’ll explore how to create a "work environment" where run Nextflow pipelines using Kubernetes.

The user will be able to create and edit the pipeline, configuration and assets into their computer and run the pipelines in the cluster in a fluent way.

The idea is to provide to the user the most complete environment in their computer so, once tested and validated, it will run in a real cluster.

## Problem

When you’re deploying Nextflow’s pipelines in kubernetes you need to find a way to share the workdir. Options are basically use `volumes` or use `fusion`

Fusion is very easy to use (basically `enable=true` in the nextflow.config) but you create an external dependency.

Volumes are more "native" solution, but you need to fight with the infrastructure, providers, etc.

Another challenge working with pipelines in kubernetes is to retrieve outputs once the pipeline is completed. Probably you need to run some `kubectl cp` commands

In this post I’ll create a cluster (with only 1 node) from scratch and run some pipelines on it. We’ll see how pods are created and how we can edit the pipeline and/or configuration using our preferred IDE (notepad, vi, VSCode,…​)

## Requirements

- a computer (and internet connection of course)

We need to have installed following command line tools:

- kubectl (if you work in kubernetes you’ve it)

- skaffold [https://skaffold.dev/docs/install/](https://skaffold.dev/docs/install/)

- k3d [https://k3d.io/v5.6.3/#releases](https://k3d.io/v5.6.3/#releases)

- k9s. Is **not required** but very useful

## Create a cluster

`k3d cluster create nextflow --port 9999:80@loadbalancer`

We’re creating a new cluster called `nextflow` (can be whatever). We’ll use 9999 port to access to our results

```
kubectl cluster-info
Kubernetes control plane is running at https://0.0.0.0:40145
CoreDNS is running at https://0.0.0.0:40145/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://0.0.0.0:40145/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
```

## Preparing our environment

Create a folder `test`

### Nextflow area

Create a subfolder `project` (will be our nextflow working area)

Create a `nextflow.config` in this subfolder

```
k8s {
   context = 'k3d-nextflow' (1)

   namespace = 'default' (2)

   runAsUser = 0
   serviceAccount = 'nextflow-sa'
   storageClaimName = 'nextflow'
   storageMountPath = '/mnt/workdir'
}

process {
   executor = 'k8s'
   container = "quay.io/nextflow/rnaseq-nf:v1.2.1"
}
```

| **1** | `k3d-nextflow` was created by k3d. If you chose another name you need to change it |
| **2** | I’ll use the `default` namespace |

### K8s area

Create a subfolder `k8s` and create following files into it:

pvc.yml

```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nextflow
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 2Gi
```

admin.yml

```
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nextflow-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: nextflow-role
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/status", "pods/log", "pods/exec"]
    verbs: ["get", "list", "watch", "create", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: nextflow-rolebind
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nextflow-role
subjects:
  - kind: ServiceAccount
    name: nextflow-sa
```

jagedn.yml

```
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jagedn (1)
  labels:
    app: jagedn
spec:
  selector:
    matchLabels:
      app: jagedn
  template:
    metadata:
      labels:
        app: jagedn
    spec:
      serviceAccountName: nextflow-sa
      terminationGracePeriodSeconds: 5
      securityContext:
        fsGroup: 0
        runAsGroup: 0
        runAsNonRoot: false
        runAsUser: 0
      containers:
      - name: nextflow
        image: jagedn
        volumeMounts:
        - mountPath: /mnt/workdir
          name: volume
      - name: nginx-container
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
          - name: volume
            mountPath: /usr/share/nginx/html
      volumes:
      - name: volume
        persistentVolumeClaim:
          claimName: nextflow
```

| **1** | `jagedn` is my nick, you can use whatever but pay attention to replace in all places |

kustomization.yaml

```
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - pvc.yml
 - admin.yml
 - jagedn.yaml
```

Basically we’re creating:

- service account and their roles

- a persistent volume claim to share across pods

- a pod

### Skaffold

In the "parent" folder create following files:

Dockerfile

```
FROM nextflow/nextflow:24.03.0-edge
RUN yum install -y tar (1)

ADD project /home/project (2)

ENTRYPOINT ["tail", "-f", "/dev/null"]
```

| **1** | Required by skaffold to sync files |
| **2** | Required by skaffold to sync files |

skaffold.yaml

```
apiVersion: skaffold/v4beta10
kind: Config
metadata:
    name: nextflow

build:
    artifacts:
        - image: jagedn
          context: .
          docker:
            dockerfile: Dockerfile
          sync:
            manual:
                - src: 'project/**'
                  dest: /home

manifests:
    kustomize:
        paths:
            - k8s

deploy:
    kubectl: {}
```

## Watching

Only for the purpose to watch how pods are created and destroyed we’ll run in a terminal console `k9s`

 ![k9s](https://jorge.aguilera.soy/images/2024/k9s.png)

## Go

Open the project with VSCode (for example)

 ![vscode](https://jorge.aguilera.soy/images/2024/vscode.png)

Open a `terminal` tab and execute:

`skaffold dev`

 ![skaffold](https://jorge.aguilera.soy/images/2024/skaffold.png)

let the terminal running

In another terminal console execute:

`kubectl exec -it jagedn-56b9fb64dc-2xw8f — /bin/bash`

<dl>
<dt>WARNING</dt>
<dd>
<p>You’ll need to use the pod id skaffold has created</p>
</dd>
</dl>

and cd to `/home/project`

 ![nextflow k3d 1](https://jorge.aguilera.soy/images/2024/nextflow-k3d-1.png)

<dl>
<dt>INFO</dt>
<dd>
<p>Using you VCode editor open <code>nextflow.config</code> and change something (a comment for example). Save the change
and in the terminal run a <code>cat</code> command to verify skaffold has been synced the file</p>
</dd>
</dl>

 ![nextflow k3d 2](https://jorge.aguilera.soy/images/2024/nextflow-k3d-2.png)

## Run

In the terminal execute

`NXF_ASSETS=/mnt/workdir/assets nextflow run nextflow-io/rnaseq-nf -with-docker -w /mnt/workdir -cache false`

If you change to the `k9s` console, you’ll see how pods are created

 ![k9s 2](https://jorge.aguilera.soy/images/2024/k9s-2.png)

After a while the pipeline is completed !!!!

 ![nextflow k3d 3](https://jorge.aguilera.soy/images/2024/nextflow-k3d-3.png)

## Extra ball

If you inspect the folder `/home/project/results` you’ll find the outputs of the pipeline, so …​ how we can inspect them?

Execute `cp -R results/ /mnt/workdir/` in the kubectl terminal and open a browser to [http://localhost:9999/results/multiqc\_report.html](http://localhost:9999/results/multiqc_report.html)

 ![tadam](https://jorge.aguilera.soy/images/2024/tadam.png)

<dl>
<dt>INFO</dt>
<dd>
<p>A better approach is to create another volume for the result so nginx sidecar pod can read directly</p>
</dd>
</dl>

## Clean up

Once you want to end with your cluster only finish (ctrl+c) the skaffold session and it will remove all resources

To delete the cluster you can run `k3d cluster delete nextflow`

## Conclusion

Don’t know if this approach is the best or maybe a little complicate but I think can be a good approach to have a very productive kubernetes environment without the need to have a full cluster