Learn Kubernetes - Create Multiple Schedulers
Tags: tutorial, kubernetes, devops, beginners
Published: 2022-06-17 16:21:00

<!-- wp:paragraph -->
<p>Howdy!</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Most big tech companies provide great documentation on their software. Especially, have you looked at Kubernetes documentation? They are quite awesome.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Particularly, I love the examples that are given pretty much for any topic. You can just blindly follow them to understand Kubernetes concepts.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Today, I chose to learn about how to configure multiple schedulers in a Kubernetes cluster, of course from their <a href="https://kubernetes.io/docs/tasks/extend-kubernetes/configure-multiple-schedulers/">documentation</a>. I also wanted to use my Minikube cluster. Learn how to install Minikube <a href="https://executeoncommand.com/?p=1736" target="_blank" rel="noreferrer noopener">here</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Start Minikube cluster.</h2>
<!-- /wp:heading -->

<!-- wp:code -->
<pre class="wp-block-code"><code><strong>%minikube start --driver=hyperkit</strong>
😄  minikube v1.25.2 on Darwin 12.4
✨  Using the hyperkit driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating hyperkit VM (CPUs=2, Memory=8192MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default</code></pre>
<!-- /wp:code -->

<!-- wp:heading -->
<h2>Find the image used by kube-scheduler pod running in your Minikube cluster.</h2>
<!-- /wp:heading -->

<!-- wp:code -->
<pre class="wp-block-code"><code>%<strong>kubectl get pod kube-scheduler-minikube -o yaml -n kube-system | grep image</strong>
<strong>    image: k8s.gcr.io/kube-scheduler:v1.23.3</strong>
    imagePullPolicy: IfNotPresent
    image: k8s.gcr.io/kube-scheduler:v1.23.3
    imageID: docker-pullable://k8s.gcr.io/kube-scheduler@sha256:32308abe86f7415611ca86ee79dd0a73e74ebecb2f9e3eb85fc3a8e62f03d0e7</code></pre>
<!-- /wp:code -->

<!-- wp:heading -->
<h2>Create the YAML file for all the required resources</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Here's the <a href="https://raw.githubusercontent.com/kubernetes/website/main/content/en/examples/admin/sched/my-scheduler.yaml">YAML file </a>provided in Kubernetes documentation that creates all the resources required to create another scheduler. Since the second scheduler is also going to use the same image as the above, let's make sure to use this. What that means is to replace the image mentioned in the documentation with the above image.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-scheduler
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-scheduler-as-kube-scheduler
subjects:
- kind: ServiceAccount
  name: my-scheduler
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: system:kube-scheduler
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-scheduler-as-volume-scheduler
subjects:
- kind: ServiceAccount
  name: my-scheduler
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: system:volume-scheduler
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-scheduler-config
  namespace: kube-system
data:
  my-scheduler-config.yaml: |
    apiVersion: kubescheduler.config.k8s.io/v1beta2
    kind: KubeSchedulerConfiguration
    profiles:
      - schedulerName: my-scheduler
    leaderElection:
      leaderElect: false    
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    component: scheduler
    tier: control-plane
  name: my-scheduler
  namespace: kube-system
spec:
  selector:
    matchLabels:
      component: scheduler
      tier: control-plane
  replicas: 1
  template:
    metadata:
      labels:
        component: scheduler
        tier: control-plane
        version: second
    spec:
      serviceAccountName: my-scheduler
      containers:
      - command:
        - /usr/local/bin/kube-scheduler
        - --config=/etc/kubernetes/my-scheduler/my-scheduler-config.yaml
       <strong> image: k8s.gcr.io/kube-scheduler:v1.23.3</strong>
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10259
            scheme: HTTPS
          initialDelaySeconds: 15
        name: kube-second-scheduler
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10259
            scheme: HTTPS
        resources:
          requests:
            cpu: '0.1'
        securityContext:
          privileged: false
        volumeMounts:
          - name: config-volume
            mountPath: /etc/kubernetes/my-scheduler
      hostNetwork: false
      hostPID: false
      volumes:
        - name: config-volume
          configMap:
            name: my-scheduler-config
</code></pre>
<!-- /wp:code -->

<!-- wp:heading -->
<h2>Create all the required objects</h2>
<!-- /wp:heading -->

<!-- wp:code -->
<pre class="wp-block-code"><code>%kubectl apply -f my-scheduler.yaml
serviceaccount/my-scheduler created
clusterrolebinding.rbac.authorization.k8s.io/my-scheduler-as-kube-scheduler created
clusterrolebinding.rbac.authorization.k8s.io/my-scheduler-as-volume-scheduler created
configmap/my-scheduler-config created
deployment.apps/my-scheduler created</code></pre>
<!-- /wp:code -->

<!-- wp:heading -->
<h2>Verify whether the new scheduler is created</h2>
<!-- /wp:heading -->

<!-- wp:code -->
<pre class="wp-block-code"><code>%kubectl get deployment -n kube-system
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
coredns        1/1     1            1           23h
<strong>my-scheduler   1/1     1            1           8s</strong>

%kubectl get pods -n kube-system
NAME                               READY   STATUS    RESTARTS   AGE
coredns-64897985d-4vstq            1/1     Running   0          23h
etcd-minikube                      1/1     Running   0          23h
kube-apiserver-minikube            1/1     Running   0          23h
kube-controller-manager-minikube   1/1     Running   0          23h
kube-proxy-w4srd                   1/1     Running   0          23h
kube-scheduler-minikube            1/1     Running   0          23h
<strong>my-scheduler-5546b885fb-mdgcn      1/1     Running   0          18s</strong>
storage-provisioner                1/1     Running   0          23h</code></pre>
<!-- /wp:code -->

<!-- wp:heading -->
<h2>Create a pod but this time make it use the new scheduler</h2>
<!-- /wp:heading -->

<!-- wp:code -->
<pre class="wp-block-code"><code><strong>%cat pod-using-second-scheduler.yaml</strong>
apiVersion: v1
kind: Pod
metadata:
  name: annotation-second-scheduler
  labels:
    name: multischeduler-example
spec:
  <strong>schedulerName: my-scheduler</strong>
  containers:
  - name: pod-with-second-annotation-container
    image: k8s.gcr.io/pause:2.0

%<strong>kubectl apply -f pod-using-second-scheduler.yaml</strong>
pod/annotation-second-scheduler created

%<strong>kubectl get event</strong>
LAST SEEN   TYPE      REASON      OBJECT                            MESSAGE
<strong>19s         Normal    Scheduled   pod/annotation-second-scheduler   Successfully assigned default/annotation-second-scheduler to minikube</strong>
18s         Normal    Pulling     pod/annotation-second-scheduler   Pulling image "k8s.gcr.io/pause:2.0"
17s         Normal    Pulled      pod/annotation-second-scheduler   Successfully pulled image "k8s.gcr.io/pause:2.0" in 1.25784015s
17s         Normal    Created     pod/annotation-second-scheduler   Created container pod-with-second-annotation-container
17s         Normal    Started     pod/annotation-second-scheduler   Started container pod-with-second-annotation-container</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>That's it. Ta-daa !</p>
<!-- /wp:paragraph -->