Machine Learning Model Deployments Using Seldon Core
Tags: machinelearning, seldoncore, kubernetes
Published: 2023-03-13 14:01:46

## Introduction

Seldon Core is an open-source platform to deploy your machine learning models on Kubernetes at a massive scale.

Seldon core converts your ML models (Tensorflow, Pytorch, H2o, etc.) or language wrappers (Python, Java, etc.) into production REST/GRPC microservices.

Seldon handles scaling to thousands of production machine learning models and provides advanced machine learning capabilities out of the box including Advanced Metrics, Request Logging, Explainers, Outlier Detectors, A/B Tests, Canaries and more.

## Prerequisites

Install necessary utility packages, here I'm using Ubuntu 20.04.5 LTS distribution.

```
$ sudo apt update

$ sudo apt install curl yq
```

Install Docker on your machine using the convenience script.

```
$ curl -fsSL https://get.docker.com -o get-docker.sh

$ sudo sh ./get-docker.sh
```

Now add your user to the docker group to execute commands without using sudo.

```
$ sudo usermod -aG docker $USER
```

Log out from your current session and log back in. Now you can execute docker commands without using sudo.

```
$ docker info
```

Install k3d, a lightweight wrapper to run k3s (Rancher Lab's minimal Kubernetes distribution) in Docker.

```
$ curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash

$ k3d version
```

Install kubectl, a command line tool that allows you to run commands against Kubernetes clusters.

```
$ curl -LO https://dl.k8s.io/release/v1.24.10/bin/linux/amd64/kubectl

$ sudo install kubectl /usr/local/bin/

$ kubectl version
```

Install Helm, a command line tool that helps you to define, install, and upgrade even the most complex Kubernetes application.

```
$ curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

$ helm version
```

Install istioctl, a command line tool that allows service operators to debug and diagnose their Istio service mesh deployments.

```
$ curl -L https://istio.io/downloadIstio | sh -

$ cd istio-1.17.1/bin

$ sudo install istioctl /usr/local/bin

$ istioctl version
```

## Configuration

Create a k3d Kubernetes cluster using the below configuration file.

```
$ cat config.yml

apiVersion: k3d.io/v1alpha4
kind: Simple
metadata:
  name: seldon-core
servers: 1
agents: 2
image: rancher/k3s:v1.24.10-k3s1
ports:
- port: 30000-30100:30000-30100
  nodeFilters:
  - server:*
registries:
  create:
    name: seldon-core
    host: 0.0.0.0
    hostPort: "5000"
options:
  k3s:
    extraArgs:
    - arg: --disable=traefik
      nodeFilters:
      - server:*
```

```
$ k3d cluster create --config=config.yml
```

Once the cluster is created verify its status.

```
$ k3d cluster list 

NAME          SERVERS   AGENTS   LOADBALANCER
seldon-core   1/1       2/2      true
```

```
$ kubectl config use-context k3d-seldon-core 

Switched to context "k3d-seldon-core".
```

```
$ kubectl -n kube-system get pods

NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
kube-system   coredns-7b5bbc6644-krxc6                  1/1     Running   0          42s
kube-system   local-path-provisioner-687d6d7765-w8mqt   1/1     Running   0          42s
kube-system   metrics-server-667586758d-b77xx           1/1     Running   0          42s
```

Deploy Istio components to our cluster and verify it's status.

```
$ istioctl install --set profile=demo -y

✔ Istio core installed                                                                                                                                                                         
✔ Istiod installed                                                                                                                                                                             
✔ Egress gateways installed                                                                                                                                                                    
✔ Ingress gateways installed                                                                                                                                                                   
✔ Installation complete
Making this installation the default for injection and validation.

Thank you for installing Istio 1.17.
```

```
$ kubectl -n istio-system get pods

NAME                                    READY   STATUS    RESTARTS   AGE
istiod-76cf8b7b8b-675wg                 1/1     Running   0          79s
istio-ingressgateway-8568ffc4d4-qr7pd   1/1     Running   0          53s
istio-egressgateway-8694db4556-wtcpj    1/1     Running   0          53s
```

Create an Istio gateway, a load balancer operating at the edge of the mesh receiving incoming or outgoing HTTP/TCP connections.

```
$ cat gateway.yml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: seldon-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
```

```
$ kubectl apply -f gateway.yml 

gateway.networking.istio.io/seldon-gateway created
```

```
$ kubectl -n istio-system get gateways

NAME             AGE
seldon-gateway   64s
```

Create a new namespace for deploying Seldon Core Operator.

```
$ kubectl create namespace seldon-system

namespace/seldon-system created
```

Deploy Seldon Core Operator to our cluster using Helm.

The Seldon Core Operator is what controls your Seldon Deployments in the Kubernetes cluster. It reads the CRD definition of Seldon Deployment resources applied to the cluster and takes care that all required components like Pods and Services are created.

```
$ helm install seldon-core seldon-core-operator \
    --repo https://storage.googleapis.com/seldon-charts \
    --set usageMetrics.enabled=true \
    --set istio.enabled=true \
    --namespace seldon-system

NAME: seldon-core
LAST DEPLOYED: Mon Mar 13 12:25:21 2023
NAMESPACE: seldon-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
```

```
$ kubectl -n seldon-system get pods

NAME                                        READY   STATUS    RESTARTS   AGE
seldon-controller-manager-b74d66684-b4r55   1/1     Running   0          53s
```

Create another namespace for our model deployment.

```
$ kubectl create namespace seldon

namespace/seldon created
```

Enable Istio injection on our newly created namespace.

When you set the istio-injection=enabled label on a namespace and the injection webhook is enabled, any new pods that are created in that namespace will automatically have a sidecar added to them.

```
$ kubectl label namespace seldon istio-injection=enabled

namespace/seldon labeled
```

Deploy a pre-packaged model server for scikit-learn and verify it's status.

```
$ cat iris-model.yml

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: iris-model
  namespace: seldon
spec:
  name: iris
  predictors:
  - graph:
      implementation: SKLEARN_SERVER
      modelUri: gs://seldon-models/v1.15.0-dev/sklearn/iris
      name: classifier
    name: default
    replicas: 1
```

```
$ kubectl apply -f iris-model.yml 

seldondeployment.machinelearning.seldon.io/iris-model created
```

```
$ kubectl -n seldon get seldondeployments

NAME         AGE
iris-model   30s
```

```
$ kubectl -n seldon get pods

NAME                                               READY   STATUS    RESTARTS   AGE
iris-model-default-0-classifier-59bd9f6c5d-zbtqw   3/3     Running   0          4m42s
```

Verify the deployed model by accessing the Istio ingress gateway load balancer IP.

```
$ kubectl -n istio-system get svc istio-ingressgateway

NAME                   TYPE           CLUSTER-IP     EXTERNAL-IP                        PORT(S)                                                                      AGE
istio-ingressgateway   LoadBalancer   10.43.182.71   172.19.0.2,172.19.0.3,172.19.0.4   15021:30288/TCP,80:31550/TCP,443:31363/TCP,31400:32342/TCP,15443:32413/TCP   53s
```

```
$ curl -s -X POST http://172.19.0.2/seldon/seldon/iris-model/api/v1.0/predictions \
	-H 'Content-Type: application/json' \
	-d '{ "data": { "ndarray": [[1,2,3,4]] } }' | jq
{
  "data": {
    "names": [
      "t:0",
      "t:1",
      "t:2"
    ],
    "ndarray": [
      [
        0.0006985194531162835,
        0.00366803903943666,
        0.995633441507447
      ]
    ]
  },
  "meta": {
    "requestPath": {
      "classifier": "seldonio/sklearnserver:1.15.0"
    }
  }
}
```

You can also test the model from Swagger UI by accessing the URL http://172.19.0.2/seldon/seldon/iris-model/api/v1.0/doc/ from the browser.

![Swagger UI](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mwtl9yh20lddja8brhni.png)

## References

https://docs.docker.com/engine/install/ubuntu/

https://k3d.io/v5.4.8/#install-script

https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/

https://helm.sh/docs/intro/install/

https://istio.io/latest/docs/setup/install/istioctl/

https://docs.seldon.io/projects/seldon-core/en/latest/install/kind.html