Cloud Native Workflow for *Private* MPT-30B AI Apps
Tags: kubernetes, ai, cloudnative, llm
Published: 2023-07-01 14:16:56

In this article, we will guide you through the process of developing your own private AI application 🤖, leveraging the capabilities of Kubernetes.

Unlike many other tutorials, we will **NOT** rely on OpenAI APIs. Instead, we will utilize a private AI instance with a Apache 2.0 licensed model [MPT-30B](https://huggingface.co/mosaicml/mpt-30b), that ensures the **confidentiality** of all 🔒 sensitive data 🔒 within your Kubernetes cluster. No data goes to the third-party cloud 🙅‍♂️ 🌩️!

To set up the development environment on Kubernetes, we will utilize [devspace](https://www.devspace.sh/). This environment includes a file sync pipeline for your AI application, as well as the backend [AI API](https://github.com/chenhunghan/ialacol) (a RESTful API service designed to replace OpenAI API) for the AI app.

Let's kick-start the process by deploying the necessary services on Kubernetes using the command `devspace deploy`. DevSpace will handle the deployment of the initial structure of our applications, along with their dependencies, including [ialacol](https://github.com/chenhunghan/ialacol). For more detailed explanations, please refer to the in-line comments provided in the code snippet below:
 
```yaml
# This is the configuration file for DevSpace
# 
# devspace use namespace private-ai # suggest to use a namespace instead of the default name space
# devspace deploy # deploy the skeleton of the app and the dependencies (ialacol)
# devspace dev # start syncing files to the container
# devspace purge # to clean up
version: v2beta1
deployments:
  # This are the manifest our private app deployment
  # The app will be in "sleep mode" after `devspace deploy`, and start when we start
  # syncing files to the container by `devspace dev`
  private-ai-app:
    helm:
      chart:
        # We are deploying the so-called Component Chart: https://devspace.sh/component-chart/docs
        name: component-chart
        repo: https://charts.devspace.sh
      values:
        containers:
          - image: ghcr.io/loft-sh/devspace-containers/python:3-alpine
            command:
            - "sleep"
            args:
            - "99999"
        service:
          ports:
          - port: 8000
        labels:
          app.kubernetes.io/name: private-ai-app
  ialacol:
    helm:
      # the backend for the AI app, we are using ialacol https://github.com/chenhunghan/ialacol/
      chart:
        name: ialacol
        repo: https://chenhunghan.github.io/ialacol
      # overriding values.yaml of ialacol helm chart
      values:
        replicas: 1
        deployment:
          image: quay.io/chenhunghan/ialacol:latest
          env:
            # We are using MPT-30B, which is the most sophisticated model at the moment
            # If you want to start with some small but mightym try orca-mini
            # DEFAULT_MODEL_HG_REPO_ID: TheBloke/orca_mini_3B-GGML
            # DEFAULT_MODEL_FILE: orca-mini-3b.ggmlv3.q4_0.bin
            # MPT-30B
            DEFAULT_MODEL_HG_REPO_ID: TheBloke/mpt-30B-GGML
            DEFAULT_MODEL_FILE: mpt-30b.ggmlv0.q4_1.bin
            DEFAULT_MODEL_META: ""
        # Request more resource if needed
        resources:
          {}
        # pvc for storing the cache
        cache:
          persistence:
            size: 5Gi
            accessModes:
              - ReadWriteOnce
            storageClass: ~
        cacheMountPath: /app/cache
        # pvc for storing the models
        model:
          persistence:
            size: 20Gi
            accessModes:
              - ReadWriteOnce
            storageClass: ~
        modelMountPath: /app/models
        service:
          type: ClusterIP
          port: 8000
          annotations: {}
        # You might want to use the following to select a node with more CPU and memory
        # for MPT-30B, we need at least 32GB of memory
        nodeSelector: {}
        tolerations: []
        affinity: {}
```

Let's wait for few seconds for the pods to become green, I am using [Lens](https://github.com/lensapp/lens), it's awesome btw.
![Waiting for pending pods](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d7jy8twpu7x43mg76cts.png)

When all pods are green. We are ready for the next step.
![Pods are ready](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/clbgbqscmj6i0rmmw5r7.png)

The private AI app we are developing is a simple web server with an endpoint `POST /prompt`. When a client sends a request with a `prompt` in the request body to `POST /prompt`, the endpoint's controller will forward the `prompt` to the backend [AI API](https://github.com/chenhunghan/ialacol), retrieve the response, and send it back to the client.

To begin, let's install the necessary dependencies on our local machine
```sh
python3 -m venv .venv
source .venv/bin/activate
pip install fastapi uvicorn
pip install openai # We are not using OpenAI API, but we can use openai client library to simplify things because our backend (ialacol) has OpenAI compatible RESTful interface.
pip freeze > requirements.txt
```

and create a `main.py` file.
```python
from fastapi import FastAPI
import openai
from pydantic import BaseModel

class Body(BaseModel):
    prompt: str
    
app = FastAPI()

@app.post("/prompt")
async def completions(
    body: Body
):
    prompt = body.prompt
    # Add more logics here, for example, you can add the context to the prompt
    # using context augmentation retrieval methods
    response = openai.Completion.create(
        prompt=prompt,
        model="mpt-30b.ggmlv0.q4_1.bin",
        temperature=0.5
    )
    completion = response.choices[0].text
    
    return completion
```

The implementation of our app's endpoint `POST /prompt` is straightforward. It acts as a proxy, forwarding the request to the backend. You can further extend it by incorporating additional functionality, such as context augmentation retrieval based on the provided `prompt`.

With the core functionality of the app in place, let's synchronize the source files to the cluster by running the command `devspace dev`. This command performs the following actions:

- It instructs devSpace to sync the files located at the root folder to the `/app` folder of the remote pod.
- Whenever changes are made to the `requirements.txt` file, it triggers a `pip install` within the pod.
- Additionally, it forwards port `8000`, allowing us to access the app at `http://localhost:8000`.

```yaml
dev:
  private-ai-app:
    # Use the label selector to select the pod for swapping out the container
    labelSelector:
      app.kubernetes.io/name: private-ai-app
    # use the name space we assign by devspace use namespace
    namespace: ${DEVSPACE_NAMESPACE}
    devImage: ghcr.io/loft-sh/devspace-containers/python:3-alpine
    workingDir: /app
    command: ["uvicorn"]
    args: ["main:app", "--reload", "--host", "0.0.0.0", "--port", "8000"]
    # expose the port 8000 to the host
    ports:
    - port: "8000:8000"
    # Add env for the pod if needed
    env:
    # This will tell openai python library to use the ialacol service instead of the OpenAI cloud
    - name: OPENAI_API_BASE
      value: "http://ialacol.${DEVSPACE_NAMESPACE}.svc.cluster.local:8000/v1"
    # You don't need to have an OpenAI API key, but OpenAI python library will complain without it
    - name: OPENAI_API_KEY
      value: "sk-xxx"
    sync:
      - path: ./:/app
        excludePaths:
        - requirements.txt
        printLogs: true
        uploadExcludeFile: ./.dockerignore
        downloadExcludeFile: ./.gitignore
      - path: ./requirements.txt:/app/requirements.txt
        # start the container after uploading the requirements.txt and install the dependencies
        startContainer: true
        file: true
        printLogs: true
        onUpload:
          exec:
          - command: |-
              pip install -r requirements.txt
            onChange: ["requirements.txt"]
    logs:
      enabled: true
      lastLines: 200
```

Wait for the files sync completed (you should see some logs in the terminal), and test our app by
```sh
curl -X POST -H 'Content-Type: application/json' -d '{ "prompt": "Hello!" }' http://localhost:8000/prompt
```
That's it, enjoy building your first private AI app  🥳!

Source code in the article [private-ai-app-starter-python](https://github.com/chenhunghan/private-ai-app-starter-python)