Deploy Mistral LLM on Google Compute Engine with Docker, GPU Support, and Hugging Face Inference Server
Tags: llm, cloud, ai, mlops
Published: 2024-01-21 20:21:46

## Introduction 
A practical guide on setting up Large Language Models (LLMs) on Google Compute Engine using GPUs. This guide is designed to walk you through the process step by step, making it easy for you to take advantage of the powerful combination of Google's cloud infrastructure and NVIDIA's GPU technology. 

## Machine Specs for the tutorial 
### Hardware Specifications:
#### GPU Information:
- GPU Type: Nvidia T4
- Number of GPUs: 2
- GPU Memory: 16 GB GDDR6 (per GPU)

#### Google compute engine Machine Type:
- Type: n1-highmem-4
- vCPUs: 4
- Cores: 2
- Memory: 26 GB

#### Disk Information:
- Disk Type: Balanced Persistent Disk
- Disk Size: 150 GB

### Software Specifications:
#### Operating System:
- Ubuntu Version: 20.04 LTS

#### CUDA version:
- CUDA version: 12.3

## 1.Setting Up Docker

Follow these simple steps to get Docker up and running on your system:

### 1.1 Adding Docker's Official GPG Key

add Docker’s official GPG key to your system. This step is crucial for validating the authenticity of the Docker packages you'll be installing
```
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
```

### 1.2 Adding Docker Repository to Apt Sources
Add Docker's repository to your system's Apt sources. This allows you to fetch Docker packages from their official repository:
```
echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
```
### 1.3 Installing Docker
Install Docker using the following command

```
sudo apt-get --reinstall install docker-ce
```

### 1.4 Create Docker Group
If not already present, add the 'docker' group to your system:
```
sudo groupadd docker
```

### 1.5 Add default User to Docker Group
Add your default user to the 'docker' group to manage Docker as a non-root user:
```
sudo usermod -aG docker $USER
```

### 1.6 check on Docker status 

```
systemctl status docker
```

---

## 2.Install NVIDIA Container Toolkit

### 2.1 Add NVIDIA GPG Key and NVIDIA Container Toolkit Repository
Start by adding the NVIDIA GPG key to ensure the authenticity of the software packages and add the NVIDIA Container Toolkit repository to your system's software sources:

```
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
```

### 2.2 Enable Experimental Features (Optional)
If you wish to use experimental features, uncomment the respective lines in the sources list:

```
sudo sed -i -e '/experimental/ s/^#//g' /etc/apt/sources.list.d/nvidia-container-toolkit.list
```

### 2.3 Update Package Index and Install NVIDIA Toollit
pdate your package index and install the NVIDIA Container Toolkit:

```
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
```

---

## 3.Configure Container Toolkit
### 3.1 Configure NVIDIA Container Toolkit
Configure the NVIDIA Container Toolkit to work with Docker:

```
sudo nvidia-ctk runtime configure --runtime=docker
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ijs3b8cbudlgzbr8nkym.png)



### 3.2 Restart Docker Service
Apply the changes by restarting the Docker service:

```
sudo systemctl restart docker
```

---

## 4.Prerequisites Before Installing CUDA Drivers

Ensure your system meets the following prerequisites before proceeding with the CUDA driver installation. For detailed guidance, refer to the official NVIDIA CUDA installation guide (https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#pre-installation-actions).

### 4.1 Verify CUDA-Capable GPU
First, confirm that your system has an NVIDIA GPU installed, this command should return information about the NVIDIA graphics card if one is present.
```
lspci | grep -i nvidia
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vfesb9cnkln1dauq7dy0.png)



### 4.2 Confirm Supported Linux Version
Ensure your Linux distribution is supported by checking its version, this command will display the architecture of your system and details about your Linux distribution : 
```
uname -m && cat /etc/*release
```

### 4.3 Check Kernel Headers and Development Packages
Verify that your system has the appropriate kernel headers and development packages, which are essential for building the NVIDIA kernel module:

```
uname -r
```
---
## 5. Installing NVIDIA Drivers

Follow these steps to install NVIDIA drivers on your system. For detailed instructions, you can refer to the NVIDIA Tesla Installation Notes (https://docs.nvidia.com/datacenter/tesla/tesla-installation-notes/index.html).

### 5.1 Install Required Kernel Headers
Start by installing the Linux kernel headers corresponding to your current kernel version:

```
sudo apt-get install linux-headers-$(uname -r)
```

### 5.2 Add the NVIDIA CUDA Repository
Identify your distribution's version and add the NVIDIA CUDA repository to your system:

```
distribution=$(. /etc/os-release;echo $ID$VERSION_ID | sed -e 's/\.//g')
wget https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring_1.0-1_all.deb
```

### 5.3 Update and Install NVIDIA Drivers
Finally, update your package lists and install the CUDA drivers, after the installation it will need a restart
```
sudo apt-get update
sudo apt-get -y install cuda-drivers
```
---

## 6. Post-Installation Steps for NVIDIA Driver

After successfully installing the NVIDIA drivers, perform the following post-installation steps to ensure everything is set up correctly. For a comprehensive guide, consult the NVIDIA CUDA Installation Guide for Linux (https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#post-installation-actions).

### 6.1 Verify NVIDIA Persistence Daemon
Check the status of the NVIDIA Persistence Daemon to ensure it's running correctly:
```
systemctl status nvidia-persistenced
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/82adao7n5ih3asgcok4q.png)


### 6.2 Monitor GPU Utilization
To confirm that your GPU is recognized and monitor its utilization, use:

```
nvidia-smi
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1svq0ged70yt0pb0qtn7.png)

---

## Define model configuration 
To set up the model configuration, you can use the following environment variables, the variable model is set to mistralai/Mistral-7B-v0.1, representing the Mistral-7B-v0.1 model for the tutorial, The variable volume is set to the present working directory ($PWD) followed by /data, indicating the directory path where data will be stored :
```
export model=mistralai/Mistral-7B-v0.1
export volume=$PWD/data 
```

## Run text-generation-inference using Docker 
To perform text generation inference, we will will use the Huggingface text generation inference server (for more details check this url https://huggingface.co/docs/text-generation-inference/index), execute the following Docker command with the following parameters, and also here is the parameters explanation:

- --gpus all: Enables GPU support for Docker containers.
- --shm-size 1g: Sets the shared memory size to 1 gigabyte.
- -p 8080:80: Maps port 8080 on the host to port 80 in the Docker container.
- -v $volume:/data: Mounts the local data volume specified by $volume inside the Docker container at the /data path.
- ghcr.io/huggingface/text-generation-inference:1.3: Specifies the Docker image for text-generation-inference with the version tag 1.3.
- --model-id $model: Passes the specified model identifier ($model) to the text-generation-inference application.
 
```
docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:1.3 --model-id $model
```

## check GPU utilisation 
Run again the GPU monitoring command to check the memory utilization after loading model weights into the GPU memory :
```
nvidia-smi
```
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6fh2mfw3kazms45meqli.png)


## Test API endpoint 
To test the API endpoint, use the following curl command:

```
curl 127.0.0.1:8080/generate -X POST -d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":20}}' -H 'Content-Type: application/json'
```
 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/igwgqbv5kp5b4tvobzmn.png)