Running Ollama 2 on NVIDIA Jetson Nano with GPU using Docker
Tags: docker, nvidia, gpu, ollama
Published: 2023-11-04 17:06:06

[Ollama](https://hub.docker.com/r/ollama/ollama) is a rapidly growing development tool, with 10,000 Docker Hub pulls in a short period of time. It is a large language model (LLM) from Google AI that is trained on a massive dataset of text and code. It can generate text, translate languages, write different kinds of creative content, and answer your questions in an informative way.

To run OLLAMA on a Jetson Nano, you will need to install the following software:

- Docker Engine
- OLLAMA Docker image

## Installing Docker

To install Docker on a Jetson Nano, follow these steps:

Update the package list:

```
sudo apt update
```

## Install Docker:

```
sudo curl -sSL https://get.docker.com/ | sh
```

## Add your user to the Docker group:

```
sudo groupadd docker
sudo usermod -aG docker $USER
```

Log out and back in for the changes to take effect.

## Install with Apt

Configure the repository

```
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
    | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
    | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
    | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
```

## Install the NVIDIA Container Toolkit packages

```
sudo apt-get install -y nvidia-container-toolkit
```

## Configure Docker to use Nvidia driver

```
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
```

## Start the container

```
sudo docker run -d --gpus=all --runtime=nvidia -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
```

## Run model locally

Now you can run a model:

```
sudo docker exec -it ollama ollama run llama2
```

```
sudo docker exec -it ollama ollama run llama2
pulling manifest
pulling 8daa9615cce3... 100% |████████████████████████████| (3.8/3.8 GB, 2.4 MB/s)
pulling 8c17c2ebb0ea... 100% |████████████████████████████████| (7.0/7.0 kB, 3.7 kB/s)
pulling 7c23fb36d801... 100% |████████████████████████████████| (4.8/4.8 kB, 2.0 kB/s)
pulling bec56154823a... 100% |█████████████████████████████████████| (59/59 B, 18 B/s)
pulling e35ab70a78c7... 100% |█████████████████████████████████████| (90/90 B, 32 B/s)
pulling 09fe89200c09... 100% |██████████████████████████████████| (529/529 B, 180 B/s)
verifying sha256 digest
writing manifest
removing any unused layers
success
```

The command `sudo docker exec -it ollama ollama run llama2` will start the OLLAMA 2 model in the ollama container. This will allow you to interact with the model directly from the command line.

To use the OLLAMA 2 model, you can send it text prompts and it will generate text in response. For example, to generate a poem about a cat, you would run the following command:

```
docker exec -it ollama ollama run llama2 --prompt "Write a poem about a cat."
```

This will generate a poem about a cat and print it to the console.

You can also use the OLLAMA 2 model to translate languages, write different kinds of creative content, and answer your questions in an informative way.

Experiment with different prompts to test the capabilities of the OLLAMA 2 model.

Here are some examples of prompts you can use with the OLLAMA 2 model:

- Translate the sentence "Hello, world!" into Spanish.
- Write a short story about a robot who falls in love with a human.
- Generate a list of ideas for new products.
- Answer the question "What is the meaning of life?"

The OLLAMA 2 model is still under development, but it has the potential to be a powerful tool for a variety of tasks.