Repository: ollama
Issue: Detecting macOS GPUs when using Podman with GPU passthrough
Labels: feature request

Podman provides support for making the local GPU on a macOS computer available from within a container. This article describes the setup for it: https://blog.podman.io/2024/07/podman-and-libkrun/.

```shell
% podman machine ssh ls -l /dev/dri
total 0
drwxr-xr-x. 2 root root         80 Jul 25 17:12 by-path
crw-rw----. 1 root video  226,   0 Jul 25 17:12 card0
crw-rw-rw-. 1 root render 226, 128 Jul 25 17:12 renderD128
```

When I run an Ollama container, it doesn't seem it can recognise the GPU. Is there some option I can use to make that work or is some new implementation needed within the Ollama project to support that?

```shell
docker run -it --rm -p 11434:11434 --device /dev/dri -e OLLAMA_DEBUG=true ollama/ollama
```

Logs:

```
2024/07/25 15:56:42 routes.go:1100: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]"
time=2024-07-25T15:56:42.248Z level=INFO source=images.go:784 msg="total blobs: 0"
time=2024-07-25T15:56:42.248Z level=INFO source=images.go:791 msg="total unused blobs removed: 0"
time=2024-07-25T15:56:42.248Z level=INFO source=routes.go:1147 msg="Listening on [::]:11434 (version 0.2.8)"
time=2024-07-25T15:56:42.249Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama2409035362/runners
time=2024-07-25T15:56:42.249Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu file=build/linux/arm64/cpu/bin/ollama_llama_server.gz
time=2024-07-25T15:56:42.249Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/arm64/cuda_v11/bin/libcublas.so.11.gz
time=2024-07-25T15:56:42.249Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/arm64/cuda_v11/bin/libcublasLt.so.11.gz
time=2024-07-25T15:56:42.249Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/arm64/cuda_v11/bin/libcudart.so.11.0.gz
time=2024-07-25T15:56:42.249Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/arm64/cuda_v11/bin/ollama_llama_server.gz
time=2024-07-25T15:56:47.047Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2409035362/runners/cpu/ollama_llama_server
time=2024-07-25T15:56:47.047Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2409035362/runners/cuda_v11/ollama_llama_server
time=2024-07-25T15:56:47.047Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cuda_v11]"
time=2024-07-25T15:56:47.047Z level=DEBUG source=payload.go:45 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2024-07-25T15:56:47.047Z level=DEBUG source=sched.go:102 msg="starting llm scheduler"
time=2024-07-25T15:56:47.047Z level=INFO source=gpu.go:205 msg="looking for compatible GPUs"
time=2024-07-25T15:56:47.048Z level=DEBUG source=gpu.go:91 msg="searching for GPU discovery libraries for NVIDIA"
time=2024-07-25T15:56:47.048Z level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libcuda.so*
time=2024-07-25T15:56:47.049Z level=DEBUG source=gpu.go:487 msg="gpu library search" globs="[/usr/local/nvidia/lib/libcuda.so** /usr/local/nvidia/lib64/libcuda.so** /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
time=2024-07-25T15:56:47.049Z level=DEBUG source=gpu.go:521 msg="discovered GPU libraries" paths=[]
time=2024-07-25T15:56:47.049Z level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libcudart.so*
time=2024-07-25T15:56:47.049Z level=DEBUG source=gpu.go:487 msg="gpu library search" globs="[/usr/local/nvidia/lib/libcudart.so** /usr/local/nvidia/lib64/libcudart.so** /tmp/ollama2409035362/runners/cuda*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]"
time=2024-07-25T15:56:47.049Z level=DEBUG source=gpu.go:521 msg="discovered GPU libraries" paths=[/tmp/ollama2409035362/runners/cuda_v11/libcudart.so.11.0]
cudaSetDevice err: 35
time=2024-07-25T15:56:47.057Z level=DEBUG source=gpu.go:533 msg="Unable to load cudart" library=/tmp/ollama2409035362/runners/cuda_v11/libcudart.so.11.0 error="your nvidia driver is too old or missing.  If you have a CUDA GPU please upgrade to run ollama"
time=2024-07-25T15:56:47.057Z level=DEBUG source=amd_linux.go:356 msg="amdgpu driver not detected /sys/module/amdgpu"
time=2024-07-25T15:56:47.057Z level=INFO source=gpu.go:346 msg="no compatible GPUs were discovered"
time=2024-07-25T15:56:47.057Z level=INFO source=types.go:105 msg="inference compute" id=0 library=cpu compute="" driver=0.0 name="" total="4.4 GiB" available="4.0 GiB"
```