Repository: ollama
Issue: docker container can't detect Nvidia GPU - intermittent "cuda driver library failed to get device context 801"
Labels: bug,nvidia,needs more info,docker

### What is the issue?

OS: Ubuntu 24.04 LTS
GPU: Nvidia Tesla P40 (24G)

I installed ollama without docker and it was able to utilise my gpu without any issues.
I then deployed ollama using the following docker compose file:
```
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    environment:
      - PUID=${PUID:-1000}
      - PGID=${PGID:-1000}
      - OLLAMA_KEEP_ALIVE=24h
      - ENABLE_IMAGE_GENERATION=True
      - COMFYUI_BASE_URL=http://stable-diffusion-webui:7860
    networks:
      - traefik
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./ollama:/root/.ollama
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.ollama.rule=Host(`ollama.local.example.com`)"
      - "traefik.http.routers.ollama.entrypoints=https"
      - "traefik.http.routers.ollama.tls=true"
      - "traefik.http.routers.ollama.tls.certresolver=cloudflare"
      - "traefik.http.routers.ollama.middlewares=default-headers@file"
      - "traefik.http.routers.ollama.middlewares=ollama-auth"
      - "traefik.http.services.ollama.loadbalancer.server.port=11434"
      - "traefik.http.routers.ollama.middlewares=auth"
      - "traefik.http.middlewares.auth.basicauth.users=${OLLAMA_API_CREDENTIALS}"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
```
When I exec in to the container and run `nvidia-smi` it successfully executes it from `within` the ollama docker container.
but the logs show that it can't detect my gpu?

```
2024/08/14 22:50:17 routes.go:1123: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:24h0m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]"
time=2024-08-14T22:50:18.674+01:00 level=INFO source=images.go:782 msg="total blobs: 5"
time=2024-08-14T22:50:18.675+01:00 level=INFO source=images.go:790 msg="total unused blobs removed: 0"
time=2024-08-14T22:50:18.677+01:00 level=INFO source=routes.go:1170 msg="Listening on [::]:11434 (version 0.3.5)"
time=2024-08-14T22:50:18.678+01:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama2940291930/runners
time=2024-08-14T22:50:30.626+01:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11 rocm_v60102]"
time=2024-08-14T22:50:30.626+01:00 level=INFO source=gpu.go:204 msg="looking for compatible GPUs"
time=2024-08-14T22:50:30.640+01:00 level=INFO source=gpu.go:260 msg="error looking up nvidia GPU memory" error="cuda driver library failed to get device context 801"
time=2024-08-14T22:50:30.640+01:00 level=INFO source=gpu.go:350 msg="no compatible GPUs were discovered"
time=2024-08-14T22:50:30.640+01:00 level=INFO source=types.go:105 msg="inference compute" id=0 library=cpu compute="" driver=0.0 name="" total="47.1 GiB" available="43.9 GiB"
2024/08/14 22:54:19 routes.go:1123: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:24h0m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]"
time=2024-08-14T22:54:19.967+01:00 level=INFO source=images.go:782 msg="total blobs: 5"
time=2024-08-14T22:54:20.012+01:00 level=INFO source=images.go:790 msg="total unused blobs removed: 0"
time=2024-08-14T22:54:20.013+01:00 level=INFO source=routes.go:1170 msg="Listening on [::]:11434 (version 0.3.5)"
time=2024-08-14T22:54:20.032+01:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama1278819119/runners
```

not sure why??

### OS

Linux, Docker

### GPU

Nvidia

### CPU

Intel

### Ollama version

0.3.5