Repository: ollama
Issue: ollama 0.5.1 is detecting my NVIDIA Tesla M40, but they are not used. 
Labels: bug

### What is the issue?

ollama 0.5.1 binary distribution is recognising the TESLA M40:

```
Dec 12 07:11:11 bigrig ollama[362206]: time=2024-12-12T07:11:11.731Z level=INFO source=types.go:123 msg="inference compute" id=GPU-c8f87326-45f6-945a-1a1a-63bd9a7fc262 library=cuda variant=v12 compute=8.6 driver=12.2 name="NVIDIA GeForce RTX 3090" total="23.7 GiB" available="23.4 GiB"
Dec 12 07:11:11 bigrig ollama[362206]: time=2024-12-12T07:11:11.731Z level=INFO source=types.go:123 msg="inference compute" id=GPU-603a9272-c602-62ea-4090-51223189bb8f library=cuda variant=v12 compute=7.5 driver=12.2 name="Tesla T4" total="15.6 GiB" available="15.5 GiB"
Dec 12 07:11:11 bigrig ollama[362206]: time=2024-12-12T07:11:11.731Z level=INFO source=types.go:123 msg="inference compute" id=GPU-4fe5252f-aa78-f5f8-958a-5a8ae3ffe9e4 library=cuda variant=v12 compute=7.5 driver=12.2 name="Tesla T4" total="14.6 GiB" available="14.5 GiB"
Dec 12 07:11:11 bigrig ollama[362206]: time=2024-12-12T07:11:11.731Z level=INFO source=types.go:123 msg="inference compute" id=GPU-e6b47121-0d1d-8c63-1ab0-14012d5eb87f library=cuda variant=v12 compute=6.1 driver=12.2 name="Tesla P40" total="23.9 GiB" available="23.7 GiB"
Dec 12 07:11:11 bigrig ollama[362206]: time=2024-12-12T07:11:11.731Z level=INFO source=types.go:123 msg="inference compute" id=GPU-079cdcf9-556e-2f0c-6e6d-042eec929d92 library=cuda variant=v11 compute=5.2 driver=12.2 name="Tesla M40 24GB" total="23.9 GiB" available="23.8 GiB"
Dec 12 07:11:11 bigrig ollama[362206]: time=2024-12-12T07:11:11.731Z level=INFO source=types.go:123 msg="inference compute" id=GPU-41ac58ae-d8b7-afdb-c25f-ca6f09b57999 library=cuda variant=v11 compute=5.2 driver=12.2 name="Tesla M40 24GB" total="23.9 GiB" available="23.8 GiB"
```

But later on, only the other GPUs are used:
```
Dec 12 07:14:01 bigrig ollama[362206]: ggml_cuda_init: found 4 CUDA devices:
Dec 12 07:14:01 bigrig ollama[362206]:   Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Dec 12 07:14:01 bigrig ollama[362206]:   Device 1: Tesla T4, compute capability 7.5, VMM: yes
Dec 12 07:14:01 bigrig ollama[362206]:   Device 2: Tesla T4, compute capability 7.5, VMM: yes
Dec 12 07:14:01 bigrig ollama[362206]:   Device 3: Tesla P40, compute capability 6.1, VMM: yes
Dec 12 07:14:01 bigrig ollama[362206]: llm_load_tensors: ggml ctx size =    2.00 MiB
Dec 12 07:14:41 bigrig ollama[362206]: llm_load_tensors: offloading 30 repeating layers to GPU
Dec 12 07:14:41 bigrig ollama[362206]: llm_load_tensors: offloaded 30/61 layers to GPU
Dec 12 07:14:41 bigrig ollama[362206]: llm_load_tensors:  CUDA_Host buffer size = 62745.29 MiB
Dec 12 07:14:41 bigrig ollama[362206]: llm_load_tensors:      CUDA0 buffer size = 21335.35 MiB
Dec 12 07:14:41 bigrig ollama[362206]: llm_load_tensors:      CUDA1 buffer size = 12801.21 MiB
Dec 12 07:14:41 bigrig ollama[362206]: llm_load_tensors:      CUDA2 buffer size = 10667.68 MiB
Dec 12 07:14:41 bigrig ollama[362206]: llm_load_tensors:      CUDA3 buffer size = 19201.82 MiB
```

Compiling ollama from source, explicitely with setting the architectures (export CMAKE_CUDA_ARCHITECTURES="50;52;61;70;75;80;90") does not change the behaviour.


[ollama-logs-M40.txt](https://github.com/user-attachments/files/18107418/ollama-logs-M40.txt)

Running _/usr/local/bin/ollama run deepseek-coder-v2:236b_ for testing. This model should be big enough to cause ollama to fill all the GPUs. But it does not do that.

![image](https://github.com/user-attachments/assets/3eb96e36-2ebb-45de-9404-e7cacd822e4e)

I already tried to invoke different CUDA-Versions by update-alternatives, but to no avail.

BTW: _export CUDA_VISIBLE_DEVICES=4,5_ does not have any effect

I have a llama.cpp 2749, self-compiled, which is using the M40. Since the output is nearly identical to the one in the ollama-log, which is only recognizing 4 CUDA-Devices, I suspect the problem is somehow related with the llama.cpp shipped with ollama 0.5.1.  This is the output in question:

```
Log start
main: build = 2749 (928e0b70)
main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
main: seed  = 1733989418
...
ggml_cuda_init: found 6 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  Device 1: Tesla T4, compute capability 7.5, VMM: yes
  Device 2: Tesla T4, compute capability 7.5, VMM: yes
  Device 3: Tesla P40, compute capability 6.1, VMM: yes
  Device 4: Tesla M40 24GB, compute capability 5.2, VMM: yes
  Device 5: Tesla M40 24GB, compute capability 5.2, VMM: yes

```



### OS

Linux

### GPU

Nvidia

### CPU

AMD

### Ollama version

0.5.1