Repository: ollama
Issue: Error: llama runner process has terminated: error loading model: unable to allocate CUDA0 buffer (4x L40S, 384GB system RAM, Deepseek-R1)
Labels: bug

### What is the issue?

I am aware that 4 x L40S only has 192GB of VRAM, but the machine also has 384GB of system RAM.  The error seems to indicate that 45108 MiB of RAM is being allocated with cudaMalloc and this is failing. This is very close to the GPU limit (46068 MiB). On my home setup (2x P40, admittedly not trying such a huge model) the GPU never gets close to its VRAM limit (typically only 22GB out of 24GB).


```
Jan 26 17:48:20 ip-172-31-3-6 ollama[1418]: llm_load_print_meta: rope_yarn_log_mul    = 0.1000
Jan 26 17:50:35 ip-172-31-3-6 ollama[1418]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 45108.64 MiB on device 0: cudaMalloc failed: out of memory
Jan 26 17:50:35 ip-172-31-3-6 ollama[1418]: llama_model_load: error loading model: unable to allocate CUDA0 buffer
Jan 26 17:50:35 ip-172-31-3-6 ollama[1418]: llama_load_model_from_file: failed to load model
Jan 26 17:51:23 ip-172-31-3-6 ollama[1418]: panic: unable to load model: /usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9
Jan 26 17:51:23 ip-172-31-3-6 ollama[1418]: goroutine 34 [running]:
Jan 26 17:51:23 ip-172-31-3-6 ollama[1418]: github.com/ollama/ollama/llama/runner.(*Server).loadModel(0xc0001a0000, {0x1b, 0x0, 0x0, 0x0, {0xc000194090, 0x4, 0x4}, 0xc00018a060, 0x0}, ...)
Jan 26 17:51:23 ip-172-31-3-6 ollama[1418]:         github.com/ollama/ollama/llama/runner/runner.go:852 +0x3ad
Jan 26 17:51:23 ip-172-31-3-6 ollama[1418]: created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1
Jan 26 17:51:23 ip-172-31-3-6 ollama[1418]:         github.com/ollama/ollama/llama/runner/runner.go:970 +0xd0d
Jan 26 17:51:24 ip-172-31-3-6 ollama[1418]: time=2025-01-26T17:51:24.077Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
Jan 26 17:51:24 ip-172-31-3-6 ollama[1418]: time=2025-01-26T17:51:24.578Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error loading model: unable to allocate CUDA0 buffer\nllama_load_model_from_file: failed to load model"
Jan 26 17:51:24 ip-172-31-3-6 ollama[1418]: [GIN] 2025/01/26 - 17:51:24 | 500 |          3m6s |       127.0.0.1 | POST     "/api/generate"
Jan 26 17:51:30 ip-172-31-3-6 ollama[1418]: time=2025-01-26T17:51:30.190Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.6116838510000004 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1>
Jan 26 17:51:30 ip-172-31-3-6 ollama[1418]: time=2025-01-26T17:51:30.947Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=6.369278328 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9
Jan 26 17:51:31 ip-172-31-3-6 ollama[1418]: time=2025-01-26T17:51:31.707Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=7.128851916 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9
Jan 26 17:51:40 ip-172-31-3-6 ollama[1418]: [GIN] 2025/01/26 - 17:51:40 | 200 |      65.972µs |       127.0.0.1 | GET      "/api/version"


ubuntu@ip-172-31-3-6:~$ ollama --version
ollama version is 0.5.7
ubuntu@ip-172-31-3-6:~$ free
               total        used        free      shared  buff/cache   available
Mem:       390837004     4152012   385439624        3248     1245368   383763992
Swap:              0           0           0


ubuntu@ip-172-31-3-6:~$ nvidia-smi
Sun Jan 26 17:48:08 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03             Driver Version: 550.144.03     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L40S                    On  |   00000000:38:00.0 Off |                    0 |
| N/A   22C    P8             22W /  350W |       4MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA L40S                    On  |   00000000:3A:00.0 Off |                    0 |
| N/A   22C    P8             21W /  350W |       4MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA L40S                    On  |   00000000:3C:00.0 Off |                    0 |
| N/A   22C    P8             44W /  350W |       4MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA L40S                    On  |   00000000:3E:00.0 Off |                    0 |
| N/A   23C    P8             22W /  350W |       4MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
```

### OS

Linux

### GPU

Nvidia

### CPU

AMD

### Ollama version

0.5.7