Repository: ollama
Issue: ollama only using cpu even with gpu found
Labels: bug

### What is the issue?

hello,
this has been reported in the past at least two times, I am here to report it a third time because something doesnt seem right.

relevant issues:
https://github.com/ollama/ollama/issues/8485
https://github.com/ollama/ollama/issues/8467

same error, same fix with ' just reinstalling within the same session no reboot no nothing just magically fixes it.

installing from official arch repos also causes the gpu not to be used, any way to .. fix that?


`

> 2025/01/22 09:42:17 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/nylle/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
> time=2025-01-22T09:42:17.713+01:00 level=INFO source=images.go:432 msg="total blobs: 30"
> time=2025-01-22T09:42:17.713+01:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
> [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
> 
> [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
>  - using env:   export GIN_MODE=release
>  - using code:  gin.SetMode(gin.ReleaseMode)
> 
> [GIN-debug] POST   /api/pull                 --> github.com/ollama/ollama/server.(*Server).PullHandler-fm (5 handlers)
> [GIN-debug] POST   /api/generate             --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers)
> [GIN-debug] POST   /api/chat                 --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers)
> [GIN-debug] POST   /api/embed                --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (5 handlers)
> [GIN-debug] POST   /api/embeddings           --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers)
> [GIN-debug] POST   /api/create               --> github.com/ollama/ollama/server.(*Server).CreateHandler-fm (5 handlers)
> [GIN-debug] POST   /api/push                 --> github.com/ollama/ollama/server.(*Server).PushHandler-fm (5 handlers)
> [GIN-debug] POST   /api/copy                 --> github.com/ollama/ollama/server.(*Server).CopyHandler-fm (5 handlers)
> [GIN-debug] DELETE /api/delete               --> github.com/ollama/ollama/server.(*Server).DeleteHandler-fm (5 handlers)
> [GIN-debug] POST   /api/show                 --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (5 handlers)
> [GIN-debug] POST   /api/blobs/:digest        --> github.com/ollama/ollama/server.(*Server).CreateBlobHandler-fm (5 handlers)
> [GIN-debug] HEAD   /api/blobs/:digest        --> github.com/ollama/ollama/server.(*Server).HeadBlobHandler-fm (5 handlers)
> [GIN-debug] GET    /api/ps                   --> github.com/ollama/ollama/server.(*Server).PsHandler-fm (5 handlers)
> [GIN-debug] POST   /v1/chat/completions      --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers)
> [GIN-debug] POST   /v1/completions           --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (6 handlers)
> [GIN-debug] POST   /v1/embeddings            --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (6 handlers)
> [GIN-debug] GET    /v1/models                --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (6 handlers)
> [GIN-debug] GET    /v1/models/:model         --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (6 handlers)
> [GIN-debug] GET    /                         --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
> [GIN-debug] GET    /api/tags                 --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers)
> [GIN-debug] GET    /api/version              --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
> [GIN-debug] HEAD   /                         --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
> [GIN-debug] HEAD   /api/tags                 --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers)
> [GIN-debug] HEAD   /api/version              --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
> time=2025-01-22T09:42:17.714+01:00 level=INFO source=routes.go:1238 msg="Listening on 127.0.0.1:11434 (version 0.5.7)"
> time=2025-01-22T09:42:17.714+01:00 level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners=[cpu]
> time=2025-01-22T09:42:17.715+01:00 level=INFO source=gpu.go:226 msg="looking for compatible GPUs"
> time=2025-01-22T09:42:17.991+01:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-f16d0b56-1989-0c40-d33b-480f8247ae00 library=cuda variant=v12 compute=8.9 driver=12.7 name="NVIDIA GeForce RTX 4070" total="11.6 GiB" available="10.9 GiB"
> [GIN] 2025/01/22 - 09:42:29 | 200 |    2.178715ms |       127.0.0.1 | GET      "/api/tags"
> [GIN] 2025/01/22 - 09:42:29 | 200 |    2.279349ms |       127.0.0.1 | GET      "/api/tags"
> [GIN] 2025/01/22 - 09:42:29 | 200 |     569.677µs |       127.0.0.1 | GET      "/api/tags"
> [GIN] 2025/01/22 - 09:42:29 | 200 |     641.407µs |       127.0.0.1 | GET      "/api/tags"
> [GIN] 2025/01/22 - 09:42:29 | 200 |     952.749µs |       127.0.0.1 | GET      "/api/tags"
> [GIN] 2025/01/22 - 09:42:29 | 404 |     2.06483ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 404 |    3.234779ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 404 |     708.286µs |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 404 |    4.649411ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 404 |     507.699µs |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   15.227486ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   14.713616ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   16.811971ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   14.913503ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |    16.80457ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   11.736522ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   14.276927ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   15.946584ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   16.378602ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   16.905593ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   19.674409ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |    9.267617ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   18.489808ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |    9.548855ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   14.015642ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   18.927918ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   10.819718ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   20.026506ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   20.160486ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   20.971805ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   22.430443ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   20.434382ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   23.104035ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   19.600649ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   12.181043ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |    9.689575ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   11.553348ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   23.011692ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   24.279914ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   24.367486ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   21.774185ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   23.247884ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   17.658695ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   23.333246ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   19.259493ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   14.342186ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   14.479045ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   26.379969ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   12.077609ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   24.800855ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   17.678988ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   27.778669ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   28.465122ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   21.794806ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   33.221807ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |   26.157838ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |    6.808224ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |    7.428048ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |    7.468563ms |       127.0.0.1 | POST     "/api/show"
> [GIN] 2025/01/22 - 09:42:29 | 200 |    7.715287ms |       127.0.0.1 | POST     "/api/show"
> time=2025-01-22T09:42:34.091+01:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/home/nylle/.ollama/models/blobs/sha256-e04bbddd58d9290a89af21ef484ce1113ff34ef35e822e95e52ff1045bac17f5 gpu=GPU-f16d0b56-1989-0c40-d33b-480f8247ae00 parallel=1 available=11592138752 required="9.1 GiB"
> time=2025-01-22T09:42:34.212+01:00 level=INFO source=server.go:104 msg="system memory" total="31.3 GiB" free="26.3 GiB" free_swap="32.0 GiB"
> time=2025-01-22T09:42:34.213+01:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[10.8 GiB]" memory.gpu_overhead="0 B" memory.required.full="9.1 GiB" memory.required.partial="9.1 GiB" memory.required.kv="4.0 GiB" memory.required.allocations="[9.1 GiB]" memory.weights.total="7.7 GiB" memory.weights.repeating="7.6 GiB" memory.weights.nonrepeating="103.4 MiB" memory.graph.full="553.8 MiB" memory.graph.partial="673.3 MiB"
> time=2025-01-22T09:42:34.213+01:00 level=INFO source=server.go:376 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /home/nylle/.ollama/models/blobs/sha256-e04bbddd58d9290a89af21ef484ce1113ff34ef35e822e95e52ff1045bac17f5 --ctx-size 8096 --batch-size 512 --n-gpu-layers 33 --threads 16 --parallel 1 --port 42953"
> time=2025-01-22T09:42:34.213+01:00 level=INFO source=sched.go:449 msg="loaded runners" count=1
> time=2025-01-22T09:42:34.213+01:00 level=INFO source=server.go:555 msg="waiting for llama runner to start responding"
> time=2025-01-22T09:42:34.214+01:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
> time=2025-01-22T09:42:34.223+01:00 level=INFO source=runner.go:936 msg="starting go runner"
> time=2025-01-22T09:42:34.223+01:00 level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | AARCH64_REPACK = 1 | CPU : LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=16
> time=2025-01-22T09:42:34.223+01:00 level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:42953"
> llama_model_loader: loaded meta data with 25 key-value pairs and 291 tensors from /home/nylle/.ollama/models/blobs/sha256-e04bbddd58d9290a89af21ef484ce1113ff34ef35e822e95e52ff1045bac17f5 (version GGUF V3 (latest))
> llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
> llama_model_loader: - kv   0:                       general.architecture str              = llama
> llama_model_loader: - kv   1:                               general.name str              = deepseek-ai
> llama_model_loader: - kv   2:                       llama.context_length u32              = 16384
> llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
> llama_model_loader: - kv   4:                          llama.block_count u32              = 32
> llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008
> llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
> llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
> llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32
> llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000001
> llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 100000.000000
> llama_model_loader: - kv  11:                    llama.rope.scaling.type str              = linear
> llama_model_loader: - kv  12:                  llama.rope.scaling.factor f32              = 4.000000
> llama_model_loader: - kv  13:                          general.file_type u32              = 3
> llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = gpt2
> llama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,32256]   = ["!", "\"", "#", "$", "%", "&", "'", ...
> llama_model_loader: - kv  16:                      tokenizer.ggml.scores arr[f32,32256]   = [0.000000, 0.000000, 0.000000, 0.0000...
> llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,32256]   = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
> llama_model_loader: - kv  18:                      tokenizer.ggml.merges arr[str,31757]   = ["Ġ Ġ", "Ġ t", "Ġ a", "i n", "h e...
> llama_model_loader: - kv  19:                tokenizer.ggml.bos_token_id u32              = 32013
> llama_model_loader: - kv  20:                tokenizer.ggml.eos_token_id u32              = 32014
> llama_model_loader: - kv  21:            tokenizer.ggml.padding_token_id u32              = 32014
> llama_model_loader: - kv  22:               tokenizer.ggml.add_bos_token bool             = true
> llama_model_loader: - kv  23:               tokenizer.ggml.add_eos_token bool             = false
> llama_model_loader: - kv  24:               general.quantization_version u32              = 2
> llama_model_loader: - type  f32:   65 tensors
> llama_model_loader: - type q4_1:  225 tensors
> llama_model_loader: - type q6_K:    1 tensors
> llm_load_vocab: missing or unrecognized pre-tokenizer type, using: 'default'
> llm_load_vocab: control-looking token:  32015 '<｜fim▁hole｜>' was not control-type; this is probably a bug in the model. its type will be overridden
> llm_load_vocab: control-looking token:  32017 '<｜fim▁end｜>' was not control-type; this is probably a bug in the model. its type will be overridden
> llm_load_vocab: control-looking token:  32016 '<｜fim▁begin｜>' was not control-type; this is probably a bug in the model. its type will be overridden
> llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
> llm_load_vocab: special tokens cache size = 256
> llm_load_vocab: token to piece cache size = 0.1792 MB
> llm_load_print_meta: format           = GGUF V3 (latest)
> llm_load_print_meta: arch             = llama
> llm_load_print_meta: vocab type       = BPE
> llm_load_print_meta: n_vocab          = 32256
> llm_load_print_meta: n_merges         = 31757
> llm_load_print_meta: vocab_only       = 0
> llm_load_print_meta: n_ctx_train      = 16384
> llm_load_print_meta: n_embd           = 4096
> llm_load_print_meta: n_layer          = 32
> llm_load_print_meta: n_head           = 32
> llm_load_print_meta: n_head_kv        = 32
> llm_load_print_meta: n_rot            = 128
> llm_load_print_meta: n_swa            = 0
> llm_load_print_meta: n_embd_head_k    = 128
> llm_load_print_meta: n_embd_head_v    = 128
> llm_load_print_meta: n_gqa            = 1
> llm_load_print_meta: n_embd_k_gqa     = 4096
> llm_load_print_meta: n_embd_v_gqa     = 4096
> llm_load_print_meta: f_norm_eps       = 0.0e+00
> llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
> llm_load_print_meta: f_clamp_kqv      = 0.0e+00
> llm_load_print_meta: f_max_alibi_bias = 0.0e+00
> llm_load_print_meta: f_logit_scale    = 0.0e+00
> llm_load_print_meta: n_ff             = 11008
> llm_load_print_meta: n_expert         = 0
> llm_load_print_meta: n_expert_used    = 0
> llm_load_print_meta: causal attn      = 1
> llm_load_print_meta: pooling type     = 0
> llm_load_print_meta: rope type        = 0
> llm_load_print_meta: rope scaling     = linear
> llm_load_print_meta: freq_base_train  = 100000.0
> llm_load_print_meta: freq_scale_train = 0.25
> llm_load_print_meta: n_ctx_orig_yarn  = 16384
> llm_load_print_meta: rope_finetuned   = unknown
> llm_load_print_meta: ssm_d_conv       = 0
> llm_load_print_meta: ssm_d_inner      = 0
> llm_load_print_meta: ssm_d_state      = 0
> llm_load_print_meta: ssm_dt_rank      = 0
> llm_load_print_meta: ssm_dt_b_c_rms   = 0
> llm_load_print_meta: model type       = 7B
> llm_load_print_meta: model ftype      = Q4_1
> llm_load_print_meta: model params     = 6.74 B
> llm_load_print_meta: model size       = 3.95 GiB (5.03 BPW) 
> llm_load_print_meta: general.name     = deepseek-ai
> llm_load_print_meta: BOS token        = 32013 '<｜begin▁of▁sentence｜>'
> llm_load_print_meta: EOS token        = 32014 '<｜end▁of▁sentence｜>'
> llm_load_print_meta: EOT token        = 32014 '<｜end▁of▁sentence｜>'
> llm_load_print_meta: PAD token        = 32014 '<｜end▁of▁sentence｜>'
> llm_load_print_meta: LF token         = 126 'Ä'
> llm_load_print_meta: FIM PRE token    = 32016 '<｜fim▁begin｜>'
> llm_load_print_meta: FIM SUF token    = 32015 '<｜fim▁hole｜>'
> llm_load_print_meta: FIM MID token    = 32017 '<｜fim▁end｜>'
> llm_load_print_meta: EOG token        = 32014 '<｜end▁of▁sentence｜>'
> llm_load_print_meta: max token length = 128
> time=2025-01-22T09:42:34.465+01:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server loading model"
> llm_load_tensors:   CPU_Mapped model buffer size =  4043.12 MiB
> llama_new_context_with_model: n_seq_max     = 1
> llama_new_context_with_model: n_ctx         = 8096
> llama_new_context_with_model: n_ctx_per_seq = 8096
> llama_new_context_with_model: n_batch       = 512
> llama_new_context_with_model: n_ubatch      = 512
> llama_new_context_with_model: flash_attn    = 0
> llama_new_context_with_model: freq_base     = 100000.0
> llama_new_context_with_model: freq_scale    = 0.25
> llama_new_context_with_model: n_ctx_per_seq (8096) < n_ctx_train (16384) -- the full capacity of the model will not be utilized
> llama_kv_cache_init: kv_size = 8096, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 32, can_shift = 1
> llama_kv_cache_init:        CPU KV buffer size =  4048.00 MiB
> llama_new_context_with_model: KV self size  = 4048.00 MiB, K (f16): 2024.00 MiB, V (f16): 2024.00 MiB
> llama_new_context_with_model:        CPU  output buffer size =     0.14 MiB
> llama_new_context_with_model:        CPU compute buffer size =   553.82 MiB
> llama_new_context_with_model: graph nodes  = 1030
> llama_new_context_with_model: graph splits = 1
> time=2025-01-22T09:42:38.977+01:00 level=INFO source=server.go:594 msg="llama runner started in 4.76 seconds"
> llama_model_loader: loaded meta data with 25 key-value pairs and 291 tensors from /home/nylle/.ollama/models/blobs/sha256-e04bbddd58d9290a89af21ef484ce1113ff34ef35e822e95e52ff1045bac17f5 (version GGUF V3 (latest))
> llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
> llama_model_loader: - kv   0:                       general.architecture str              = llama
> llama_model_loader: - kv   1:                               general.name str              = deepseek-ai
> llama_model_loader: - kv   2:                       llama.context_length u32              = 16384
> llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
> llama_model_loader: - kv   4:                          llama.block_count u32              = 32
> llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008
> llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
> llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
> llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32
> llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000001
> llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 100000.000000
> llama_model_loader: - kv  11:                    llama.rope.scaling.type str              = linear
> llama_model_loader: - kv  12:                  llama.rope.scaling.factor f32              = 4.000000
> llama_model_loader: - kv  13:                          general.file_type u32              = 3
> llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = gpt2
> llama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,32256]   = ["!", "\"", "#", "$", "%", "&", "'", ...
> llama_model_loader: - kv  16:                      tokenizer.ggml.scores arr[f32,32256]   = [0.000000, 0.000000, 0.000000, 0.0000...
> llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,32256]   = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
> llama_model_loader: - kv  18:                      tokenizer.ggml.merges arr[str,31757]   = ["Ġ Ġ", "Ġ t", "Ġ a", "i n", "h e...
> llama_model_loader: - kv  19:                tokenizer.ggml.bos_token_id u32              = 32013
> llama_model_loader: - kv  20:                tokenizer.ggml.eos_token_id u32              = 32014
> llama_model_loader: - kv  21:            tokenizer.ggml.padding_token_id u32              = 32014
> llama_model_loader: - kv  22:               tokenizer.ggml.add_bos_token bool             = true
> llama_model_loader: - kv  23:               tokenizer.ggml.add_eos_token bool             = false
> llama_model_loader: - kv  24:               general.quantization_version u32              = 2
> llama_model_loader: - type  f32:   65 tensors
> llama_model_loader: - type q4_1:  225 tensors
> llama_model_loader: - type q6_K:    1 tensors
> llm_load_vocab: missing or unrecognized pre-tokenizer type, using: 'default'
> llm_load_vocab: control-looking token:  32015 '<｜fim▁hole｜>' was not control-type; this is probably a bug in the model. its type will be overridden
> llm_load_vocab: control-looking token:  32017 '<｜fim▁end｜>' was not control-type; this is probably a bug in the model. its type will be overridden
> llm_load_vocab: control-looking token:  32016 '<｜fim▁begin｜>' was not control-type; this is probably a bug in the model. its type will be overridden
> llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
> llm_load_vocab: special tokens cache size = 256
> llm_load_vocab: token to piece cache size = 0.1792 MB
> llm_load_print_meta: format           = GGUF V3 (latest)
> llm_load_print_meta: arch             = llama
> llm_load_print_meta: vocab type       = BPE
> llm_load_print_meta: n_vocab          = 32256
> llm_load_print_meta: n_merges         = 31757
> llm_load_print_meta: vocab_only       = 1
> llm_load_print_meta: model type       = ?B
> llm_load_print_meta: model ftype      = all F32
> llm_load_print_meta: model params     = 6.74 B
> llm_load_print_meta: model size       = 3.95 GiB (5.03 BPW) 
> llm_load_print_meta: general.name     = deepseek-ai
> llm_load_print_meta: BOS token        = 32013 '<｜begin▁of▁sentence｜>'
> llm_load_print_meta: EOS token        = 32014 '<｜end▁of▁sentence｜>'
> llm_load_print_meta: EOT token        = 32014 '<｜end▁of▁sentence｜>'
> llm_load_print_meta: PAD token        = 32014 '<｜end▁of▁sentence｜>'
> llm_load_print_meta: LF token         = 126 'Ä'
> llm_load_print_meta: FIM PRE token    = 32016 '<｜fim▁begin｜>'
> llm_load_print_meta: FIM SUF token    = 32015 '<｜fim▁hole｜>'
> llm_load_print_meta: FIM MID token    = 32017 '<｜fim▁end｜>'
> llm_load_print_meta: EOG token        = 32014 '<｜end▁of▁sentence｜>'
> llm_load_print_meta: max token length = 128
> llama_model_load: vocab only - skipping tensors
> [GIN] 2025/01/22 - 09:43:28 | 200 | 54.425610491s |       127.0.0.1 | POST     "/api/chat"

`

### OS

Linux

### GPU

Nvidia

### CPU

AMD

### Ollama version

 0.5.7