Draft Model: expected 76, got 69
❯ sha256sum /srv/models/llama-cache/laguna-s-2.1-DFlash-BF16.gguf
24614292a4477f3ae5203c3875edcde0bc219f02616a9c9f65791e29b18a67ee /srv/models/llama-cache/laguna-s-2.1-DFlash-BF16.gguf
0.28.687.288 I load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
0.28.707.025 E llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 76, got 69
0.28.707.030 E llama_model_load_from_file_impl: failed to load model
0.28.707.032 E common_speculative_init_result: failed to load draft model, '/srv/models/llama-cache/laguna-s-2.1-DFlash-BF16.gguf'
0.28.707.037 E srv load_model: failed to load draft model, '/srv/models/llama-cache/laguna-s-2.1-DFlash-BF16.gguf'
0.28.707.044 I srv operator(): operator(): cleaning up before exit...
0.28.708.173 E srv llama_server: exiting due to model loading error
I'm able to load it but I only get 10% draft acceptance at 0 token depth and at 40k token depth I get 0% acceptance. Are you using poolside's llama cpp fork? It won't work without it for now.
Update: It seems like a draft acceptance fix just got merged like 2 minutes before I commented. Trying it now
I'm getting very low draft acceptance rates and very few drafts as well: 2.09.228.447 I slot print_timing: id 3 | task 14 | draft acceptance = 0.40000 ( 2 accepted / 5 generated), mean len = 2.00
What settings are recommended for the dflash draft?
enlighten me.. what am i doing wrong?
llama-server \
--host 0.0.0.0 --port 8000 \
-hf poolside/Laguna-S-2.1-GGUF:Q4_K_M \
--jinja --flash-attn on --no-mmap --n-gpu-layers 99 --parallel 1 --ctx-size 262144 --cache-ram 0 --no-context-shift \
-lv 4 \
--reasoning on \
-hfd poolside/Laguna-S-2.1-GGUF:BF16 \
--spec-type draft-dflash --spec-draft-n-max 15
I get the same error message (expected 76, got 69) with a fresh build of llama.cpp including https://github.com/ggml-org/llama.cpp/pull/25165
> llama-cli --version
version: 10095 (e8e6c7af2)
built with AppleClang 17.0.0.17000604 for Darwin arm64
but only when I specify -hfd poolside/Laguna-S-2.1-GGUF:BF16, the model by itself seems to work fine
Ah, now I see its not yet added https://github.com/ggml-org/llama.cpp/pull/25165#issuecomment-5043878753
Thanks your comments. Please use our laguna branch, following instructions from https://huggingface.co/poolside/Laguna-S-2.1-GGUF#serving?
With Laguna llama branch
12.25.840.897 I slot print_timing: id 0 | task 2 | prompt eval time = 67838.88 ms / 40306 tokens ( 1.68 ms per token, 594.14 tokens per second)
12.25.840.900 I slot print_timing: id 0 | task 2 | eval time = 345062.86 ms / 9141 tokens ( 37.75 ms per token, 26.49 tokens per second)
12.25.840.900 I slot print_timing: id 0 | task 2 | total time = 412901.75 ms / 49447 tokens
12.25.840.901 I slot print_timing: id 0 | task 2 | graphs reused = 3103
12.25.840.904 I slot print_timing: id 0 | task 2 | draft acceptance = 0.27524 ( 6017 accepted / 21861 generated), mean len = 2.93
12.25.841.683 I slot release: id 0 | task 2 | stop processing: n_tokens = 49446, truncated = 0
12.25.841.697 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = 5120418944352
12.26.024.798 I slot launch_slot_: id 0 | task 25 | processing task, is_child = 0
12.28.129.437 I slot print_timing: id 0 | task 25 | prompt eval time = 137.22 ms / 5 tokens ( 27.44 ms per token, 36.44 tokens per second)
12.28.129.440 I slot print_timing: id 0 | task 25 | eval time = 1967.40 ms / 40 tokens ( 49.19 ms per token, 20.33 tokens per second)
12.28.129.440 I slot print_timing: id 0 | task 25 | total time = 2104.62 ms / 45 tokens
12.28.129.441 I slot print_timing: id 0 | task 25 | graphs reused = 3123
12.28.129.444 I slot print_timing: id 0 | task 25 | draft acceptance = 0.08861 ( 14 accepted / 158 generated), mean len = 1.58
Speculative decoding performed poorly here, with just 27.5% draft acceptance.
DFlash loading error with upstream llama.cpp
I can confirm that the main model loads and runs correctly with the latest upstream llama.cpp, but the DFlash draft model fails to load.
Environment
- macOS (Apple Silicon)
llama.cppofficial upstream- Version:
10101 (4310aa4f8) - Built with AppleClang 21.0.0.21000101
Main model
laguna-s-2.1-Q4_K_M.gguf
Loads successfully and inference works as expected.
Draft model
laguna-s-2.1-DFlash-BF16.gguf
Fails during loading with:
error loading model:
done_getting_tensors:
wrong number of tensors; expected 76, got 69
Command used:
llama-server \
--model laguna-s-2.1-Q4_K_M.gguf \
--model-draft laguna-s-2.1-DFlash-BF16.gguf \
--spec-type draft-dflash \
--spec-draft-n-max 15
The draft model fails before inference starts.
Is DFlash currently expected to work with upstream llama.cpp, or is the Poolside laguna branch still required for this model?
Thanks!