Inconsistent reasoning trigger on llama-server
Appreciate the wonderful release first and foremost!
I'm running the Q4_K_M through the shared custom fork's llama-server and I notice the model's reasoning trigger to be inconsistent. It triggers for select prompts like "Prove sqrt(2) is irrational", "What is 17*23? Think it through.", etc.., but completely skips it for the rest of my prompts (complex ones) even when I nudge it to think/reason before answering.
My launch command:
build/bin/llama-server -m ../Models/GGUFs/laguna-s-2.1-Q4_K_M.gguf -c 0 --fit on -fa on --port 1234 --jinja --no-mmap -b 8192 -ub 8192 --temp 0.7 --top-p 0.95 --chat-template-kwargs '{"enable_thinking": true}'
Try omitting your sampling parameters. The ones you have there a recommended for rope scaling at 1M token context. I haven't had any issue with reasoning triggering
--jinja
--flash-attn on
--parallel 1
--no-warmup
-m /models/Laguna-S-2.1-GGUF-poolside/laguna-s-2.1-Q4_K_M.gguf
--model-draft /models/Laguna-S-2.1-GGUF-poolside/laguna-s-2.1-DFlash-BF16.gguf
--spec-type draft-dflash
--spec-draft-n-max 15
--fit off
-c 262144
--n-gpu-layers 999
--main-gpu 0
--chat-template-kwargs '{"enable_thinking":true}'
Hi i have the same issue with llama.cpp vulkan backend. Simple question like "what is agile?" Creates a reasoning trace but a complex prompt with requirements and success criteria does not
@barnesea Removal of sampling params doesn't fix it for me either...The Q8 has the same issue as well. Surprised to hear it's working fine for you. Are you testing it directly via llama-server or via an agentic harness like OpenCode/Pi?
Edit: Facing the same issue with mainline llama.cpp post PR-merge
Hi guys! Thank you for your comments.
I'd like to ask how you're interacting with llama-server: it may be a harness issue. In any case, we've also noticed that our chat template didn't ship with preserved thinking properly enabled, which might be contributing to this. We've uploaded a new chat template to this repo, and we plan to repackage the GGUFs to contain the proper formatting later today. I'll update here once it's done.
I noticed same non-thinking replies in llama.cpp web ui, but in "normal" agent harness works ok, for ~1k messages now (10 user requests + 10 responses + 990 tool call/response), 0 transport-level/content-reasoning_content confusion/full-non-thinking problems. I suspect it may be that llama web ui sends empty system prompt, and template-default is omitted
I upgraded to the server-vulkan-b10088 image and manually forced the new chat-template-file. Still does not trigger. My front end is open webui now for testing. I also tried adding a system message, "you are a deep thinking assistant", did not help.
Hi guys! Thank you for your comments.
I'd like to ask how you're interacting with llama-server: it may be a harness issue. In any case, we've also noticed that our chat template didn't ship with preserved thinking properly enabled, which might be contributing to this. We've uploaded a new chat template to this repo, and we plan to repackage the GGUFs to contain the proper formatting later today. I'll update here once it's done.
Just tested running it via Pi agent and the model's reasoning surprisingly works there consistently. The reported issue is reproducible when interacting directly with llama-server's Web UI.