poolside

/

Laguna-S-2.1-GGUF

like 141

Follow

Poolside 1.1k

--top-k 0 sounds a bit odd, but will give it a try too now

Yeah basically just disabling top-k. I'm also not using Dflash at the moment so your mileage may vary if you use that.

I tested it with Myric/Laguna-S-2.1-APEX-GGUF (quality) and it was a little worse there, but that is a higher quant. Also tested with dealignai/Laguna-S-2.1-CRACK-GGUF (Q4_K_M) an it works well, even slightly better based on quick vibes. This makes me think this model is really sensitive to quantization and we'd probably have to find optimal sampler settings for the other quants.

I cannot tell if I see any difference with topk on 0

Ran a controlled measurement on these settings — I've been chasing the same intermittent "reasons a long time, then stops without emitting the tool call" behavior, so I put numbers on it. Sharing in case it's useful.

Setup: llama.cpp (poolside laguna fork), laguna-s-2.1-Q4_K_M.gguf @ 256k, -fa on, q8_0 KV. One identical single-turn edit request (expects a tool call), replayed N=12 per setting. Metric: emitted ≥1 tool call ("commit") vs stopped after reasoning with none.

Observations:

• Your recipe vs default is flat (Fisher exact ≈ 1.0) at N=12 — no measurable harm, but I couldn't confirm a commit-rate gain either.
• temp 0.3 was worse — cooling seems to lock in the modal trajectory, which here is the early stop.
• Across all 36 draws, every commit came from a long reasoning trace (14–61k chars) that then acted; failures mostly quit under 30k. Committing seems to need diverging from the most-likely short-stop path — argues for more entropy, not less.
• The repetition penalty tripled finish=length (loop-till-budget) — feeds the long-loop mode rather than breaking it.

Net, on this quant, sampler settings didn't move the commit rate off ~25% — the stop-after-reasoning looks weights/decoding-level rather than tuning-fixable. Curious whether others see the same. Thanks for kicking off the settings thread!

I have been testing every single update since this model's release but have had no success. I think I will now wait for the updated weights by poolside team before I touch this model again. If they are able to solve it then this is perfect for the M5 Max 128gb.

I tried many setting till it ran with somewhat usable speeds on my Strix Halo AI Max 395. Below are the flags i am using at the moment, getting about 15 t/s avg while decoding.
KV type f16 was faster then q8_0. For spec draft, n-max 3 was my sweet spot with ~45% acc rate avg, with n-max 2 being at ~30% and n-max 4+ also starting at 30% and going down as the max goes up.
Last, the context size would work for me with 512k, but leaving only 10gb ram free, so atm i use 384k.

-m ../laguna-s-2.1-Q4_K_M.gguf --alias Laguna-S-2.1-Q4 --chat-template-file ../chat_template.jinja -md ../laguna-s-2.1-DFlash-BF16.gguf --spec-type draft-dflash --spec-draft-n-min 1 --spec-draft-n-max 3 -c 393216 -fa on -ngl 999 --no-mmap --cache-type-k f16 --cache-type-v f16 --cache-ram 16384 --jinja --reasoning on --no-warmup -b 2048 -ub 2048 --ctx-checkpoints 2 --kv-unified -np 1 -t 16 --host 0.0.0.0 --metrics --port 8081 --sleep-idle-seconds 900 --temp 1.0 --top-p 1.0 --top-k 20 --min-p 0.00 --presence-penalty 0.0 --repeat-penalty 1.0

I think people seeing reasoning go off the rails are messing with batch size. In my experience, leaving that at 512 is the ticket to keeping it from reasoning itself off the rails, once I kept -b at 512 and -ub at 2048 I stopped having to do all the shenanigans with top-p/top-k and penalties.

Ran a controlled measurement on these settings — I've been chasing the same intermittent "reasons a long time, then stops without emitting the tool call" behavior, so I put numbers on it. Sharing in case it's useful.

Setup: llama.cpp (poolside laguna fork), laguna-s-2.1-Q4_K_M.gguf @ 256k, -fa on, q8_0 KV. One identical single-turn edit request (expects a tool call), replayed N=12 per setting. Metric: emitted ≥1 tool call ("commit") vs stopped after reasoning with none.

Setting	Commit	fail split (stop / length)
temp 0.7, top_p 0.95 (default)	3/12 (25%)	8 / 1
your recipe (temp 0.85, top-k 0, min-p 0, repeat-last-n 64 @ 1.1)	4/12 (33%)	5 / 3
temp 0.3	1/12 (8%)	10 / 1

Observations:

• Your recipe vs default is flat (Fisher exact ≈ 1.0) at N=12 — no measurable harm, but I couldn't confirm a commit-rate gain either.
• temp 0.3 was worse — cooling seems to lock in the modal trajectory, which here is the early stop.
• Across all 36 draws, every commit came from a long reasoning trace (14–61k chars) that then acted; failures mostly quit under 30k. Committing seems to need diverging from the most-likely short-stop path — argues for more entropy, not less.
• The repetition penalty tripled finish=length (loop-till-budget) — feeds the long-loop mode rather than breaking it.

Net, on this quant, sampler settings didn't move the commit rate off ~25% — the stop-after-reasoning looks weights/decoding-level rather than tuning-fixable. Curious whether others see the same. Thanks for kicking off the settings thread!

I have been testing every single update since this model's release but have had no success. I think I will now wait for the updated weights by poolside team before I touch this model again. If they are able to solve it then this is perfect for the M5 Max 128gb.

Thanks for all the effort, just a clarification - repeat-last-n defaults to 64 in llama.cpp but is only active if repeat penalty is set above 1 (which is deactivated). The GUI I use for llama.cpp still outputs the repeat last n regardless of whether repeat penalty is active or not. It has no effect if it isn't active.