poolside

/

Laguna-S-2.1-NVFP4

like 128

Follow

Poolside 1.04k

this might be because of problems with flashInfer. I've been working to get the model working in sglang. this PR has some relevant detail: https://github.com/sgl-project/sglang/pull/32119 (I actually came here to ask poolside what the original strategy was that they were releasing towards, which apparently none of the major runtimes have picked up yet.)

BTW, I'm still in the process of figuring out how to pin down and explicitly reproduce the behavior, so I can test for ways to resolve it. So, this might not really be relevant. It just looks like it superficially.

just ask it to write a 5000 token story about some topic. it loops on original and on this one. i generally use that as a speed benchmark. vllm 25.1 single rtx pro 6k blackwell.

True but I mean, figuring out a specific reproduction.

Just doing fairly random samples like this, FWIW, my early results agree with yours:

• New checkpoint still requires my PR; without it, output is corrupt. (totally expected)
• It does not fix the existing arithmetic verification loop: both old and new checkpoints loop on 3/15 tests.

I had looping with XS2.1, but that was because I used the temperature from my Qwen setup. With a temperature of 1.0 it works fine, might also be the case here? I see most people using 0.7.

This is on the model card:

Recommended sampling

For the best balance of quality and reliability we recommend sampling with temperature 0.7 and top_p 0.95.

There are a lot of loops with new weights, unfortunately. Temperature 0.7 and top_p 0.95, repetition_penalty 1.05. mtp - 7 tokens

hey @Gedeliuk , thanks for raising this, could you please share verbatim prompts that you used and observed loops, if you happen to have them?

I so far haven't had loops, temp 0.7, top_p 0.95, top_k=20 (tested 64 to see how it affects it, also no loops but output got a bit weird), repetition_penalty 1.15.

According to my vllm dashboard, till now I've generated 227.3K tokens, 'processed' 63.5M (though i think this is counting cached KV all over again as it seems too big otherwise).
I've set dflash to draft 5, 1.31 tok/draft avg (close to 4 during code, close to 1 during thinking), Accepted 150K tok, Draft 574K tok. Running on a DGX Spark with gpu util at 0.88 reaching ~900k total max ctx it can hold across requests at the same time at any given point (fp8 kv). Longest single session compacted at 400k (max-tokens i've set in vllm/config) and worked for 64k more tokens before finishing up.

I tested it with Oh My Pi. The last time it stuck on implementation from detailed implementation.md .
Reasoning:
############
For the backend, the key implementation areas are:

1. Auth service (Google OAuth + session management)
2. Employee model and repository
3. Metrics calculator
4. Handler implementations
5. Swagger docs

For the frontend, the key implementation areas are:

1. Login page
2. Dashboard page
3. Metric cards
4. Charts
5. Employee management page

Let me write the task descriptions now.

Actually, I want to make sure the backend implementation is correct. Let me think about the metrics calculation one more time.
############
And after that, he started from the beginning. Also, it often got stuck after the tool call if there was an error in the tool response. I had 50-60k context at that time.

For me it also starts to loop with the new weights, no matter the harness. Also it loops when invoking it with Strix for some security scans.

Still looping on RC1. I get this weird effect where it seems to loop when trying to generate a think token, but then I also have this other strange effect where it just gets stuck to coding over and over and over at an extremely high TPS.

Hi @cbert33 , @Felixkruemel , @darkmatter2222 - thanks for flagging, would you mind sharing exact prompts that triggered looping/ overthinking? this would be very helpful!

@eliz-yoshida Here's a prompt that triggers it reliably enough to be useful.

Setup: DGX Spark (GB10), native install per the NVFP4 model card — vLLM 0.25.1, flashinfer 0.6.15.dev20260712, CUTE_DSL_ARCH=sm_121a. Checkpoint 07614121b (current main). DFlash draft 15, bf16 KV, 256K ctx, --gpu-memory-utilization 0.85. No agent harness — direct /v1/chat/completions, streaming, max_tokens: 8000, temp 0.7 / top_p 0.95 / top_k 20, thinking enabled.

Prompt:

This cache is meant to be thread-safe and evict least-recently-used entries. It passes single-threaded tests and passes a load test with 8 threads doing random gets. In production under heavy concurrent writes it occasionally returns a value for a key that was never inserted.

Find the bug, explain the exact interleaving that causes it, and give the minimal fix.

class LRUCache:
    def __init__(self, capacity):
        self.capacity = capacity
        self.data = {}
        self.order = []
        self.lock = threading.Lock()

    def get(self, key):
        with self.lock:
            if key not in self.data:
                return None
            self.order.remove(key)
            self.order.append(key)
            return self.data[key]

    def put(self, key, value):
        with self.lock:
            if key in self.data:
                self.order.remove(key)
            elif len(self.data) >= self.capacity:
                oldest = self.order.pop(0)
                del self.data[oldest]
            self.order.append(key)
        self.data[key] = value

(The bug: self.data[key] = value is outside the lock.)

Results, 3 seeds, everything else identical:

2 of 3 spend the entire 8k budget on reasoning and never emit an answer.

The interesting part is seed 22: it produced zero reasoning and answered correctly and immediately. The two failures both generated ~32k chars of reasoning. So on this prompt the failure doesn't look like "reasoning is slow" — it looks like once it starts reasoning it doesn't terminate, and the runs that succeed are the ones where it skips reasoning entirely.

I also tried repetition_penalty 1.15 (per @sleepyeldrazi above): 2/3 answered vs 1/3 without. Suggestive but n=3, so I can't claim it as a fix.

Happy to run more seeds or variations on this — it's a stable repro here.

Ran more seeds as offered — the repro is stronger than my first post suggested, and repetition_penalty does measurably help.

Same prompt, same setup, 8 seeds per arm, max_tokens: 8000, temp 0.7 / top_p 0.95 / top_k 20, thinking on, direct to vLLM:

Without a repetition penalty it never completes — all 8 hit finish_reason: length with ~30,000 chars of reasoning_content and 0 chars of answer.

With 1.15, half complete and every completion gets the bug right.

The split is sharp: successful runs produce 0 / 12,882 / 13,504 / 26,223 chars of reasoning, failures all sit at ~33,000 (the 8k-token ceiling). It doesn't look like slow reasoning — it looks like reasoning that never reaches a stopping point, and the runs that succeed are the ones that happened to stay short.

Caveat: fixed seeds aren't reproducible here (speculative decoding + batching), so these are 8 independent samples rather than 8 repeatable cases. repetition_penalty mitigates, it doesn't fix.

@d4rkdpg64 have you tried testing with bigger budgets? I have seen it output over 32k of reasoning on some questions, this model thinks a lot by default, and is mentioned as a 'limitation' of the model. As evident by your seed 22, I have also noticed that it doesn't necessarily need to think and can figure it out without thinking sometimes, but once it enters its "i need to re-verify multiple times to be sure" routine, it can spend a lot of tokens, for better or worse.

Regarding repetition_penalty, I did try lowering it to 1.05 and i hit my first loop with the model, although I was at ~320k ctx, which is above the recommended limit for the nvfp4 quant, so I'm not sure if i can fully blame the model/quant for that one. I have had it close to 400k multiple times now, so it can work at that length, but it may be a little too risky. I did notice that the dflash avg accepted length jumped to ~2.4 with the lower repetition penalty, which I also found interesting.

Also, I ran your prompt using curl with my config below (the one i've been using, but with penalty 1.05), and out of 5 attempts, the first one completed it in ~7k tokens, 2nd one in ~9.5k tokens, two at ~11k tokens and the final one looped (marking the 2nd time i saw actual looping). All correctly found the solution. I'll run it again with 128k for 5 runs with 1.15, but I worry that the repetition penalty might hurt its real-world performance, so treat it as a band-aid until we have other tips from the devs.

My vllm launch script:

#!/bin/bash
export CUTE_DSL_ARCH=sm_121a
export PATH=/usr/local/cuda/bin:$PATH
export MAX_JOBS=4
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
source ~/unsloth-nvfp4-env/bin/activate

vllm serve /home/sleepy/models/laguna \
  --speculative-config '{"model":"/home/sleepy/models/laguna-dflash","num_speculative_tokens":7,"method":"dflash"}' \
  --enable-auto-tool-choice \
  --tool-call-parser poolside_v1 \
  --reasoning-parser poolside_v1 \
  --served-model-name vLLM \
  --override-generation-config '{"temperature":0.7,"top_p":0.95,"top_k":20,"repetition_penalty":1.05}' \
  --max-num-seqs 32 \
  --max-model-len 800000 \
  --gpu-memory-utilization 0.88 \
  --host 0.0.0.0 --port 17777

Environment:

Hardware	NVIDIA GB10 (DGX Spark), 128 GB unified memory
Driver	610.43.02
vLLM	0.25.1
FlashInfer	0.6.13
torch	2.11.0+cu130 (CUDA 13.0)
Python	3.12 (venv ~/unsloth-nvfp4-env)
Model	poolside/Laguna-S-2.1-NVFP4 
Drafter	poolside/Laguna-S-2.1-DFlash  (DFlash, num_speculative_tokens=7)

It shouldn't be relevant, but I am using a newer nvidia driver than what the dgx spark ships with.

Note: I did test the nvfp4-dflash too, but the default 256k ctx caused issues before i altered it, and the acceptance felt a little lower, but i haven't done proper A/B to confirm (it was 1.78 on a task that the other drafter had 2.2, but it wasn't at temp 0, so not a clean comparison).

At repetition_penalty=1.15 :
Run 1: 8744 tokens
Run 2: 3122 tokens
Run 3: 7791 tokens
Run 4: 15899 tokens
Run 5: 8285 tokens
All found the bug and avg acceptance len for dflash was 1.01
At 1.1:
Run 1: 13621 tokens
Run 2: 12000 tokens
Run 3: 3548 tokens
Run 4: 10062 tokens
Run 5: 12855 tokens
Run 6: 7326 tokens
Run 7: 13414 tokens
Run 8: 16076 tokens
Run 9: 3355 tokens
Run 10: 7491 tokens
All found the bug and avg acceptance len for dflash was 1.27