poolside

/

Laguna-S-2.1-FP8

like 11

Follow

Poolside 849

Big issues with looping esp at fp8 kv cache. Basically unusable and dflash broken as well. (sglang lastest, rtx pro 6000)
Getting without spec decoding sys prompt leakage at 80 to 60 tok s as it goes to 200k context

Regular looping here as well, generally after a long reasoning period. 8 RTX 3090 on vLLM nightly, llama-swap config as follows -

  "laguna-s-2.1":
    name: "Laguna S 2.1"
    env:
      - "OMP_NUM_THREADS=1"
      - "CUDA_DEVICE_ORDER=PCI_BUS_ID"
      - "CUDA_VISIBLE_DEVICES=0,3,6,7,2,4,5,1"
      - "HF_TOKEN=hf_token"
      - "HF_XET_HIGH_PERFORMANCE=1"
      - "CUDA_HOME=/opt/cuda"
      - "VLLM_MARLIN_USE_ATOMIC_ADD=1"
      - "VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1"
      - "VLLM_FLOAT32_MATMUL_PRECISION=high"
      - "VLLM_USE_NCCL_SYMM_MEM=0"
      - "SAFETENSORS_FAST_GPU=1"
      - "VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0"
      - "VLLM_ALLOW_LONG_MAX_MODEL_LEN=1"
      - "LD_LIBRARY_PATH=/home/hubby/.venv/lib/python3.12/site-packages/nvidia/cu13/lib/:LD_LIBRARY_PATH"
    capabilities:
      in:
        - text
      out:
        - text
      tools: true
      context: 524288
    cmd: |
      uv run vllm serve poolside/Laguna-S-2.1-FP8
       --host 0.0.0.0
       --port ${PORT}
       --seed 3407
       --disable-custom-all-reduce
       --async-scheduling
       --enable-prefix-caching
       --enable-chunked-prefill
      --served-model-name laguna-s-2.1
      --tensor-parallel-size 8
      --enable-expert-parallel
      --max-model-len 524288
      --gpu-memory-utilization 0.96
      --attention-backend flashinfer
      --gdn-prefill-backend flashinfer
      --enable-flashinfer-autotune
      --reasoning-parser poolside_v1
      --tool-call-parser poolside_v1
      --enable-auto-tool-choice
      --max-num-seqs 4
      --max-num-batched-tokens 16382
      --dtype bfloat16
      --generation-config auto
      --default-chat-template-kwargs '{"enable_thinking": true}'
      --override-generation-config '{"temperature": 0.7, "top_p": 0.95}'
      --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","fast_moe_cold_start":true}'

Hi all.

Thank you for your comments. We're really sorry that the model's been acting strangely.

We've pushed a new version to this repo that takes away these issues in our internal testing. Please let us know if it doesn't take away the issues for you. We have one more update planned (for better 1M context support), but this one seems to work well at 256K.

We'll also make similar changes for INT4 & NVFP4 later today (although NVFP4 will not yet live on main).

If I don't hear back in the next 24 hours or so, I'll close this issue :)

Thanks for the fast response, Joe! Re-downloading now.

Do the DFlash models also need updates?

@Aur0r You can use the same DFlash checkpoint you've been using but it may offer slightly lower acceptance rates with updated target's checkpoint. We'll be following with updated DFlash checkpoints shortly.

I'm afraid that I am still seeing looping. Interestingly, sometimes, while the reasoning does still tend to loop sections, it also intersperses it with new reasoning, and does generally manage to correct itself. Unfortunately, this comes with a pretty hefty token usage penalty, given the large amount of repetition. Other times, it stays in the loop, albeit less frequently than before.

Also, even when looping does not occur, almost every attempt now finishes mid sentence for me while reasoning in long context tasks. I tested 3 times with a code review task that successfully completed yesterday, but each time I tried today, the turn was ended mid sentence while reasoning, without outputting an answer, generally once around 76000 tokens are spent. This only seems to happen when the assistant turn is purely interleaved reasoning and tool calls. If the model outputs content, it doesn't seem to happen.

If I reply with a prompt to continue, the model continues, and finishes the task.

Tried with both the config I posted earlier, and with the max context reduced to 262144. Also tried with both sending a max tokens limit of 32768 with the request, and removing the limit altogether from the request, made no difference.

EDIT: this new issue was on my end. I updated vLLM, pruned my huggingface cache, and reloaded the updated model. No issues since then 😄