poolside

/

Laguna-S-2.1-NVFP4

like 182

Follow

Poolside 1.42k

did you download poolside/Laguna-S-2.1-DFlash-NVFP4 again? I'm using dflash work fine with new dflash.

I'll try to redownload it

getting 1% acceptance rate average 5t/s thouput with dflash, not working well
I'm using official VLLM docker image as seen above, that might be the issue, the dflash isn't part of it

Here is a working version for a single user on DGX Spark. Note kv_cache_dtype: fp8 is essential for DFlash.

Using this base:
https://github.com/eugr/spark-vllm-docker

# Recipe: poolside/Laguna-S-2.1-NVFP4
# poolside Laguna-S-2.1 (117.6B total / 8.5B active MoE) in NVFP4 (W4A4) with DFlash
# draft speculation. Fits a single DGX Spark. Requires a recent vLLM (main) build
# with the poolside architecture + dflash speculative method.

# ID                                        REVISION                                   SIZE LAST_MODIFIED REFS
# ----------------------------------------- ---------------------------------------- ------ ------------- ----
# model/poolside/Laguna-S-2.1-DFlash-NVFP4  b3b5921a900b9e0a1e27e50bdaeb480692a6d19b   2.2G 1 day ago     main
# model/poolside/Laguna-S-2.1-NVFP4         f8fdfcdc4e7b0c474a0102430a8cae0a3a358669  99.7G 2 days ago    main

recipe_version: "1"
name: Laguna-S-2.1
description: vLLM serving poolside/Laguna-S-2.1-NVFP4 with DFlash speculation

# HuggingFace model to download (optional, for --download-model)
model: poolside/Laguna-S-2.1-NVFP4

solo_only: true

# Container image to use
container: vllm-node

# No mods required
mods: []

# Default settings (can be overridden via CLI)
defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 1
  gpu_memory_utilization: 0.90
  max_model_len: 262144
  max_num_batched_tokens: 2048
  max_num_seqs: 1
  num_speculative_tokens: 3
  kv_cache_dtype: fp8

# Environment variables
env:
  TORCH_CUDA_ARCH_LIST: 12.1f
  CUTE_DSL_ARCH: sm_121a
  VLLM_USE_FLASHINFER_SAMPLER: "1"
  MAX_JOBS: "4"

# The vLLM serve command template
command: |
  vllm serve poolside/Laguna-S-2.1-NVFP4 \
    --served-model-name poolside/Laguna-S-2.1 \
    --host {host} \
    --port {port} \
    --max-model-len {max_model_len} \
    --max-num-batched-tokens {max_num_batched_tokens} \
    --max-num-seqs {max_num_seqs} \
    --gpu-memory-utilization {gpu_memory_utilization} \
    --tensor-parallel-size {tensor_parallel} \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --enable-auto-tool-choice \
    --tool-call-parser poolside_v1 \
    --reasoning-parser poolside_v1 \
    --kv-cache-dtype {kv_cache_dtype} \
    --trust-remote-code \
    --speculative-config '{{"method": "dflash", "model": "poolside/Laguna-S-2.1-DFlash-NVFP4", "num_speculative_tokens": {num_speculative_tokens}}}' \
    --default-chat-template-kwargs '{{"enable_thinking": true}}' \
    --override-generation-config '{{"temperature": 0.7, "top_p": 0.95}}'

Hello @TeAmErX and thank you for your post and feedback. 🙏 Can you please try to clean up the mounted HF cache and pin the models' revisions explicitly? Also please add "method": "dflash" into your speculative-config.

You will get a very low acceptance rate for dflash if you don't set kv-cache-dtype=fp8. I suspect an implementation defect in VLLM dealing with mixed precision.

Hello @TeAmErX and thank you for your post and feedback. 🙏 Can you please try to clean up the mounted HF cache and pin the models' revisions explicitly? Also please add "method": "dflash" into your speculative-config.

remove hf cache completely and redownloaded everything
with speculative decoding there is never enough memory even at 64k context size and 1 max-num-seqs

You will get a very low acceptance rate for dflash if you don't set kv-cache-dtype=fp8. I suspect an implementation defect in VLLM dealing with mixed precision.
thanks for the tip, tried it, not enough memory with speculative decoding enabled

Strange. It definitely works on a Linux box but barely:

free -h
               total        used        free      shared  buff/cache   available
Mem:           121Gi       118Gi       3.0Gi       104Mi       1.0Gi       2.9Gi
Swap:           15Gi       2.5Gi        13Gi

are you using the new weights?, those numbers are what the old weights used to be, for me its taking the whole 130.6 GB ram and more than 4gb of swap

I added the hash above:

# ID                                        REVISION                                   SIZE LAST_MODIFIED REFS
# ----------------------------------------- ---------------------------------------- ------ ------------- ----
# model/poolside/Laguna-S-2.1-DFlash-NVFP4  b3b5921a900b9e0a1e27e50bdaeb480692a6d19b   2.2G 1 day ago     main
# model/poolside/Laguna-S-2.1-NVFP4         f8fdfcdc4e7b0c474a0102430a8cae0a3a358669  99.7G 2 days ago    main

which vllm version are you using?

Im using this build https://github.com/eugr/spark-vllm-docker on a Dell GB10 Spark

TeAmErX maybe try RC3? seems to work a lot better and the weights are back to their original size

RC3 Works, also verified that the DFLASH file is the latest b3b5921a900b9e0a1e27e50bdaeb480692a6d19b
getting on average 13t/s (10 to 16) , quality looks worse on large jobs (I did a large refactor on RC2 stored to branch, reverted and did the same refactor using the same prompt using RC3, 40% of the requirements were ignored as will do later, RC3 is lazy 😃 )

What number of tokens for speculative decoding should i use?
i tried 15 as per documentation, averaged 7t/s, but saw spikes to 50tps (!!!) during long responses
currently at speculative_tokens 3 and seeing the 13tps average!
what are the recommend temperature and top_p ? how would they affect output quality?

Hi all, glad to hear that DFlash works! For the time being, we recommend using the main checkpoints for the best quality. We appreciate your patience, and please rest assured that we are working on reducing the memory footprint to fit DGX Spark users' hardware comfortably while maintaining quality and avoiding looping issues :). DFlash-wise, we recommend using num_speculative_tokens=7 for the best speedup results on agentic/coding tasks.