Is there a recommended vLLM recipe for RTX Pro 6000?
Share your NVFP4 Quantum startup parameters on RTX Pro 6000 via vLLM, please.
No matter how much I try, both with and without DFlash, the model eats up all the VRAM memory in the first 10-20 requests and falls out to 5-10 GB in RAM. It wasn't like that with other models.
My startup parameters are now (can you tell me what the error is):
vllm serve /mnt/e/Models/Laguna-S2.1-118B
--host 0.0.0.0
--port 8000
--served-model-name laguna-s2.1
--trust-remote-code
--enable-auto-tool-choice
--tool-call-parser poolside_v1
--reasoning-parser poolside_v1
--default-chat-template-kwargs '{"enable_thinking": false}'
--override-generation-config '{"temperature":0.7,"top_p":0.95}'
--max-num-seqs 2
--compilation-config '{"cudagraph_capture_sizes": [1, 2]}'
--max-model-len 262144
--kv-cache-dtype fp8
--gpu-memory-utilization 0.92
--enable-prefix-caching
--enable-chunked-prefill
--performance-mode throughput
try maybe vllm v0.25.1 as the latest version (0.26) changed cudagraph defaults to eat much more vram by default. At least this helped in my case.
I am running this: https://github.com/iphands/local-mind/blob/dev/server/vllmcustom/run-laguna
Laguna NVFP4 is looking badly at the moment but when I dont hit loops the output seems awesome.
I came from Qwen 3.5 122b and I think Laguna is better for OpenCode coding.
YMMV though :D
Your current recipe is perfect -- Just change 1 line : vllm serve /mnt/e/Models/Laguna-S2.1-118B with Qwen 3.5 122b.