为什么全是！！！！！！！！！
设备是dgx-spark，是我参数的问题嘛？
services:
qwen3.6-27b-nvfp4:
image: vllm/vllm-openai:v0.24.0-cu129-ubuntu2404
#image: vllm/vllm-openai:nightly
container_name: qwen3.6-27b-nvfp4
restart: unless-stopped
ipc: host
shm_size: '16gb' # 维持高性能多模态所需的共享内存
ports:
- "8000:8000"
environment:
- TZ=Asia/Shanghai
- VLLM_USE_FLASHINFER_SAMPLER=1
- TORCH_MATMUL_PRECISION=high
volumes:
#- ./huggingface:/root/.cache/huggingface
- ./Qwen3.6-27B-NVFP4:/Qwen3.6-27B-NVFP4
- ./vllm_cache/vllm:/root/.cache/vllm
- ./vllm_cache/triton:/root/.cache/triton
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
command: >
--model /Qwen3.6-27B-NVFP4
--served-model-name qwen3.6-27b-nvfp4
--host 0.0.0.0
--port 8000
--tensor-parallel-size 1
--trust-remote-code
--quantization modelopt
--kv-cache-dtype fp8
--moe-backend marlin
--attention-backend flashinfer
--gpu-memory-utilization 0.3
--max-model-len 262144
--max-num-seqs 2
--max-num-batched-tokens 24576
--enable-chunked-prefill
--async-scheduling
--enable-prefix-caching
--speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}'
--load-format fastsafetensors
--reasoning-parser qwen3
--tool-call-parser qwen3_xml
--enable-auto-tool-choice
--override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"max_tokens":8192}'
没事了，我从vllm/vllm-openai:nightly换成vllm/vllm-openai:v0.24.0-cu129-ubuntu2404就不会出现这种问题了
same issue on vllm/vllm-openai:v0.24.0
我也是一直用vllm/vllm-openai:nightly,刚好昨天也遇到这个一直!!!!!!!!的问题, 我还以为是我下的模型有问题, 原来是这个nightly的锅...
I encountered the same issue on both vllm==0.24.0 and vllm-nightly.
Additionally, on 0.24.0, the flashinfer version was 0.6.12, which caused an anomaly in AutoTuner, where messages like [AutoTuner]: Tuning fp8_gemm: 100% appeared over three hundred times. In the nightly build, flashinfer was upgraded to 0.6.13, which fixed this issue, but it did not resolve the problem of the model outputting "!!!" during CoT.
我在 vllm==0.24.0 和 vllm-nightly 上都遇到了同种问题。
另外在 0.24.0 时 flashinfer 版本是 0.6.12, 这导致 AutoTuner 出现异常，[AutoTuner]: Tuning fp8_gemm: 100% 这种字样会出现超过三百次。在 nightly 中 flashinfer 被升级到 0.6.13, 修复了这个问题，但是没有解决模型在 CoT 中输出 "!!!" 的问题。
The problem should be solved now in the latest vllm/vllm-openai:v0.25.1.