[need help] vllm 0.24.0 on dgx spark warn Your GPU does not have native support for FP4
vllm serve ~/models/hf/Qwen3.6-27B-NVFP4 --trust-remote-code --served-model-name qwen36-27b --gpu-memory-utilization 0.45 --dtype bfloat16 --max-num-seqs 4 --max-model-len 131072 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --speculative-config '{"method":"mtp","num_speculative_tokens":3}' --max-num-batched-tokens 16384 --enable-chunked-prefill --async-scheduling --enable-prefix-caching --quantization modelopt
(APIServer pid=2593872) INFO 07-03 12:22:48 [api_utils.py:339]
(APIServer pid=2593872) INFO 07-03 12:22:48 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2593872) INFO 07-03 12:22:48 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.0
(APIServer pid=2593872) INFO 07-03 12:22:48 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/shenshen/models/hf/Qwen3.6-27B-NVFP4
(APIServer pid=2593872) INFO 07-03 12:22:48 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2593872) INFO 07-03 12:22:48 [api_utils.py:339]
(APIServer pid=2593872) INFO 07-03 12:22:48 [api_utils.py:273] non-default args: {'model_tag': '/home/shenshen/models/hf/Qwen3.6-27B-NVFP4', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'model': '/home/shenshen/models/hf/Qwen3.6-27B-NVFP4', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_model_len': 131072, 'quantization': 'modelopt', 'served_model_name': ['qwen36-27b'], 'reasoning_parser': 'qwen3', 'gpu_memory_utilization': 0.45, 'enable_prefix_caching': True, 'max_num_batched_tokens': 16384, 'max_num_seqs': 4, 'enable_chunked_prefill': True, 'async_scheduling': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 3}}
(EngineCore pid=2593964) WARNING 07-03 12:25:11 [marlin.py:34] Your GPU does not have native support for FP4 computation but FP4 quantization is being used. Weight-only FP4 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads.
(EngineCore pid=2593964) WARNING 07-03 12:25:12 [kv_cache.py:134] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for FP8 Attention backends (flash-attn or flashinfer).
(EngineCore pid=2593964) WARNING 07-03 12:25:12 [kv_cache.py:148] Using KV cache scaling factor 1.0 for fp8_e4m3. If this is unintended, verify that k/v_scale scaling factors are properly set in the checkpoint.
I think the warning statically coded in the Marlin kernel: https://github.com/vllm-project/vllm/issues/27471
FP4 is supported and (possibly) also used.
I think it seems due to quantization=modelopt_mixed, I checked that config.json, and it explains exactly why vllmprints FP8.The model is not pure NVFP4 everywhere. It is a mixed ModelOpt quantization checkpoint.
Anyone have a conclusion on this?