unsloth

/

Qwen3.8-27B-NVFP4

like 327

Follow

Unsloth AI 32.3k

Nice!

gpu-memory-utilization can be lowered to 0,42. Rules of thumb I go by are:

1. Look at the storage size it requires on https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4/tree/main (it says 23,4GB)
2. From 23,4, add 16 (for 261k tokens, approximately) = 39,4
3. Set the context memory rounded to that number = 0,39

I haven't tested lower than 0,42 though.

gpu-memory-utilization can be lowered to 0,42. Rules of thumb I go by are:

1. Look at the storage size it requires on https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4/tree/main (it says 23,4GB)
2. From 23,4, add 16 (for 261k tokens, approximately) = 39,4
3. Set the context memory rounded to that number = 0,39

I haven't tested lower than 0,42 though.

I tried 0.40 and it seems to be working on my DGX Spark.

I'm very far from an expert on this but it seems to be working fine for my hermes agents' api calls.

From my docker config:
vllm-qwen38:
<<: *vllm-common
profiles: ["qwen38"]
container_name: vllm-qwen38
ports:
- "8000:8000"
command:
- ${QWEN38_MODEL:-unsloth/Qwen3.8-27B-NVFP4}
- "--host"
- "0.0.0.0"
- "--port"
- "8000"
- "--served-model-name"
- "${QWEN38_SERVED_NAME:-spark-coder}"
- "--tensor-parallel-size"
- "${QWEN38_PAR_SIZE:-1}"
- "--max-model-len"
- "${QWEN38_MAX_MODEL_LEN:-262144}"
- "--gpu-memory-utilization"
- "${GPU_MEM_UTIL:-0.40}"
- "--max-num-seqs"
- "${MAX_NUM_SEQS:-4}"
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "${QWEN38_TOOL_PARSER:-qwen3_coder}"
- "--kv-cache-dtype"
- "fp8"
- "--reasoning-parser"
- "qwen3"
- "--language-model-only"
- "--speculative-config"
- '{"method":"mtp","num_speculative_tokens":2}'
- "--api-key"
- "${VLLM_API_KEY:-spark-local-change-me}"
- "--trust-remote-code"

Here's mine:

services:
  vllm:
    container_name: vllm
    image: vllm/vllm-openai:qwen38
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    ports:
      - "8000:8000"
    environment:
      - HF_TOKEN=${HF_TOKEN}
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    configs:
      - source: vllm_runtime_config
        target: /etc/vllm/config.yaml
    command: --config /etc/vllm/config.yaml

configs:
  vllm_runtime_config:
    content: |
      model: "unsloth/Qwen3.8-27B-NVFP4"
      trust_remote_code: true
      gpu_memory_utilization: 0.8
      enable_per_request_metrics: true
      enable_return_routed_experts: true
      max_model_len: 262144
      reasoning_parser: "qwen3"
      # needed?
      default_chat_template_kwargs: '{"preserve_thinking": true}'
      enable_auto_tool_choice: true
      tool_call_parser: "qwen3_coder"
      # for effectively serving concurrent requests
      enable_prefix_caching: true
      # JSON string converted to native YAML map
      speculative_config:
        method: "mtp"
        num_speculative_tokens: 2

Single Spark. Single request is 15-23 tokens/s. I've hand-wavey measured 5 concurrent requests before tokens/s/request begin to consistently drop below 15.

I have a warning log message in my vLLM execution on DGX Spark:

Model unsloth/Qwen3.8-27B-NVFP4 is treated as multimodal but has no registered multimodal processor; running in text-only mode.

Does anyone know how to fix this? At the moment my model cannot read image inputs.

I have a warning log message in my vLLM execution on DGX Spark:

Model unsloth/Qwen3.8-27B-NVFP4 is treated as multimodal but has no registered multimodal processor; running in text-only mode.

Does anyone know how to fix this? At the moment my model cannot read image inputs.

I didn't even know this, but honestly, this model is just broken in so many ways, it's best to wait until they release post-training/adjustments/etc. Just stick with 3.6 for now.