giangndm/muse-glimmer-30b-nvfp4-tiny

Muse-Glimmer-30B NVFP4 Tiny

Summary

English/Vietnamese text-only version, freshly quantized from the original BF16 model using ModelOpt AutoQuantize. NVIDIA's original NVFP4 release is the quantized reference.

Weights use a 4.5-bit effective-bit budget, NVFP4, and the original 202,048-token vocabulary and tokenizer. GPU weights: 13.890 GiB; CPU embedding: 2.505 GiB (BF16, copied from the original model). The lm_head rows are NVFP4 with the same recipe and weight_scale_2 as the rest of the head. Audio/vision are omitted; long-context audio-agent capacity is untested.

Validation on one RTX 5070 Ti with ninfer-rs (--slots 1 --context 4096, greedy, low reasoning):

Check	Result
Vietnamese diagnostic answers (exact)	25/25
Tool calls: one and three get_device_status calls, JSON and SSE, tool results, next turn	pass
Tool calls with the DFlash assistant on 16 GB	out of memory (needs a larger GPU)

Run with vLLM

Use vLLM 0.30.0 on Blackwell. Apply the included embedding-offload fix once; stock Muse does not register the embedding with its CPU offloader. The fix backs up the installed source and uses CPU-pinned UVA; decoder/head stay on GPU.

hf download giangndm/muse-glimmer-30b-nvfp4-tiny \
  install_vllm_embedding_offload.py \
  --local-dir /tmp/muse-setup
python /tmp/muse-setup/install_vllm_embedding_offload.py

CUDA_VISIBLE_DEVICES=0 \
VLLM_FLASHINFER_WORKSPACE_BUFFER_SIZE=67108864 \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
vllm serve giangndm/muse-glimmer-30b-nvfp4-tiny \
  --dtype bfloat16 --tensor-parallel-size 1 --enforce-eager \
  --max-model-len 2048 --max-num-seqs 2 --max-num-batched-tokens 128 \
  --gpu-memory-utilization 0.97 \
  --cpu-offload-gb 2.6 --cpu-offload-params embed_tokens.weight --offload-backend uva \
  --kv-cache-dtype fp8 --kv-cache-memory-bytes 268435456 \
  --attention-backend FLASHINFER \
  --kernel-config '{"linear_backend":"cutlass","enable_flashinfer_autotune":false}' \
  --enable-auto-tool-choice --tool-call-parser muse_glimmer --reasoning-parser muse_glimmer

Send chat_template_kwargs={"reasoning_strength":"low"} with chat requests. The tested setup uses FP8 KV (256 MiB) and FlashInfer workspace (64 MiB). Long-context concurrency and audio capacity are untested. This vLLM recipe has not been re-validated with the 202,048-token vocabulary (0.41 GiB more GPU memory, 2.6 GiB embedding offload).