rockerBOO/moss-audio-nvfp4-convrot-int8

MOSS-Audio-8B-Thinking — NVFP4 + ConvRot INT8

A single-file quantization of OpenMOSS-Team/MOSS-Audio-8B-Thinking, targeting ComfyUI's native quantized-kernel loading path.

Files

File	Size	Notes
moss-audio-8b-thinking_nvfp4_convrot_int8.safetensors	8.8 GB	Derived from the original bf16 checkpoint (~17 GB).

Quantization method

Qwen3 decoder MLP weights (language_model.layers.*.mlp.{gate,up,down}_proj, 108 tensors) are quantized to NVFP4. Qwen3 decoder attention projections (language_model.layers.*.self_attn.{q,k,v,o}_proj, 144 tensors across 36 layers) are quantized to INT8 with ConvRot (group size 256). The audio encoder, audio adapter, deepstack merger, lm_head, embed_tokens, and norm are left at source precision (bf16) — excluded by role, following this project's PROCEDURE.md sensitivity heuristics.

Reproduce with:

ctq -i model.safetensors \
    -o moss-audio-8b-thinking_nvfp4_convrot_int8.safetensors \
    --nvfp4 --fallback fp8 \
    --layer-config moss_audio_8b_thinking_layer_config.json \
    --comfy_quant --save-quant-metadata

Verification

Tested end-to-end in ComfyUI (loader + generate, real audio input): produces coherent speech transcription and style/mood output. Not yet compared side-by-side against the original bf16 model's output quality.

Hardware requirements

Requires a Blackwell-class NVIDIA GPU (Compute Capability ≥ 10.0) for NVFP4 inference. ConvRot INT8 layers also require ComfyUI's ConvRot-aware quantized kernels.

License

Apache License 2.0, per the upstream OpenMOSS-Team/MOSS-Audio-8B-Thinking model card.