mrysh/qwen3.5-4b-nvfp4

qwen3.5-4b - nvfp4

Quantized version of Qwen/Qwen3.5-4B made for a quantization benchmark (4 models x 8 techniques, evaluated with vLLM on BFCL, Energy-QA, hiring-bias and MedQA).

• method: nvfp4, about 4 bits, group size 16
• library: nvidia-modelopt 0.47.0
• calibration samples: 512
• size on disk: 3.307 GB

NVFP4 (NVIDIA 4-bit floating point, group size 16, made with NVIDIA ModelOpt, calibration set of 512 mixed samples). Loads in vLLM (vllm serve <repo>). The speed gain needs a Blackwell GPU (e.g. RTX PRO 6000, B200). The vision/audio parts are kept in 16 bit.

Evaluated as a text model (vLLM, thinking off). Same license as the base model; see the base model card.