qwen3.5-9b - prune_width_ffn50
Pruned version of Qwen/Qwen3.5-9B made for a pruning benchmark (4 models x 8 pruning variants). The benchmark compares the variants with each other and with quantization on BFCL, hiring-bias and MedQA, run with vLLM.
- method: width, level 0.5
- library: nvidia-modelopt 0.46.0rc1
- size on disk: 13.096 GB
- reduction of the language model: 27.0% of its parameters (of the whole checkpoint: 27.0%)
Width pruning with NVIDIA ModelOpt Minitron: the FFN size of every layer 12288 -> 6144 (50% requested), the number of layers is unchanged. Importance of the neurons was measured on a calibration set of 512 mixed samples.
Only the language part is pruned. The vision tower and the multi-token-prediction head of Qwen3.5 are removed, so this is a text-only model (like the quantized Qwen checkpoints of this benchmark).
No distillation or fine-tuning was done after pruning, so expect a quality drop against the base model; the benchmark measures that drop. In the benchmark it is evaluated as a text model (vLLM, thinking off). Same license as the base model; see the base model card.
- Downloads last month
- 25