pavanperi/Voxtral-Mini-4B-Realtime-2602_W4A16_G128

Voxtral-Mini-4B-Realtime-2602 — GPTQ W4A16 Compressed

Compressed checkpoint of mistralai/Voxtral-Mini-4B-Realtime-2602 produced with llm-compressor using GPTQ W4A16 quantization on the decoder's linear layers.

---

1. Environment Setup (uv recommended)

Create the project

mkdir voxtral-serve && cd voxtral-serve

copy pyproject.toml in the directory

[project]
name = "voxtral-serve"
version = "0.1.0"
description = "Compress Voxtral-Mini-4B-Realtime-2602 using llmcompressor"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "vllm",
    "mistral-common>=1.5.4",
    "soundfile>=0.12",
    "librosa>=0.10",
    "soxr>=0.3",
]

Install everything in an isolated venv:

uv venv .venv --python 3.12
source .venv/bin/activate
uv sync

serve the compressed model

uv run vllm serve --config vllm_config.yaml

---

2. Compression technique

GPTQ W4A16

The script applies GPTQ inthe W4A16 configuration:

GPTQ isa one-shot, data-driven post-training quantization algorithm. For each Linear layer it:

1. Collects the Hessian of the layer's reconstruction error using a small calibration dataset (256 audio samples here).
2. Quantizes weight columns sequentially, ans it compensates the remaining unquantized columns using the Hessian inverse.
3. Stores the resulting tensors.

• W4 — weights are quantized to 4-bit integers (INT4), stored in a blocked/grouped layout.
• A16 — activations remain in bfloat16 at inference time; only the matrix-multiply dequantizes weights on the fly.
• Group size 128 — the 4-bit scale and zero-point factors are computed per group of 128 consecutive weight columns, which id a good balance between accuracy and model size.

What is quantized and what is not

Only the decoder Linear layers are compressed. We preserved all the audio components which are very sensitive to precision loss and quantizing those would be a minimal size gain.

---

3. Capabilities of the Compressed Model

Metric	Impact
Model size on disk	~2× smaller (bfloat16 → 4-bit weights for decoder linears)
GPU VRAM at inference	Reduced by roughly the same ratio
Audio encoder fidelity	Unchanged — encoder is full precision