Qwen2.5-32B-Instruct — AWQ 4-bit
Auto-quantized from Qwen/Qwen2.5-32B-Instruct
by fastserve's self-quantization pipeline
(fastserve/publish/), so it can be trusted the way a random community AWQ
requant sometimes can't — two of the community quants fastserve auto-detected
during its own benchmark run turned out to loop garbage tokens instead of
answering (see fastserve/benchmarks/RESULTS.md). Every checkpoint here
passed the same accuracy gate before being uploaded.
Quantization
- Method: AWQ (W4A16_ASYM), group size 128, via llm-compressor
- Calibration: 256 samples from
ultrachat-200k
Validation (accuracy gate — this is why you can trust it)
| bf16 baseline | this AWQ checkpoint | |
|---|---|---|
| GSM8K accuracy (n=30) | 0.9333 | 0.9333 |
Passed: quantized accuracy is within 0.1 (absolute) of the bf16 baseline, and less than 30% of responses looked degenerate (repeated- token loops — the failure mode found in the broken community quants above). A checkpoint that failed this gate would not have been uploaded.
Use
pip install vllm
python -m vllm.entrypoints.openai.api_server --model seoilgun/Qwen2.5-32B-Instruct-AWQ
Or with fastserve — point it at the original
Qwen/Qwen2.5-32B-Instruct id and it'll find this checkpoint automatically once
seoilgun/Qwen2.5-32B-Instruct-AWQ's namespace is registered as a priority search location.
License
Inherits the base model's license — see
Qwen/Qwen2.5-32B-Instruct for terms.
- Downloads last month
- -