saisaketh/Qwen2.5-0.5B-Instruct-armq-265MB

Qwen2.5-0.5B-Instruct — armq 265 MB (mixed precision)

A mixed-precision quantization of Qwen/Qwen2.5-0.5B-Instruct, built with armq.

Rather than giving every layer the same bit-width, armq measures how sensitive each layer actually is and solves for the bit allocation that minimises quality loss subject to a hard size budget. This checkpoint occupies exactly the same bytes as a uniform 4-bit build of the same model, and scores better.

Measured against uniform 4-bit at identical size

	uniform 4-bit	this model
Size	265.1 MB	265.1 MB	identical
bits/weight	4.502	4.502	identical
Perplexity ↓	21.3998	20.4113	-4.6%
Generation speed	187.9 tok/s	162.7 tok/s	-13%

Perplexity on 32 × 512-token windows (16352 predicted tokens) of held-out wikitext-2 test. Calibration used the train split only, so the recipe was never tuned on the text it is scored against.

Measured on Apple M2 (arm64), 8.0 GB unified memory, MLX 0.32.0, mlx-lm 0.31.3.

Generation speed changed by -13% on the test machine. Mixed precision means adjacent layers no longer share a kernel configuration, and whether that helps or hurts is architecture-dependent — it went the other way on the other model in this pair. armq optimises quality-at-size, not latency.

Usage

Loads like any MLX checkpoint — no custom code:

from mlx_lm import load, generate

model, tokenizer = load("saisaketh/Qwen2.5-0.5B-Instruct-armq-265MB")
print(generate(model, tokenizer, "Arm devices are", max_tokens=100))

pip install mlx-lm
mlx_lm.generate --model saisaketh/Qwen2.5-0.5B-Instruct-armq-265MB --prompt "Arm devices are"

The allocation

3-bit x3, 4-bit x87, 5-bit x33, 6-bit x38, 8-bit x8

The full per-layer assignment ships in armq_recipe.json in this repo. It is a few KB and reproduces this checkpoint exactly from the base model:

pip install git+https://github.com/saketh8/arm-mlx-optimizer
armq build Qwen/Qwen2.5-0.5B-Instruct --recipe armq_recipe.json --out ./rebuilt

The pattern armq found is not one you would guess: value projections and the output head are protected at high precision, while query, gate and up projections tolerate 3–4 bits.

Limitations

• Apple Silicon / MLX only. The method is portable; this build is not.
• Perplexity is a proxy for quality, not a downstream task score. It is the right proxy for this particular comparison — same base model, same data, one variable — but it is not a benchmark suite.
• Calibration is general-purpose (a few thousand wikitext tokens). A domain-specific deployment should re-profile on representative text.
• Throughput is not guaranteed to improve — see above.

License & attribution

Weights are a derivative of Qwen/Qwen2.5-0.5B-Instruct and inherit its apache-2.0 license. The armq tooling that produced them is MIT.

Method, benchmarks and reproduction instructions: https://github.com/saketh8/arm-mlx-optimizer