zk0hf/Bonsai-27B-mlx-1bit-vision4bit

Bonsai-27B-mlx-1bit, vision tower at 4-bit

A repack of prism-ml/Bonsai-27B-mlx-1bit that quantizes the vision tower. 5.129 GB → 4.666 GB (−9.0%). The language model is untouched, bit for bit.

All credit for the model belongs to Prism ML; this repo only changes how one component is stored.

Why

The published pack quantizes the language model to 1 bit but ships the vision tower dense F16, at 0.921 GB — nearly a fifth of the total, in a model whose whole premise is extreme compression. Quantizing those Linear layers to 4-bit is ordinary MLX work.

language model, 498 modules (1-bit, untouched)	3.362 GB
scales + biases	0.840 GB
vision tower, 83 Linears → 4-bit/group-64	0.244 GB
vision tower, 27 Linears left at F16	0.216 GB
total	4.666 GB

Those 27 are a hard limit, not a choice: MLX requires in_features % group_size == 0 and its Metal kernels exist only for group 32/64/128, but mlp.linear_fc2 has in_features 4304 = 16 × 269, which none of them divide.

Quality

On a Japanese receipt with known ground truth, output is byte-identical to the original pack — every field, every line item, and every amount (subtotal ¥1,188, tax ¥95, total ¥1,283). Measured peak GPU memory falls from 6.54 GB to 6.07 GB at 558k pixels.

This is one document, not a benchmark. Treat it as evidence that 4-bit is not obviously lossy for the vision path, not as a quality guarantee.

Requirements

Needs the PrismML fork of mlx-swift (or their mlx fork) for 1-bit kernels. Upstream MLX rejects bits=1 — "The supported bits are 2, 3, 4, 5, 6 and 8" — and compiles quantized Metal kernels only for those widths, so on stock MLX this checkpoint is unloadable rather than merely slow.

Note it does not fit a 6 GB iPhone. 4.666 GB of weights and a 6.07 GB peak need a 12 GB-class device.

Notes on what is not reducible

The 1-bit language model cannot be compressed further. Its codes measure exactly 1.0000 bits of entropy per weight, 100% of sampled 128-bit group patterns are distinct, and lzma -9 --extreme and bzip2 -9 both return ≥100% of the original size on 67 MB of real codes. Sub-1-bit would have to discard information rather than compress it, and these weights are already at the 1-bit floor from Bonsai's QAT.

A further 0.420 GB is free: the pack stores biases that equal -scale/2 bitwise across 210.1M of 210.1M groups, so they carry no information, and PrismML's affine_sym kernels rederive them. Not applied here because mlx-swift-lm's loader currently rejects a biasless QuantizedLinear (verify: [.all] demands the key even though the property is already optional).

Reproducing

scripts/repack_bonsai.py in ondevice-vlm-ios.