jmbarrancoidener/PaddleOCR-VL-1.6-mlx-4bit

PaddleOCR-VL-1.6 · MLX 4-bit

An MLX 4-bit quantisation of PaddlePaddle/PaddleOCR-VL-1.6, for Apple Silicon. 682M on disk, 6.214 bits per weight after mixed-precision quantisation.

All the modelling work is PaddlePaddle's. This repository holds converted weights and nothing else.

Reproducing it

pip install "mlx-vlm>=0.3.11"
mlx_vlm convert --hf-path PaddlePaddle/PaddleOCR-VL-1.6 \
  -q --q-bits 4 --mlx-path paddleocr-vl-4bit --trust-remote-code

Deterministic: a fresh convert reproduces model.safetensors byte-identically, checked by SHA-256. The build here is the output of exactly that command, so this repository is a convenience rather than a source of truth. Downloading it saves pulling the 1.8 GB bf16 first.

sha256  5d1ce8b52bf14c528a4b860ef8ed74a8136aecf21d98ed514c09522a19cdf559

Quality

Measured against the bf16 original on the same pages of the same PDFs, counting layout regions whose recognised text differs.

build	size	agreement with bf16
bf16	1.8 GB	—
8-bit	1.0 GB	2 differing regions
4-bit	682 MB	8 differing regions

Those counts only mean something against a control: bf16 compared with itself across two runs differs in 1 region, and it is the same flip-flopping superscript that separates it from the 8-bit build.

So 8-bit sits inside the model's own non-determinism. 4-bit sits outside it, and its differences are systematic rather than random: markdown bullets degrade into literal • characters and the occasional character drops mid-word.

Neither is faster than bf16. At 0.9B the weights are not the bottleneck; image prefill is, at roughly 2.5 s to emit two tokens per layout region. Pick a quantisation for disk and memory, not for speed.

Use 8-bit if you want the gigabyte back. 4-bit costs accuracy and returns nothing but disk.

Use

from mlx_vlm import load, generate

model, processor = load("jmbarrancoidener/PaddleOCR-VL-1.6-mlx-4bit", trust_remote_code=True)

It also works as the recognition backend for the full PaddleOCR-VL pipeline, which is what it was converted for:

from paddleocr import PaddleOCRVL

pipeline = PaddleOCRVL(
    vl_rec_backend="mlx-vlm-server",
    vl_rec_server_url="http://localhost:8111/",
    vl_rec_api_model_name="jmbarrancoidener/PaddleOCR-VL-1.6-mlx-4bit",
)

Licence

Apache-2.0, inherited from the base model. Quantisation changes the weights and nothing about the licence or the attribution.

Converted for Vela, a research agent that reads papers.