OvisOCR2-6bit

6-bit MLX quantization of ATH-MaaS/OvisOCR2, an 853M-parameter OCR / document-parsing VLM. Converted with mlx-vlm 0.6.8 (mlx 0.32.0) for inference on Apple Silicon.

Fidelity was measured against the unquantized bf16 source, not assumed. The numbers below and the method that produced them are in the tables that follow.

pip install mlx-vlm
python -m mlx_vlm generate \
    --model mlx-community/OvisOCR2-6bit \
    --image document.png \
    --prompt "Transcribe this document to markdown." \
    --max-tokens 512

Quantization

Bits	6
Group size	64
Mode	affine
Effective bits/weight	7.626
Size on disk	0.81 GB
Quantized tensors	187 (language model)
Unquantized tensors	153 (vision tower, bf16)

Effective bits per weight exceeds the nominal 6 because mlx-vlm quantizes only the language model and leaves the vision tower in bf16 by design. Verified, not assumed: of 153 vision tensors, zero carry a .scales entry.

Note this model ties its input and output embeddings, and the embedding is 33.8% of all quantized parameters, a much larger share than in a typical 7B. Weight error is therefore dominated by the embedding rather than the attention layers.

Measured fidelity

Weight-level metrics compare dequantized weights against the bf16 source tensor by tensor, streamed from disk. No prompts, no sampling, hardware-independent.

Variant	bpw	SNR	Relative L2	Cosine	CER vs bf16	Decode tok/s	Peak RAM
4-bit	5.863	20.58 dB	9.35%	0.995632	0.3311	220.3	0.86 GB
6-bit	7.626	32.84 dB	2.28%	0.999740	0.0000	183.5	1.08 GB
8-bit	9.389	42.67 dB	0.74%	0.999973	0.0000	160.9	1.31 GB
bf16 (source)	16	n/a	n/a	n/a	0.0000	91.0	1.83 GB

Measured on an M2 Pro / 32 GB, greedy decoding.

OCR accuracy vs ground truth

Because OCR has ground truth, transcription can be scored directly rather than inferred from token distributions. Five documents were rendered locally with exactly known content (invoice, clinical lab report, shipping label, receipt, motor-controller datasheet), covering tabular line items, currency, dates and identifiers.

Variant	Field accuracy	Content accuracy	Numeric recall	CER vs bf16
4-bit	0.9677	1.0000	1.0000	0.3311
6-bit	1.0000	1.0000	1.0000	0.0000
8-bit	1.0000	1.0000	1.0000	0.0000
bf16 (source)	1.0000	1.0000	1.0000	0.0000

• Field accuracy: required strings present, exact match (format-sensitive).
• Content accuracy: the same, ignoring markup and whitespace.
• Numeric recall: ground-truth numbers present, ignoring markup. For an invoice or a datasheet the numbers are the payload.

Every variant preserves 100% of content and 100% of the 77 ground-truth numbers. At 6 and 8 bits the transcriptions are character-identical to bf16 (CER 0.0000 across all five documents), not merely close.

Formatting stability

At this bit-width the transcription is byte-for-byte identical to the bf16 source across all five documents. The 4-bit variant instead emits HTML tables where bf16 emits markdown; it preserves all content and every number, but is not byte-stable. See OvisOCR2-4bit for that trade-off.

Of the two byte-stable variants, 6-bit is the better default: identical output, meaningfully faster, and smaller.

What was not measured

• No distributional metrics (perplexity, top-1 agreement, KL). They were run and are deliberately not reported. This model is an OCR specialist, and on general chat text the bf16 source itself degenerates (asked for the capital of Japan it emits Tokyo / Nara / Nara / Nara ...). Its perplexity spans 49.3x across passages of ordinary prose, so an aggregate over them measures domain mismatch in the source, not quantization error. Per-passage ratios do improve monotonically from 4 to 8 bits.
• No standard OCR benchmark. No OmniDocBench, no DocVQA, no CER against a public dataset. The five documents are synthetic and rendered with a default font: exact ground truth, but not photographs, not handwriting, not skewed scans, and not non-Latin scripts.
• No judged quality, no layout-structure scoring (reading order, cell spanning), no throughput figures for any chip other than an M2 Pro.

Sibling variants

Variant	bpw	Character-identical to bf16
OvisOCR2-4bit	5.863	no, formatting differs
OvisOCR2-6bit	7.626	yes
OvisOCR2-8bit	9.389	yes

Credits

All credit for the underlying model belongs to ATH-MaaS. This is a format conversion and quantization; no training or fine-tuning was performed. See the source model card for license, intended use and limitations.