shadowrock-io/Unlimited-OCR-Community-FP8-Dynamic

Unlimited-OCR — Community FP8-Dynamic

Unofficial community quantization — not a Baidu release.

FP8 W8A8 build of baidu/Unlimited-OCR (revision 07dea832), the 3.34B DeepSeek-V2-style MoE OCR model ("one-shot long-horizon parsing"), validated against a BF16 baseline of the published checkpoint. All credit for the model belongs to Baidu; this repo changes only the numeric precision of the text-decoder weights. Checkpoint size drops from 6.7 GB to 3.9 GB.

Pick this variant for Ada/Hopper/Blackwell GPUs where you want near-lossless behavior with native FP8 execution. The companion NVFP4 build — calibrated, unlike prior community 4-bit builds — shrinks the footprint further for memory-constrained Blackwell deployments.

What is quantized

Scheme: FP8_DYNAMIC via llm-compressor — per-channel FP8 weights, dynamic per-token FP8 activations, data-free. Only the DeepSeek-V2 MoE text-decoder linears are quantized (2196 modules: 2112 routed-expert, 33 shared-expert/dense-MLP, 48 attention, 3 dense-layer). Everything the vision path depends on stays BF16:

• SAM-ViT-B + CLIP-L DeepEncoder vision tower and projector
• embed_tokens / lm_head
• MoE router gates and all norms

Validation

Greedy OCR on the fixture set (vLLM 0.26.0, SM120) is character-identical to the BF16 baseline — CER 0.0 on every fixture, grounding-box coordinates included. The same parity holds under transformers + compressed-tensors.

	BF16	FP8-Dynamic (this repo)
Mean CER vs BF16	—	0.0000
invoice / memo / table CER	—	0.0 / 0.0 / 0.0
Decode throughput (tok/s, greedy)	45.3	39.4
Checkpoint size	6.7 GB	3.9 GB

Raw per-fixture transcripts ship under evidence/, produced by the source repo harness.

Note: the shipped quantization_config.ignore uses prefix-agnostic regex patterns — required for vLLM to correctly skip the vision tower (source-repo issue #5). Do not replace it with expanded module names.

Usage

import torch
from transformers import AutoModel, AutoTokenizer

repo = "shadowrock-io/Unlimited-OCR-Community-FP8-Dynamic"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True,
                                  torch_dtype=torch.bfloat16, device_map="cuda").eval()
text = model.infer(tok, prompt="<image>\n<|grounding|>OCR this image.",
                   image_file="document.png", output_path="./out",
                   base_size=1024, image_size=1024, crop_mode=False, eval_mode=True)

About

Quantized by Matt Busi at ShadowRock. Reproduction scripts (quantizer, sanitizer, parity harness, fixtures) live in the source repo. Raw evaluation outputs ship under evidence/.