shadowrock-io/Unlimited-OCR-Community-NVFP4

Unlimited-OCR — Community NVFP4 (calibrated)

Unofficial community quantization — not a Baidu release.

Calibrated NVFP4 build of baidu/Unlimited-OCR (revision 07dea832), the 3.34B DeepSeek-V2-style MoE OCR model. All credit for the model belongs to Baidu; this repo changes only the numeric precision of the text-decoder weights. Checkpoint size drops from 6.7 GB to ~2.8 GB.

Unlike prior community 4-bit builds of this model (data-free, weight-only), this build is GPTQ-calibrated through the MoE decoder with an OCR-domain corpus (document markdown, tables, invoices, multilingual passages), so expert weights are error-compensated against realistic activation statistics, and activations carry calibrated NVFP4 global scales for native FP4 execution on Blackwell.

Pick this variant for memory-constrained Blackwell deployments (RTX 50-series, Jetson Thor, B200). For near-lossless behavior on Ada/Hopper/Blackwell, use the companion FP8-Dynamic build.

What is quantized

Scheme: NVFP4 via llm-compressor GPTQ (offloaded hessians, 32 calibration sequences, max length 2048). Only the DeepSeek-V2 MoE text-decoder linears are quantized (2196 modules). Kept in BF16:

• SAM-ViT-B + CLIP-L DeepEncoder vision tower and projector
• embed_tokens / lm_head
• MoE router gates and all norms

Three of 768 routed-expert instances were never activated by the calibration corpus; their weights are quantized data-free from the original BF16 checkpoint and their activation scales derived from sibling-expert statistics (see source-repo issue #6 and quantization/repair_dead_experts.py). All other experts are GPTQ-calibrated.

Validation

Greedy OCR on the fixture set (vLLM 0.26.0, SM120, Marlin NVFP4 MoE backend), CER vs the BF16 baseline after stripping grounding tags:

	BF16	Data-free community NVFP4	This repo (calibrated)
Mean CER vs BF16	—	4.27*	0.0068
invoice / memo / table CER	—	unstable*	0.0 / 0.0 / 0.02
Decode throughput (tok/s, greedy)	45.3	64.2*	40.8
Checkpoint size	6.7 GB	2.8 GB	2.8 GB

* Prior community data-free build measured on the same harness: runaway repetition to the token cap on two fixtures and immediate EOS on the third — its higher tok/s reflects degenerate generation, not usable speed.

Two of three fixtures are character-identical to BF16 including box coordinates; the third differs by a single short span. See evidence/ for raw per-fixture transcripts, CER vs BF16, decode throughput, and peak VRAM from the source repo harness. Calibration corpus and provenance ship with the repo (QUANT_PROVENANCE.json).

Usage

import torch
from transformers import AutoModel, AutoTokenizer

repo = "shadowrock-io/Unlimited-OCR-Community-NVFP4"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True,
                                  torch_dtype=torch.bfloat16, device_map="cuda").eval()
text = model.infer(tok, prompt="<image>\n<|grounding|>OCR this image.",
                   image_file="document.png", output_path="./out",
                   base_size=1024, image_size=1024, crop_mode=False, eval_mode=True)

Note: transformers loads of NVFP4 checkpoints may require TORCH_COMPILE_DISABLE=1.

About

Quantized by Matt Busi at ShadowRock. Reproduction scripts (quantizer, calibration corpus, sanitizer, parity harness, fixtures) live in the source repo. Raw evaluation outputs ship under evidence/.