naklitechie/Unlimited-OCR-DeepEncoder-ONNX

Unlimited-OCR DeepEncoder — ONNX (browser-ready vision stack)

The complete vision encoder of baidu/Unlimited-OCR (the DeepSeek-OCR family DeepEncoder: SAM ViT-B → CLIP-L fusion → linear projector) exported as a single ONNX graph, verified numerically identical to the PyTorch reference (torch-vs-onnxruntime cosine 1.0000000, maxAbsDiff 5e-5).

Paired with the language-model GGUF, this runs the full Unlimited-OCR pipeline entirely in a browser tab — image in, det-boxed markdown out, no server. Reference implementation: NakliTechie/gemma4-webgpu (hand-written WGSL WebGPU engine with DeepSeek-V2-MoE decoder support; see ocr-demo.html). Measured on Apple Metal-3: vision 1.5 s (onnxruntime-web WebGPU EP) + decode at ~125 tok/s → single page OCR'd in < 3 s in-tab.

Files

file	what
deepencoder_fp32.onnx	SAM→CLIP→projector, opset 18, static [1,3,1024,1024] → [1,256,1280] (fp32, 1.6 GB)
deepencoder_extras.npz	image_newline, view_seperator embeddings ([1280] f32 each) — spliced host-side, not part of the graph
deepencoder_ref_in.npy / deepencoder_ref_out.npy	parity fixtures (seed-42 input + reference output) — verify your runtime reproduces cosine ≈ 1.0
control_doc.png	the ground-truth test document used in the e2e verification

I/O contract

• Input pixel_values [1,3,1024,1024] f32 — RGB, normalized (x/255 − 0.5)/0.5 (mean/std 0.5, per the upstream processor config).
• Output vision_embeds [1,256,1280] f32 — 256 vision tokens (16×16 grid) already projected to the decoder's hidden size.

Splicing into the decoder sequence (single 1024² global view)

[BOS(0)]
+ 16 rows × ( 16 patch embeds [row-major] + image_newline )
+ view_seperator                       → 273 embedding rows total
+ prompt token ids

The 273 rows replace <image> placeholder tokens (id 128815) — feed them as inputs_embeds.

⚠ The prompt matters more than you think

Use this model family's canonical prompt: <image>document parsing. DeepSeek-OCR-v1 phrasings fail on Unlimited-OCR — <image>\nFree OCR. produces an immediate EOS even in the bf16 reference, and <|grounding|>Convert the document to markdown. makes it recite instruction boilerplate. (Verified against the full-precision HF stack; see the engine repo's reference/pytorch/hf_image_control.py.)

The rest of the pipeline

• Decoder GGUF (DeepSeek-V2 MoE, 64×550M, 12 layers — no MLA despite the family name: use_mla: false, plain Llama MHA): community K-quants at sahilchachra/Unlimited-OCR-GGUF (Q4_K_M 1.95 GB works with the engine's in-shader q4k/q8 storage).
• Browser engine: NakliTechie/gemma4-webgpu — WGSL kernels incl. on-GPU top-6 MoE routing and batched expert GEMVs; crossLabDiff-verified against the HF bf16 reference (per-layer sweep, argmax match).

Provenance & license

Weights are a mechanical export of baidu/Unlimited-OCR (MIT, © 2026 Baidu — notice retained per license). Export script: reference/pytorch/export_deepencoder_onnx.py in the engine repo. fp32; an fp16 (~800 MB) pass is planned — mind LayerNorm precision if you convert yourself.