granite-docling-258M — LiteRT-LM (on-device document conversion, DocTags)

ibm-granite/granite-docling-258M converted to the LiteRT-LM (.litertlm) format for on-device document conversion with Google's LiteRT-LM runtime.

granite-docling is IBM's document-conversion VLM — the model behind Docling. Give it a page image and the prompt Convert this page to docling.; it emits DocTags, a structured markup with layout elements, OTSL table structure, code and formulas, which docling-core converts losslessly to Markdown, HTML or JSON. At 258M parameters (~338 MB here) it is one of the smallest document-AI models that captures table structure, not just text.

File	granite-docling-258M.litertlm (~338 MB)
Vision	SigLIP-base p16 (512×512, 1024 patches) + pixel-shuffle ×4 + Linear connector, int8 → 64 image tokens
Decoder	Llama-architecture granite decoder (576-dim, 30 layers, GQA 9/3, 100k vocab), int8 weights with float compute
Context (KV cache)	4096
Image input	must be pre-resized to 512×512 BILINEAR (see contract below)
Backend	CPU
Base model	ibm-granite/granite-docling-258M (Apache-2.0)

⚠️ Input contract: pre-resize to 512×512 BILINEAR

Resize the page to exactly 512×512 with BILINEAR resampling in your app before sending it. Do not hand the runtime a larger image: its internal resampler downscales with a different filter, and the model then produces a hallucinated page instead of the real one.

This is a base-model trait, not conversion damage: granite-docling is trained with image splitting (tiles + a global view), and this bundle runs the single-global-512 path — sharp on a 512-legible page, but sensitive to the exact resampling filter. The same full-precision model run the same way in transformers flips between perfect and degenerate on a BILINEAR↔LANCZOS change. Pre-sized 512 BILINEAR input is the verified path.

Scope. Best for pages whose text is legible at 512²: report-style pages, tables, layout extraction. Dense multi-column pages and small formulas degrade in single-512 mode — also in the original model run this way. For those, tile the page app-side and send crops.

Quality — measured, this exact bundle

Structure gate (synthetic report page with a 5×6 table), greedy decoding:

• macOS CPU (litert-lm 0.15.0): exact title, complete 5×6 OTSL grid, 25/25 data cells correct, clean stop — 5.2 s/page (~240 output tokens, Apple M4 Max).
• Galaxy S26 CPU (SM8850, LiteRT-LM v0.16.1 CLI): same page — exact title, complete grid, 25/25 cells, clean stop — 35.4 s/page (1025 output tokens). Two runs byte-identical.
• Conversion parity: the vision tower converts with corr 1.000 vs the original (fp32; shipped int8 corr 0.98, structurally identical output). Full-precision and int8-weight decoders produce the same table structure; int4 was tested and rejected — integer-compute quantization corrupts DocTags on this decoder, so the bundle ships int8 weights with float compute.

On-device demo (Galaxy S26, Google AI Edge Gallery 1.0.18, CPU)

Page + prompt	DocTags output (all 25 cells correct)

Captured live in the Gallery Ask Image task (imported via + → From local model file, max tokens 4096, TopK 1, CPU): the synthetic table page converts to DocTags with the exact title, the complete 5×6 grid and every cell value, finishing in 34.7 s — matching the 35.4 s CLI measurement on the same phone. The Gallery UI hides the <otsl>/<fcel> markup tags, so the table renders as concatenated text on screen; feed the raw output to docling-core for Markdown/HTML.

Performance (measured)

Text path, litert-lm benchmark -p 256 -d 256 --runs 3 --cache no; backend gated on a real docling generation first:

Device	Backend	Prefill (tok/s)	Decode (tok/s)	TTFT
Apple M4 Max (litert-lm 0.15.0)	CPU	912	64.0	0.57 s

The vision encoder runs once per image (~0.5 s on the M4 Max CPU) and is not included in the table.

GPU: the shipped bundle does not create a GPU engine — the GPU delegate rejects a quantized 576×576 projection at kernel init (Shape mismatch: {576,1,1,576} vs {1,1,576,576}, same signature on macOS and Android OpenCL). An fp16-decoder variant runs correctly on Android OpenCL but ~4× slower than CPU, so it is not shipped. CPU is the intended backend for this model.

Run on Android — Google AI Edge Gallery

Verified with Google AI Edge Gallery 1.0.18 on a Galaxy S26 (the demo above):

1. Put granite-docling-258M.litertlm on the phone (e.g. adb push … /sdcard/Download/).
2. In the Gallery: ☰ menu → Models → + (bottom-right) → From local model file, pick the file.
3. In the Import dialog set max tokens 4096 and turn on "Support image", then Import. The model appears under Imported models in the Ask Image task (Accelerator: CPU).
4. Open Ask Image, pick the model, attach a 512×512 pre-resized page, and prompt Convert this page to docling. (TopK 1 in the model configs reproduces the greedy gate output.)

The Gallery renders DocTags as plain text (markup tags hidden); convert the copied output with docling-core for Markdown/HTML.

Run on iPhone / macOS

Use the LiteRT-LM Swift runtime (swift-litert-lm). Load the bundle with the vision tower enabled (Modality.textImage — vision-only bundle, no audio tower), attach the pre-resized page, and send Convert this page to docling. One page per conversation; start a new one for the next page.

Conversion notes

• LiteRT-LM fast_vlm bundle: VISION_ENCODER ([1,512,512,3]→[1,1024,768], SigLIP static-position path, normalization baked in) + VISION_ADAPTER (pixel-shuffle ×4 + Linear 12288→576 → [1,64,576]) + single-token EMBEDDER + PREFILL_DECODE. Single global 512² view, no tiling → fixed 64 soft tokens.
• The idefics3 vision tower is architecturally identical to SmolVLM2's and converts with the same scripts, unchanged (corr 1.000, no FLEX/custom ops).
• Decoder: the granite 100k-vocab Llama-architecture decoder, int8 weights + float compute, prefer_activation_type=fp32_fp16 declared in-bundle. Integer-compute int8/int4 corrupt DocTags structure — document-parsing decoders at this size are quantization-sensitive (same behavior family as PaddleOCR-VL's ERNIE decoder).
• Tokenizer: the HF tokenizer.json (GPT-2 byte-BPE) with all DocTags added tokens at their exact ids. No BOS token (add_bos_token=false); stop token <|end_of_text|>.
• Template: granite chat format. The newline after <|end_of_text|> is part of the format — this model is format-exact, and dropping that single token turns output into a hallucinated blank page. If you re-template, protect the \n.

License

Apache-2.0, inherited from the base model ibm-granite/granite-docling-258M.