Qwen3-VL-8B-DocBBox-LoRA

Extract any fields you define from scanned or digital documents, and get back exactly where on the page each value was found.

The problem

Companies pull data out of invoices, loan files, contracts and forms every day, and every extracted value has to be checked. Vision-language models can read documents, but they rarely say where a value came from, so a reviewer has to hunt for it on the page. Classic extraction tools can point to the text, but need OCR plus a model trained for each document type.

This adapter makes Qwen3-VL-8B do both: you write the fields you want as a JSON schema, and every value comes back with a bounding box, so it can be verified at a glance, highlighted in a review UI, or checked automatically.

A noisy 1961 typewritten form scan, same schema and prompt for both. Both models read 15/18 values, but the base model puts only 1/18 boxes on the value (most land on the printed label or the wrong column); with this adapter 17/18 boxes are on the value.

What it does

Input: 1–4 page images + your schema.

{"seller_name": "string | name of the seller", "invoice_date": "date | invoice date", "total": "number | total amount after tax"}

Output: the same structure, each field with its value, the exact printed text, the page and a box.

{"seller_name":  {"value": "Stein-Fernandez", "raw_text": "Stein-Fernandez", "page": 1, "bbox": [6, 70, 164, 82]},
 "invoice_date": {"value": "2003-09-30",      "raw_text": "30-Sep-2003",     "page": 1, "bbox": [837, 93, 955, 107]},
 "total":        {"value": 597.89,            "raw_text": "597.89 USD",      "page": 1, "bbox": [673, 473, 779, 485]}}

• Any schema, no retraining: nested objects, arrays of line items, typed fields (string | number | date | boolean).
• Values are normalised (dates to YYYY-MM-DD, numbers without currency symbols), and the original text is kept in raw_text.
• No guessing: fields not on the page come back as null.
• Boxes are [x1, y1, x2, y2] in 0–1000, relative to page width and height; multi-line values get one box per line.

How much better than the base model

400 test pages (1,801 fields present on the page), same prompt for both models.

Per field - share of individual fields that are right:

	Value exactly right	Box on the right text	Box on the right text, document types never seen in training
Qwen3-VL-8B-Instruct	52%	41%	31%
+ this adapter	79%	74%	60%

Per page - share of pages where every field is right:

	All values exactly right	All values and all boxes right
Qwen3-VL-8B-Instruct	36%	28%
+ this adapter	61%	54%

"Exactly right" is strict: the value must equal the gold value after normalising case, whitespace and number formatting (no partial credit; with partial credit for near-miss strings, per-field value accuracy is 84%). A box counts as right when its centre lands on the gold text. Fields that are not on the page (342) came back as null 100% of the time, with no invented values. These are results on the BoundingDocs test mix below; accuracy on your own documents will differ, so test on a sample first.

📦 Files

File	What
adapter_model.safetensors, adapter_config.json	LoRA r=64, alpha=128 on all language-model linear layers (vision encoder untouched), bf16
system_prompt.txt	The output contract the adapter was trained with - always send it as the system message
examples/infer_transformers.py	Single document → JSON + image with boxes drawn (transformers + peft)
examples/infer_vllm.py	Fast batch inference with vLLM's LoRA support
examples/schema_invoice.json, schema_loan.json, sample_invoice.jpg	Example inputs

🚀 Usage

pip install -U "transformers>=4.57" peft accelerate pillow

import json, torch
from huggingface_hub import hf_hub_download
from peft import PeftModel
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

BASE, ADAPTER = "Qwen/Qwen3-VL-8B-Instruct", "greedydecoder/Qwen3-VL-8B-DocBBox-LoRA"

processor = AutoProcessor.from_pretrained(BASE)
model = AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16, device_map="cuda")
model = PeftModel.from_pretrained(model, ADAPTER).eval()
system_prompt = open(hf_hub_download(ADAPTER, "system_prompt.txt"), encoding="utf-8").read().strip()

page = Image.open(hf_hub_download(ADAPTER, "examples/sample_invoice.jpg")).convert("RGB")
schema = {
    "seller_name": "string | name of the seller",
    "invoice_date": "date | invoice date",
    "total": "number | total amount after tax",
}

messages = [
    {"role": "system", "content": [{"type": "text", "text": system_prompt}]},
    {"role": "user", "content": [
        {"type": "image", "image": page},
        {"type": "text", "text": "Extract the following schema from the document.\n" + json.dumps(schema, indent=1)},
    ]},
]
inputs = processor.apply_chat_template(messages, tokenize=True, add_generation_prompt=True,
                                       return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=4096, do_sample=False)
result = json.loads(processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
print(json.dumps(result, indent=1))

# bbox -> pixels on the image you sent
W, H = page.size
x1, y1, x2, y2 = result["total"]["bbox"]
print("total is at", (x1 * W / 1000, y1 * H / 1000, x2 * W / 1000, y2 * H / 1000))

Command-line version with PDF support and box drawing:

python examples/infer_transformers.py invoice.pdf examples/schema_invoice.json --pages 1-2 --out boxes.png
python examples/infer_transformers.py invoice.pdf examples/schema_invoice.json --no-adapter   # base model, to compare

vLLM (Linux, ~110 pages/min on one RTX 5090):

llm = LLM(model="Qwen/Qwen3-VL-8B-Instruct", enable_lora=True, max_lora_rank=64, max_model_len=16384)
outs = llm.chat(conversations, SamplingParams(temperature=0, max_tokens=4096),
                lora_request=LoRARequest("docbbox", 1, snapshot_download("greedydecoder/Qwen3-VL-8B-DocBBox-LoRA")))

Full script: examples/infer_vllm.py.

⚙️ Recommended parameters

• Greedy decoding (do_sample=False / temperature=0), max_new_tokens 4096.
• Image budget: 2 MP per page (what it was trained at; sides rounded to multiples of 32). 3–4 MP gives slightly better boxes at ~20–40% lower throughput (see below).
• Send the system_prompt.txt contract and the user text Extract the following schema from the document.\n<schema> exactly - the adapter is trained on that format.
• Leaf descriptions matter: "date | date the loan closed" disambiguates better than "date | date".
• VRAM: ~18 GB for bf16 base + adapter.

📊 Evaluation

400 single pages from the BoundingDocs test split (stratified over 9 datasets / 11 sub-sources), greedy, same prompt for both models. All metrics are per field, over the 1,801 fields present on the page. field value exact = normalised exact match of value; missed = field returned as null although it is on the page; center-hit = predicted box centre lies inside the gold box (the right occurrence was found); held-out = document families never seen in training (VRDU ad-buy / registration forms, Kleister NDA / Charity).

Model	field value exact	mean IoU	IoU ≥ 0.5	center-hit	held-out center-hit	missed
Qwen3-VL-8B-Instruct (base)	0.522	0.241	0.252	0.408	0.309	0.313
+ this LoRA, 2 MP	0.790	0.630	0.704	0.744	0.600	0.049
+ this LoRA, 4 MP	0.811	0.639	0.722	0.767	0.642	0.043

The base model often ignores the leaf envelope (returns bare values with no boxes), which explains most of its "missed".

🏋️ Training

1. SFT - LoRA r=64 / alpha=128 / dropout 0.05 on all LLM linear layers, 1 epoch over 13k pages with randomly generated schemas per page (random key style, descriptions, nesting, absent fields), so the model follows any schema rather than memorising document types. Loss on the assistant JSON only. ms-swift, bf16, ~2 MP images.
2. GRPO - 400 steps (G=4, 800 hard prompts with the largest reward spread across SFT samples). Per-leaf reward = 0.3 value match + 0.5 box IoU + 0.2 box-centre hit; absent fields score 1 only for a null; averaged over leaves, 0 for invalid JSON. Raised held-out center-hit from 0.542 to 0.600.

Data (pages used for SFT; the GRPO prompts are a subset of the same sources):

Source	Pages	Licence of the source
DeepForm (via BoundingDocs)	3,713	public FCC political-ad filings
SP/MP-DocVQA (via BoundingDocs)	2,831	DocVQA terms of use (research use); documents from the UCSF Industry Documents Library
FATURA (via BoundingDocs)	2,259	as published by the authors (synthetic invoices)
DUDE (via BoundingDocs)	2,120	as published by the authors
XFUND (via BoundingDocs)	1,158	CC BY-NC-SA 4.0
CORD-v2	795	CC-BY-4.0
FUNSD	179	non-commercial research only

BoundingDocs v1.0 (CC-BY-4.0) supplied the question/answer locations for its pages. VRDU and Kleister documents were kept out of training and are used only as a held-out test set.

⚠️ Limitations

• Repeated text: when the same value is printed several times (e.g. a date in a header and in a table row), the box sometimes points at the wrong copy. Weakest family: VRDU ad-buy tables (center-hit 0.56).
• Evaluated on English, mostly printed business documents (invoices, receipts, forms, contracts). The training data includes some multilingual forms (XFUND), but other languages and handwriting were not evaluated.
• Boxes come from a language model: verify anything that matters, and use raw_text to cross-check against OCR if you need exact word boxes.

License

The adapter weights are released under CC BY-NC-SA 4.0 - non-commercial use only, with attribution, and derivatives must be shared under the same licence. This follows the most restrictive training sources:

• XFUND is licensed CC BY-NC-SA 4.0 (non-commercial, share-alike);
• FUNSD is licensed for non-commercial research only;
• DocVQA documents are distributed under research-use terms.

Commercial use of these weights is not permitted. The base model, Qwen3-VL-8B-Instruct, is Apache-2.0 and is not affected by this licence; only the adapter is. The example images are from the FUNSD and FATURA test sets and remain under their own licences.

💼 Commercial use and custom models

These weights are for non-commercial use. For a commercial licence, or a model trained on your own document types, open a discussion in the Community tab.

Citation

@misc{qwen3vl_docbbox_lora_2026,
  title  = {Qwen3-VL-8B-DocBBox-LoRA: schema-driven document extraction with bounding boxes},
  year   = {2026},
  url    = {https://huggingface.co/greedydecoder/Qwen3-VL-8B-DocBBox-LoRA}
}