form-field-vlm-v2 · commercial
Detect and extract interactive form fields from a page image. The Nutrient v2 system returns each field's bounding box, fine type, text label, description, and the entered value as constrained JSON — across empty, filled, and handwritten pages. It performs the full semantic job in a single pass: field detection, fine typing, labeling, value reading, description, and radio-group linking. An optional hybrid tightens boxes with the Nutrient RF-DETR detector.
The runnable weights are commercial and are not downloadable from this repository. This public page is the product specification and scorecard.
- 🎯 Try it: form-field-vlm-v2-demo
- 🏆 Leaderboard: form-field-vlm-v2-leaderboard
- 📊 Benchmark: form-field-vlm-v2-benchmark
Headline result
Systems are ranked by End-to-end F1: a field counts only when the box (IoU ≥ 0.5), the field type,
and the entered value are all correct. Cells below show IoU 0.5 / IoU 0.2 — the strict production
threshold and a looser one that diagnoses approximately-correct-but-loose boxes. Detect isolates
find-and-type; Value and Label are read over correctly-detected fields (free text by multilingual
bge-m3 cosine, choice values exact). Evaluated on 354 held-out form pages / 8,220 gold fields (empty,
filled, handwritten), macro-averaged over field types.
| System | End-to-end .5 / .2 |
Detect .5 / .2 |
Value | Label | sec/page |
|---|---|---|---|---|---|
| Nutrient Hybrid (detector + VLM) | 0.565 / 0.582 | 0.628 / 0.650 | 0.917 | 0.713 | 2.5 |
| Nutrient VLM v2 | 0.518 / 0.582 | 0.550 / 0.621 | 0.952 | 0.775 | 2.2 |
| GPT-5.6 Sol | 0.338 / 0.400 | 0.461 / 0.547 | 0.689 | 0.673 | — |
| GPT-5.6 Terra | 0.315 / 0.388 | 0.407 / 0.533 | 0.792 | 0.677 | — |
| Gemini Flash | 0.245 / 0.309 | 0.375 / 0.452 | 0.687 | 0.726 | — |
| Gemini Pro | 0.216 / 0.282 | 0.330 / 0.447 | 0.657 | 0.713 | — |
| Gemini Flash-Lite | 0.168 / 0.227 | 0.264 / 0.336 | 0.700 | 0.664 | — |
| GPT-5.6 Luna | 0.125 / 0.226 | 0.195 / 0.349 | 0.672 | 0.678 | — |
| Claude Haiku 4.5 | 0.011 / 0.045 | 0.016 / 0.077 | 0.867 | 0.574 | — |
| Claude Opus 4.8 | 0.010 / 0.038 | 0.018 / 0.072 | 0.795 | 0.695 | — |
| Claude Sonnet 5 | 0.009 / 0.040 | 0.015 / 0.069 | 0.684 | 0.624 | — |
The Nutrient VLM reaches 0.518 End-to-end, and the detector+VLM hybrid 0.565 — well ahead of the strongest cloud VLM here, GPT-5.6 Sol (0.338). Two things the breakdown makes clear:
- The gap is detection, not reading. Every model reads reasonably once a field is found (Value 0.66–0.95), but localizing tightly on dense forms is where the Nutrient models separate (Detect 0.55 vs ≤0.46). The hybrid supplies tighter RF-DETR boxes, converting approximately-correct detections into strict IoU ≥ 0.5 hits (Detect 0.550 → 0.628; End-to-end 0.518 → 0.565). At the loose IoU 0.2 the VLM and hybrid converge (both 0.582), confirming the hybrid's contribution is box tightness.
- The Claude family cannot ground boxes on dense forms (Detect ~0.016) despite reading labels well — a known visual-grounding limitation; their fields land in roughly the right area but rarely within IoU 0.5.
Detailed result
Each cell below is IoU 0.5 / IoU 0.2, heat-shaded by the strict value; Detect and seconds/page close out each row.
End-to-end F1 by render condition
| System | Overall | Empty | Filled | Handwritten |
|---|---|---|---|---|
| Nutrient Hybrid (detector + VLM) | 0.565 | 0.692 | 0.702 | 0.543 |
| Nutrient VLM v2 | 0.518 | 0.590 | 0.694 | 0.544 |
| GPT-5.6 Sol | 0.338 | 0.492 | 0.436 | 0.345 |
| GPT-5.6 Terra | 0.315 | 0.542 | 0.393 | 0.282 |
| Gemini Flash | 0.245 | 0.373 | 0.282 | 0.268 |
| Gemini Pro | 0.216 | 0.303 | 0.219 | 0.262 |
| Gemini Flash-Lite | 0.168 | 0.310 | 0.167 | 0.147 |
| GPT-5.6 Luna | 0.125 | 0.249 | 0.242 | 0.113 |
| Claude Haiku 4.5 | 0.011 | 0.031 | 0.029 | 0.001 |
| Claude Opus 4.8 | 0.010 | 0.028 | 0.021 | 0.005 |
| Claude Sonnet 5 | 0.009 | 0.027 | 0.017 | 0.002 |
End-to-end F1 by field type (IoU 0.5)
| System | Text | Checkbox | Radio | Select | Signature |
|---|---|---|---|---|---|
| Nutrient Hybrid | 0.743 | 0.590 | 0.487 | 0.300 | 0.706 |
| Nutrient VLM v2 | 0.641 | 0.481 | 0.456 | 0.300 | 0.712 |
| GPT-5.6 Sol | 0.594 | 0.391 | 0.184 | 0.083 | 0.439 |
| GPT-5.6 Terra | 0.516 | 0.300 | 0.332 | 0.156 | 0.271 |
| Gemini Flash | 0.484 | 0.228 | 0.181 | 0.000 | 0.332 |
| Gemini Pro | 0.572 | 0.161 | 0.141 | 0.000 | 0.207 |
| Gemini Flash-Lite | 0.309 | 0.237 | 0.134 | 0.000 | 0.158 |
| GPT-5.6 Luna | 0.273 | 0.091 | 0.119 | 0.045 | 0.096 |
| Claude Haiku 4.5 | 0.016 | 0.009 | 0.001 | 0.021 | 0.007 |
| Claude Opus 4.8 | 0.024 | 0.003 | 0.001 | 0.019 | 0.006 |
| Claude Sonnet 5 | 0.018 | 0.006 | 0.021 | 0.000 | 0.000 |
Localization & error diagnostics (IoU 0.5)
| System | Detect P / R | Box recall | FP/page | count-MAE | pages returned |
|---|---|---|---|---|---|
| Nutrient Hybrid | 0.749 / 0.807 | 0.895 | 4.25 | 4.28 | 352/354 |
| Nutrient VLM v2 | 0.700 / 0.567 | 0.613 | 4.61 | 8.01 | 336/354 |
| GPT-5.6 Sol | 0.611 / 0.575 | 0.646 | 6.84 | 6.37 | 295/354 |
| GPT-5.6 Terra | 0.547 / 0.541 | 0.609 | 8.81 | 5.01 | 306/354 |
| Gemini Flash | 0.533 / 0.505 | 0.664 | 6.57 | 5.59 | 300/354 |
| Gemini Pro | 0.532 / 0.566 | 0.684 | 8.80 | 4.56 | 314/354 |
| Gemini Flash-Lite | 0.443 / 0.324 | 0.392 | 7.88 | 9.87 | 275/354 |
| GPT-5.6 Luna | 0.283 / 0.271 | 0.314 | 14.91 | 5.68 | 305/354 |
| Claude Haiku 4.5 | 0.022 / 0.018 | 0.019 | 19.06 | 9.70 | 302/354 |
| Claude Opus 4.8 | 0.035 / 0.033 | 0.037 | 20.84 | 6.20 | 302/354 |
| Claude Sonnet 5 | 0.030 / 0.022 | 0.024 | 16.24 | 10.13 | 286/354 |
The interactive leaderboard adds per-density breakdowns and both IoU thresholds on every sub-metric.
Latency: the VLM runs at ≈2.2 s/page on a single A100 with vLLM (constrained JSON decoding); the hybrid adds the detector pass (≈2.5 s/page). Cloud systems are labeled with the generation returned by the evaluated batch; provider batch APIs do not provide comparable per-page latency.
Output
[
{
"box": [82, 164, 418, 205],
"type": "text",
"label": "Account number",
"description": "The applicant's bank account number.",
"value": "0042 1195 88",
"group_id": null
}
]
Coordinates are [x0, y0, x1, y1] on a 0–1000 page grid. Types are text, choice_checkbox,
choice_radio, choice_select, and signature. For choice fields the value carries the state
(checked / unchecked) or the selected option.
Intended use and limits
- Empty, filled, and handwritten forms are supported.
- Multilingual forms, field labels, and values are supported.
- Very dense pages can require longer generation budgets; the optional hybrid improves localization.
descriptionis a natural-language field summary — a convenience output, not a ranked metric.
License and attribution
The Nutrient VLM weights are offered under a commercial Nutrient license. The evaluation set is public and
reproducible through
nutrientdocs/form-field-vlm-v2-benchmark.
The hybrid uses the Nutrient form-field-v1-medium
RF-DETR detector for box localization only. Benchmark pages derive from CommonForms, CC-BY-4.0.
📩 Get access
form-field-vlm-v2is commercial and its weights are not downloadable here. To evaluate or deploy it on-prem, with documents remaining in your infrastructure, contact Nutrient: nutrient.io/contact-sales.
About the author
This project is maintained and funded by Nutrient - The deterministic document infrastructure enterprises run their highest-stakes workflows on: replayable output, clear exceptions, and full audit trails on the messy, regulated documents where AI alone breaks.