webbrain-one/minicpmv47-webbrain-v1

MiniCPM-V-4.7 WebBrain v1

A WebBrain fine-tune of MiniCPM-V-4.7-1B, built to be one on-device model for three jobs in WebBrain:

• page-state decisions (verify protocol plus probe)
• the local visual fallback (screenshot observations and grounding)
• the text and tool-calling model for offline mode

This repository holds the merged float32 weights: the LoRA folded into the base, with the fine-tuned merger included. Companion repositories:

• webbrain-one/minicpmv47-webbrain-v1-lora: the LoRA adapter and the merger weights.
• webbrain-one/minicpmv47-webbrain-v1-onnx: the WebGPU ONNX bundle (q8 text graphs, fp16 vision encoder, JS runtime and decision probes).

Training

• Base: openbmb/MiniCPM-V-4.7-1B (revision 379f120c2ff3588e6e06559ec6b9ec15cc05114e): SigLIP vision tower, a 16x vision-to-language merger, and a Qwen3.5-0.8B-style language model (18 Gated DeltaNet + 6 attention layers).
• Data: one pass over 344,139 samples.
  • WebBrain-VL-4 (the VL-3 draw schedule, 335,572 draws): VL-2 six-section page observations, curated VL-3 observations, MolmoPoint GUI grounding tasks (intent → point/box, point/box → element, grounded interaction, actionable inventory) and WebChain trajectory steps.
  • The WebBrain Compass v3 maximal text release (8,583 rows): ask, compact browser actions, WebLINX-derived compact actions, escalation.
• Method: LoRA r = 64, alpha = 64 on all 186 language-model linear layers (43.3M parameters, fp32 adapters) plus a full fine-tune of the merger (26.0M, fp32). The vision tower is frozen.
  • AdamW: LoRA 1e-4, merger 2e-5. 3% warmup, cosine decay to 10%, effective batch 64 (5,144 optimizer steps).
  • Loss is the per-sample mean over completion tokens only, with a 32K maximum sequence length.
  • Thinking is off: the model's chat template with enable_thinking=False.
• Compute: one RTX 5090, about 15 hours.
• Final validation loss (full sets): VL 0.295 (VL-2 0.216, curated VL-3 0.542, MolmoPoint 0.498); Compass 1.138.
• Held out: d1's balanced-V3 final test split was never used for training or model selection.

Results

Visual fallback: WebBrain VL-3 frozen validation, same 1,344 rows

Greedy generation, max_new_tokens = max(128, expected + 96), scored with the VL-3 pipeline's own scorer (run_webbrain_vl3_gpu_pipeline._score_generation). The comparison is WebBrain's current visual fallback, LFM2.5-VL-450M fine-tuned on VL-3 (selected checkpoint, step 3750), on exactly the same rows: every MolmoPoint grounding row except actionable inventory (first 100 of 250), every curated VL-3 row, every 10th VL-2 row.

Metric	MiniCPM-V WebBrain v1	LFM2.5-VL-450M (VL-3)
Schema-valid outputs	90.1%	85.5%
Point within 0.05 (510 rows)	58.8%	45.9%
Grounded interaction: point within 0.05 (255)	51.0%	29.8%
Grounded interaction: mean point distance	0.139	0.189
Box IoU	0.113	0.050
Point inside the target box	23.3%	10.4%
Six-section observation valid, VL-2 / VL-3	99.7% / 69.3%	97.1% / 44.6%
Actionable inventory valid (100)	19%	10%

On intent → point/box the mean point distance is worse (0.248 vs 0.132) although accuracy within 0.05 is higher (66.7% vs 62.1%): a minority of far misses. Long inventory lists often do not terminate for either model.

Text and tool calls: WebBrain test/llm routing benchmark (compact tier)

Same suite and grader as the Compass Tiny cards (run-llamacpp.mjs first turns, run-scenarios.mjs scenarios), served from PyTorch BF16 with seed 3407 and thinking off. The 89 scenarios are the scored ones shared with the Compass Tiny v2.1 run; strict = ideal, loose = ideal + ideal_name.

Model	First-turn structured calls	Strict exact action	Loose tool family	Discouraged actions
MiniCPM-V WebBrain v1 (0.8B language model)	97/100	10/89	38/89	5
MiniCPM-V-4.7-1B, untuned	98/100	3/89	30/89	2
Compass Tiny v2.1 (2.6B, WebGPU q4f16)	98/100	15/89	43/89	3
Compass Tiny XS v3 (1.7B, BF16, from its card)	97/100	9/89	39/89	n/a

Page-state decisions: d1 balanced-V3 final test (126 categorical questions)

WebBrain's verify protocol (one Yes/No continuation per option on a shared prefill) plus a linear probe on the margin and decision_hidden, fit on d1's train split with its strength chosen on validation; the final test is never trained on. The conservative probe weighs a false "completed" four times in its fit.

Model	Correct	Completions found	False completions	Unknown recognized	Time per request
v1, WebGPU q8, probe	103/126	22/24	11/48	13/24	0.85 s (RTX 5090)
v1, WebGPU q8, conservative probe	101/126	14/24	4/48	20/24	0.85 s
MiniCPM-V-4.7-1B untuned, WebGPU q8, probe	104/126	23/24	8/48	16/24	n/a
Bonsai 2 2B Vision + probe (reference)	110/126	21/24	4/48	20/24	2.2 s
d1-omni-600M (reference)	92/126	n/a	1/48	n/a	0.75 s

The fine-tune did not improve decisions: its data has no decision-format examples.

Usage

import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

repo = "webbrain-one/minicpmv47-webbrain-v1"
processor = AutoProcessor.from_pretrained(repo)
model = AutoModelForImageTextToText.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda").eval()

messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Describe this page."}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False, tokenize=False)
inputs = processor(text=[prompt], images=[Image.open("screenshot.png").convert("RGB")], return_tensors="pt").to(model.device)
out = model.generate(**inputs, do_sample=False, max_new_tokens=512)
print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Tool calls use the chat template's native format: <tool_call>, then <function=NAME>, then one <parameter=KEY> block per argument. Pass OpenAI-style tools= to apply_chat_template, and parse the reply with a Qwen3.5 / Qwen3-coder style tool parser that types each argument from the tool's JSON schema.

License

The base model is Apache-2.0. This fine-tune was trained on data that includes noncommercial-licensed sources (for example the WebLINX-derived Compass subset), so the fine-tuned weights are released for noncommercial research only under the WebBrain noncommercial research restrictions. Private hosting does not expand those rights. Model output is never proof that a browser action succeeded. Keep WebBrain's schema validation, grounding checks and confirmations in front of any action.

Training data includes WebChain v2 (webagentlab/webchain, WebChain Academic Research License v1.0): Fan, Sicheng, Rui Wan, Yifei Leng, Gaoning Liang, Li Ling, Yanyi Shang, and Dehan Kong. "WebChain: A Large-Scale Human-Annotated Dataset of Real-World Web Interaction Traces." arXiv:2603.05295, 2026. It also includes WebLINX (McGill-NLP, CC BY-NC-SA 4.0) and MolmoPoint-GUISyn (Ai2, ODC-BY).