webbrain-one/webbrain-compass-vld-tiny-v5-onnx

WebBrain Compass VLD Tiny v5: WebGPU ONNX

This is the in-browser build of webbrain-one/webbrain-compass-vld-tiny-v5. One model covers WebBrain's page-state decisions, the local visual fallback and offline text and tool calls, without loading or unloading models between them. On d1's final test it answers 120 of 126 decision questions with no false completions, in 0.94 s per request on an RTX 5090.

Files

File	Size	Role
onnx/vision_encoder_fp16.onnx (+_data)	1.10 GB	SigLIP and the fine-tuned merger. Input pixel_values float32[num_crops,3,14h,14w]; output image_features float32[num_crops,(h/4)(w/4),1024]
onnx/decoder_model_merged_q8f16.onnx (+_data)	0.83 GB	Recommended language model: MatMulNBits 8-bit (block 32), fp16 activations and KV cache. Chunked Gated DeltaNet prefill. Outputs last-position logits and decision_hidden
onnx/embed_tokens_q8f16.onnx (+_data)	0.28 GB	Token embeddings, 8-bit
onnx/decoder_model_merged_q4f16.onnx (+_data)	0.44 GB	Lite: the same language model with 4-bit weights
onnx/embed_tokens_q4f16.onnx (+_data)	0.15 GB	Lite: token embeddings, 4-bit
runtime/minicpmv-runtime.mjs		Processor port (PIL bicubic, MiniCPM-V slicing, image placeholders), canvas M-RoPE positions, sessions, prefill/forward with cache reuse, greedy generate
runtime/minicpmv-decisions.mjs		WebBrain's decision protocol (drop-in next to src/chrome/src/offscreen/webgpu-decisions.js)
runtime/minicpmv-decision-probe-ft2.js, ...-ft2-cs.js		Optional decision probes: standard, and conservative on false completions
runtime/webgpu-check.mjs, runtime/decision-check.mjs		Parity and end-to-end decision checks
validation/		Export report, training eval, decision, VL and routing results

config.json names the graph sets under webbrain_runtime: recommended (q8) and lite (q4). The download is about 2.2 GB with the q8 graphs, or 1.7 GB with the lite graphs.

• Lite (q4) graphs: Round 1's 4-bit build failed the quality bar (80/126 decisions, 25/48 false completions). This build's 4-bit graphs match q8 on d1 decisions: 120/126, 0/48 false completions. Text and visual generation were not validated on q4, so prefer q8 when the model generates.
• Tested with: Transformers.js 4.2.0 (tokenizer), onnxruntime-web 1.27.0 (WebGPU EP), Chromium, an RTX 5090.
• Low-end GPUs are not a target: on a Quadro T400 (4 GB), one decision request takes about 58 s, 45 s of it in the vision encoder.

Usage

import * as ort from './ort.webgpu.mjs';
import { AutoTokenizer } from './transformers.web.js';
import { MINICPMV, MiniCPMVRuntime, preprocessImage } from './runtime/minicpmv-runtime.mjs';
import { runMiniCPMVDecision } from './runtime/minicpmv-decisions.mjs';

const tokenizer = await AutoTokenizer.from_pretrained(BUNDLE_URL);
const tok = { encode: text => tokenizer.encode(text, { add_special_tokens: false }) };
const rt = await MiniCPMVRuntime.create(ort, tok, {      // sessions are created one at a time
  embed: `${BUNDLE_URL}/onnx/embed_tokens_q8f16.onnx`,      // lite: embed_tokens_q4f16.onnx
  vision: `${BUNDLE_URL}/onnx/vision_encoder_fp16.onnx`,
  decoder: `${BUNDLE_URL}/onnx/decoder_model_merged_q8f16.onnx`,   // lite: decoder_model_merged_q4f16.onnx
});
// decisions: images = [{ rgb: Uint8Array, width, height }], entries = [[id, d1-style question], ...]
// Without a probe each option is scored by its raw Yes − No margin. To use a probe, import minicpmvProbeScores
// from './runtime/minicpmv-decision-probe-ft2.js' and pass probe: { scores: minicpmvProbeScores }.
const { answers } = await runMiniCPMVDecision(rt, { state, images, entries });
// text or visual generation (greedy): a rendered chat prompt, one image placeholder per preprocessed image
const prompt = `<|im_start|>user\n${MINICPMV.imagePad}\nDescribe this page.<|im_end|>\n`
  + '<|im_start|>assistant\n<think>\n\n</think>\n\n';
const { ids } = await rt.generate(prompt, [preprocessImage(rgb, width, height)], { maxNewTokens: 256 });
const text = tokenizer.decode(ids, { skip_special_tokens: true });

Validation

These checks use the d1 balanced-V3 final test and WebBrain's own prompt helpers.

• Processor: token ids and canvas positions are exact against the PyTorch processor. Pixels differ by at most 2/255 on about 0.01% of values.
• WebGPU q8 vs PyTorch (144 questions):
  • Argmax agreement is 98.6% (2 flips).
  • The median Yes − No margin difference is 0.38 (mean 0.76).
  • The median decision_hidden cosine is 0.999.
  • The tails are wider than round 1's (worst margin difference 1.06, minimum cosine 0.990). 12 of 144 questions differ by more than 2 logits. One two-option auxiliary question flips outright: margins +4.0 / −7.4 in WebGPU against −7.4 / +5.5 in PyTorch, hidden cosine 0.85.
  • End to end, the q8 build scores 120/126 without a probe, against 121/126 in PyTorch.
• End-to-end decisions (RTX 5090, one request = a screenshot plus its questions):
  • q8 with the standard probe: 120/126 correct, 24/24 completions found, 0/48 false completions, 24/24 unknown recognized. The conservative probe and the q4 lite graphs score the same.
  • Time: 0.94 s per request on average (vision 0.28 s, prefill 0.12 s, questions 0.54 s). Model load takes 9.1 s (7.8 s for lite).
• Generation smoke test (q8, greedy): a 31-token page description in 1.7 s.

Results

Page-state decisions: d1 balanced-V3 final test (126 categorical questions)

Scored through WebBrain's decision protocol in the browser (WebGPU) on an RTX 5090. One request is a screenshot, its page state and its questions. The d1 final test split was never trained on.

Model	Correct	Completions found	False completions	Unknown recognized	Time per request
v5, WebGPU q8 + probe	120/126	24/24	0/48	24/24	0.94 s
v5, WebGPU q4 (lite) + probe	120/126	24/24	0/48	24/24	0.94 s
Round 1, WebGPU q8 + probe	103/126	22/24	11/48	13/24	0.85 s
MiniCPM-V-4.7-1B untuned, WebGPU q8 + probe	104/126	23/24	8/48	16/24	n/a
Bonsai 2 2B Vision + probe	110/126	21/24	4/48	20/24	2.2 s
d1-omni-600M	92/126	n/a	1/48	n/a	0.75 s

• The probe is no longer needed. Without it, v5's raw Yes − No verify margins score 121/126 in PyTorch and 120/126 in WebGPU q8. In PyTorch the standard and conservative probes (fit on d1 train, chosen on validation) also score 121/126, and the generated option readout alone scores 120/126.
• A caution on scope: d1 is a synthetic benchmark. v5 was fine-tuned on d1's train families, which are disjoint from the test families but follow the same design. Bonsai's backbone was not trained on d1, only its probe was. Validate v5 on real extension decisions before relying on these margins.

Visual fallback: WebBrain VL-3 frozen validation, same 1,022 rows

Greedy generation, scored with the VL-3 pipeline's own scorer. The rows are every MolmoPoint grounding row except actionable inventory (first 20), the first 80 curated VL-3 rows and every 20th VL-2 row. LFM2.5-VL-450M (VL-3, step 3750) is WebBrain's current visual fallback.

Metric	v5	Round 1	LFM2.5-VL-450M
Schema-valid outputs	96.2%	95.8%	92.7%
Point within 0.05 (510 rows)	65.5%	58.8%	45.9%
Mean point distance	0.156	0.194	0.161
Grounded interaction: point within 0.05 (255)	57.3%	51.0%	29.8%
Intent → point/box: point within 0.05 (255)	73.7%	66.7%	62.1%
Box IoU	0.175	0.113	0.050
Point inside the target box	33.3%	23.3%	10.4%
Six-section observation valid, VL-2 / VL-3	98.7% / 72.5%	99.4% / 65.0%	97.5% / 38.8%
Actionable inventory valid (first 20)	25%	30%	10%

On intent → point/box the mean point distance is still worse than LFM's (0.190 vs 0.132; round 1 0.248), although more answers land within 0.05: a minority of far misses. Long inventory lists often do not terminate, for every model.

Text and tool calls: WebBrain test/llm routing benchmark (compact tier)

Served from PyTorch BF16 with seed 3407 and thinking off. The table counts the 89 scenarios scored in the Compass Tiny v2.1 run: strict = ideal, loose = ideal + ideal_name.

Model	First-turn structured calls	Strict exact action	Loose tool family	Discouraged actions
v5 (0.8B language model)	95/100	9/89	48/89	3
Round 1	97/100	10/89	38/89	5
MiniCPM-V-4.7-1B, untuned	98/100	3/89	30/89	2
Compass Tiny v2.1 (2.6B, WebGPU q4f16)	98/100	15/89	43/89	3

License

The base model is Apache-2.0. This fine-tune was trained on data that includes noncommercial-licensed sources (for example the WebLINX-derived Compass subset), so the fine-tuned weights are released for noncommercial research only under the WebBrain noncommercial research restrictions. Private hosting does not expand those rights. Model output is never proof that a browser action succeeded. Keep WebBrain's schema validation, grounding checks and confirmations in front of any action.

Training data includes WebChain v2 (webagentlab/webchain, WebChain Academic Research License v1.0): Fan, Sicheng, Rui Wan, Yifei Leng, Gaoning Liang, Li Ling, Yanyi Shang, and Dehan Kong. "WebChain: A Large-Scale Human-Annotated Dataset of Real-World Web Interaction Traces." arXiv:2603.05295, 2026. It also includes WebLINX (McGill-NLP, CC BY-NC-SA 4.0) and MolmoPoint-GUISyn (Ai2, ODC-BY).