senpaicq/m2w-gemma4-e2b-pointer

Gemma 4 E2B + LoRA pointer head for web element selection (Multimodal-Mind2Web)

A small, fast "which element do I click next?" model with a confidence score. Research release, not a ready-to-use web agent.

Given a task, the previous actions, a full-page screenshot and a list of candidate DOM elements, the model points at one candidate (or at a learned "none of these" slot) and predicts the operation type (click / type / select). It does not generate the text to type.

• Backbone: google/gemma-4-E2B (frozen) + LoRA (rank 32, attention + MLP, layers 0-29, 180 modules) + a small pointer head trained from scratch.
• Trained on Multimodal-Mind2Web for about 8 hours on one A100-40GB.
• Speed: in the live form test below (short pages, one screenshot strip, about 33 candidates) a decision took about 0.2 s for the model plus 0.1 s for candidate extraction and screenshot on an A100. Inputs of Mind2Web size (several strips, up to 4600 tokens) are slower; we did not benchmark that separately.
• Honest summary of what it is good and bad at is in Summary of results and Limitations.

Role: a fast decision module ("System 1"). The model is meant as the quick, cheap first decision in a larger agent: it picks the next element or abstains with a confidence score, and low-confidence steps are escalated to a stronger model ("System 2", for example a large VLM) or a human. It is a decision/routing component, not a standalone agent.

Approach: a calibrated decision head, "Kev-like". Instead of generating the next action as text, the model makes a decision over a closed set: a query vector (from the task, history and screenshot) is scored against one key per candidate element plus a learned "none of these" key, and a softmax gives a probability for every option. This pointer-style, "Kev-like" decision head has three consequences we care about: (1) one forward pass picks the element, no token-by-token decoding; (2) the top-1 probability is a ready-made confidence score, and "none" is an explicit option, so the model can abstain; (3) because decisions and confidences are explicit numbers, they can be evaluated and calibrated with standard tools (ECE, Brier score, risk-coverage curves, temperature scaling) and used to route uncertain steps to a stronger model. In this release the decision quality and the confidence ranking are measured (see results); the confidences themselves are still raw and need temperature scaling on held-out data before being read as probabilities. The "Kev-like" design follows Kev by Jared Palmer (open-source family of "System One" decision models: a small LoRA adapter plus a pointer head on a Qwen base, weights on Hugging Face, Apache-2.0), which in turn is described as Jev-like; see Related work.

Status: single training run, single seed. Confidence scores are raw (not temperature-scaled) for this checkpoint. Numbers are not directly comparable with published Mind2Web numbers (see Evaluation protocol).

Summary of results

Element accuracy on Multimodal-Mind2Web test splits (n = steps; 95% bootstrap CI). A step counts as correct if the top-1 candidate is a ground-truth element. A "none" prediction counts as an error.

Split	n	Accuracy (CI95)	Accuracy if the answer is in the list	Filter ceiling	ECE (raw)	AUROC of confidence	Acc. on most confident 20%
test_task	1339	0.465 [0.439, 0.493]	0.556	0.837	0.189	0.710	0.761
test_website	1019	0.384 [0.355, 0.413]	0.483	0.795	0.189	0.698	0.662
test_domain	4060	0.385 [0.370, 0.400]	0.456	0.845	0.199	0.737	0.703

"Filter ceiling" is the fraction of steps in which our rule-based candidate filter keeps a correct element at all (see below). Nobody can score above it with this candidate list. Validation (579 steps carved from the training websites): accuracy 0.368 [0.326, 0.408].

Progression across our own runs (same candidate filter, same splits):

Run	test_task	test_website	test_domain
First text-only run, LoRA r16 attention	0.375	0.275	0.308
+ full-page screenshot, LoRA r16 attention, 24-token candidate cap	0.416	0.317	0.351
This model: screenshot, LoRA r32 attention+MLP, tree-format candidates, gold candidate always present in training	0.465	0.384	0.385

Risk-coverage (accuracy if the model only answers its most confident x% of steps; raw confidences):

Split	100%	80%	60%	40%	20%
test_task	0.465	0.515	0.583	0.660	0.761
test_website	0.384	0.426	0.471	0.574	0.662
test_domain	0.385	0.445	0.513	0.591	0.703

Other metrics (test_task / test_website / test_domain): operation-type accuracy 0.881 / 0.866 / 0.864; Brier 0.256 / 0.249 / 0.240; fraction of steps where the model answers "none" 0.120 / 0.119 / 0.092; "none" predicted when no correct element is in the list (abstention success) 0.367 / 0.373 / 0.237.

Take-aways

1. The screenshot helps, and the gain is driven by the pixels. In an ablation on the earlier vision run (not repeated for this checkpoint), replacing the screenshot with a gray image or another page removed the gain (paired CI of the gain over the text-only model: +3.2 to +5.4 points).
2. The confidence score carries information: accuracy rises from 0.38-0.47 on all steps to 0.66-0.76 on the most confident 20%. This is the main reason to use this model (selective prediction, routing hard cases to a stronger model), not its raw accuracy.
3. Raw confidences are over-confident (ECE about 0.19-0.20). On the earlier vision model temperature scaling fitted on validation reduced ECE to roughly 0.05-0.09; we have not measured this for this checkpoint. Fit a temperature on your own held-out data before using the numbers as probabilities.
4. Absolute accuracy is modest. About 15-20% of steps lose the correct element in our candidate filter, and even when the correct element is in the list the model finds it only 46-56% of the time.

Intended use

• Research on small, fast GUI/web element grounding with calibrated abstention.
• A cheap first-pass element selector whose low-confidence steps are escalated to a larger model or a human (cascade/routing experiments).
• A baseline and a set of documented negative results for pointer-style heads on top of a small VLM.

Model details

Architecture. The full Gemma 4 E2B model (35 layers, hidden size 1536) runs on the whole sequence; we read the hidden states of layer 30 and attach:

• a query from the last token (LayerNorm -> linear, d_k = 256),
• one key per candidate from the mean of the candidate's token hidden states (LayerNorm -> linear),
• a learned "none" key, so the softmax is over n_candidates + 1 slots,
• an operation head (linear, 3 classes: CLICK / TYPE / SELECT) on the query.

LoRA is applied to layers 0-29 (q/k/v/o and gate/up/down projections where they exist; the last 20 layers of Gemma 4 E2B share KV and have no k/v projections), rank 32, alpha 64, dropout 0.05. The vision tower and the language model weights are frozen. Hidden states of layers 30-34 are not used by the head (they are still computed in the forward pass; truncating the model at layer 30 would be an easy speed-up we did not implement).

Input. [BOS] [screenshot strips] Task: ... Previous actions: ... Candidates: - <candidate 1> - <candidate 2> ... Task: ... Previous actions: ... Next element to interact with:

• Screenshot: the full-page screenshot is cut top-to-bottom into 1280x1280 strips (a trailing piece under 200 px is merged), at most 5 strips (6400 px), 280 vision tokens per strip. About 19% of Mind2Web pages are taller than that and are truncated at the bottom. No target-dependent cropping is used (it would leak the answer).
• Candidates: text only, in "tree" format: role "accessible-ish name" [state], for example textbox "Where from?", link "Flights", button "Search". At most 64 tokens per candidate and 4600 tokens in total per step (candidate lines are shortened uniformly when a step is too long). The model is not given element positions, so it must match candidate text to what it sees in the screenshot.
• Candidate list: Multimodal-Mind2Web provides 331-743 candidates per step. We order them by TF-IDF similarity to the task and recent actions, then apply a rule filter (visible, interactive tags first, then text-bearing common tags) and keep at most K = 160.

Loss. Multi-positive softmax cross-entropy over candidates plus "none" (target is "none" if no correct candidate is in the list), plus 0.3 x cross-entropy for the operation type.

Training

Data	Multimodal-Mind2Web train (7775 steps): 7196 for training, 579 for validation (validation = websites whose crc32(website) % 8 == 0)
Candidate handling in training	rule filter K = 160, then the dropped correct candidate is added back to the list (--always_pos). Validation and test lists are never modified
Epochs	4 run, checkpoint of epoch index 2 (third epoch) kept (best validation accuracy: 0.249 / 0.363 / 0.368 / 0.358; ECE rose 0.119 -> 0.164 -> 0.260 -> 0.362, i.e. over-fitting in the last epoch)
Optimiser	AdamW, LoRA lr 2e-4, head lr 1e-3, cosine schedule, 3% warm-up, gradient clipping 1.0, micro-batch 1 with accumulation 8, bf16
Candidate order	random permutation in training; original TF-IDF order at test time
Hardware / time	1x A100-SXM4-40GB, 29,604 s for 4 epochs (about 1.03 s per example), peak memory 38.6 GB
Skipped	4 training steps skipped for out-of-memory (inputs of about 13.8k tokens); 11 steps without a screenshot
Seeds	one (seed 0)
Base model revision	google/gemma-4-E2B @ d29ff6b45f081a49ee2733a859c9c9c2d95d1a6f

Before this run we swept LoRA capacity, learning rate, candidate text format, and the query layer on a 1500-example, 2-epoch text-only setup (val 579 steps, single seed): rank-32 attention+MLP was the only clear winner (about +6.6 points on validation over rank-16 attention-only); rank 64, lr 1e-4 and 4e-4, and query layer 35 were not better; the tree format was as accurate as the long attribute format and shorter; a 24-token candidate cap did not measurably hurt in the short regime. These differences are from one seed and a small validation set, treat them as hints.

Evaluation protocol

• Splits: the official Multimodal-Mind2Web test_task, test_website, test_domain.
• Metric: top-1 element accuracy over our candidate list (K = 160, rule filter). This is not the official candidate list (the Mind2Web paper uses a DeBERTa ranker's top-50), and steps with no correct element in the list count as errors. Our numbers must not be compared with published element accuracy without this caveat; we make no claim of parity or superiority with any published system.
• "Accuracy if the answer is in the list" is conditional on the correct element being among the candidates and is higher by construction; it should not be compared with unconditional numbers.
• ECE uses 15 equal-width bins on the top-1 probability. AUROC measures how well the top-1 probability separates correct from incorrect steps.
• Bootstrap confidence intervals are over steps. Several steps come from the same task or website, so the true uncertainty is somewhat larger than the CIs suggest.

A stress test outside the training distribution: live form filling (negative result)

We ran the model in a real browser (Playwright) on 12 synthetic forms and, when reachable, 5 public practice pages (no submit buttons were pressed). One step per field; the field label and value come from the task sentence; the model only picks the element. We compare two candidate styles: as in the dataset (raw attributes, no <label> binding) and with labels (label text of the associated <label> added as the accessible name), and a trivial no-model baseline (TF-IDF match of the field label against candidate text; this baseline is told the target label directly).

Setting (86 steps, 17 forms)	Step accuracy (history = ground truth)	Fields correctly filled when the model drives (free run)	Forms fully correct
Gemma pointer, dataset-style candidates	0.36	0.29	2/17
Gemma pointer, label-bound candidates	0.55	0.49	3/17
Lexical-match baseline (knows the label)	0.78	0.70	6/17

So: this model is not a usable form filler. What we found:

• Candidate lines like textbox "fld3" (opaque id/name) cannot be matched with the visible label, because the model never sees element positions. Adding the label text gave +19 points. Placeholder-based forms (text is in the candidate) scored 0.77, forms whose label is an unbound <div> 0.21.
• By type: text 0.50-0.75, select 0.36-0.57, checkbox/radio close to 0.
• The model often picks button "Submit" for fields that are not done yet, with 96-99% confidence (a learned "what comes next" prior). Confidence ranking is still informative overall (label-bound free run: AUROC 0.88, 0.94 accuracy on the most confident 20%), but confidently wrong steps exist.
• The operation head dropped from 0.88 on Mind2Web to 0.56-0.63 here: the task phrasing (many fields in one sentence) and the pages differ from the training distribution.
• 86 steps is a small sample: differences of about 10 points are within noise.

Limitations and what this model is not

• Not an agent. It selects an element and an operation type. It does not produce typed text, does not plan, and does not recover from errors.
• Accuracy is modest (0.38-0.47 end-to-end on our lists) and below what larger systems report; the candidate filter alone caps it at 0.80-0.85.
• Over-confident out of the box (ECE about 0.19) and over-confident on some systematic mistakes (see the form test). Do not trigger irreversible actions (payments, deletions, sending messages) based on its confidence.
• No element positions in the input, full-page screenshots only (no viewport/scroll handling), pages truncated after 6400 px.
• Distribution: English web pages from the 2022-2023 Mind2Web snapshot; modern single-page apps, pop-ups, shadow DOM and iframes are not covered. Live DOM to tree conversion in code/ is an approximation of the dataset format (the dataset has almost no state attributes).
• One seed, one run. No ablation isolates the effect of always_pos, the tree format, and rank 32 in the final configuration separately.
• Possible contamination with the pre-training data of the Gemma backbone (web pages) cannot be ruled out.

How to use

Requirements: a GPU with bf16 support (inference memory was not measured; training peaked at 38.6 GB with gradients), torch, transformers (with Gemma 4 support), peft, playwright, lxml, scikit-learn, pillow. The code lives in code/; inference_example.py shows a complete loop.

import sys; sys.path.insert(0, "code")
from m2w_form_demo import Gemma, aday_listesi, softmax

model = Gemma("senpaicq/m2w-gemma4-e2b-pointer", shots_dir="tmp_shots")  # downloads adapter + head

# `page` is a Playwright page (viewport width 1280). Candidates are extracted from the live DOM:
cands, raw = aday_listesi(page, task="Find flights from NYC to Paris", history=[], arm="etiketli")
png = page.screenshot(full_page=True)
ex = {"task": "Find flights from NYC to Paris", "history": [], "cands": cands, "pos_idx": [], "op": "CLICK"}
logits, op_logits = model(ex, png, None)     # logits has n_candidates + 1 entries; the last one is "none"
p = softmax(logits, T=1.0)                   # T = 1.0 is raw; fit your own temperature on held-out data
j = int(p.argmax())
print("none" if j == len(cands) else cands[j]["text"], float(p[j]))

Files in this repository: adapter_model.safetensors and adapter_config.json (LoRA), head.pt (pointer and operation heads), m2w_config.json (inference settings), results/ (validation log, final metrics, per-step top-1 confidence and correctness for each split; no dataset content), code/ (data preparation, filtering, formatting, training, evaluation and live-demo scripts), inference_example.py.

Reproducing

The scripts in code/ reproduce the pipeline: m2w_prep.py (candidates, TF-IDF order) -> m2w_filter.py (K = 160) -> m2w_format.py --fmt tree -> m2w_ekran_goruntuleri_cikar.py (screenshots) -> m2w_train_lora_gorsel.py (arguments used: --layer 30 --epochs 4 --accum 8 --rank 32 --lora_mlp --max_cand_tok 64 --max_seq 4600 --img_tokens 280 --max_strips 5 --always_pos). Comments are partly in Turkish. The dataset is not redistributed here; download it from its original source.

License, data and attribution

• Our adapter weights and code are released under Apache-2.0. The base model google/gemma-4-E2B is listed as Apache-2.0 on its Hugging Face page; see its license terms.
• The model was trained on Multimodal-Mind2Web. We did not verify the dataset's license for this card; check the dataset page and terms before any commercial use. No dataset content is included in this repository.
• Screenshots and DOM snapshots in the dataset come from third-party websites.

Related work and credit

• Kev (Jared Palmer, 2026): open-source, Apache-2.0 family of small "System One" decision models that answer typed questions (choice, score, ...) with probabilities, built as a LoRA adapter plus a small pointer head on a Qwen base (code, weights). The idea of a LoRA adapter with a pointer head over a closed set of options, and of reading calibrated probabilities from it, comes from this line of work. We are not affiliated with it, and this model is not a Kev checkpoint.
• What is different here: a different backbone (Gemma 4 E2B, a vision-language model), screenshots as input, a task-specific candidate set built from the DOM (hundreds of web elements per step), a learned "none" key and an operation head, LoRA on attention and MLP projections of layers 0-29 (rank 32), and an evaluation on Multimodal-Mind2Web with calibration and risk-coverage metrics. We did not compare against Kev checkpoints.
• Benchmarks and data: Mind2Web and Multimodal-Mind2Web (see Citation).

Citation

If you use this model, please also cite the benchmark and the multimodal extension:

@inproceedings{deng2023mind2web,
  title     = {Mind2Web: Towards a Generalist Agent for the Web},
  author    = {Deng, Xiang and Gu, Yu and Zheng, Boyuan and Chen, Shijie and Stevens, Samuel and Wang, Boshi and Sun, Huan and Su, Yu},
  booktitle = {NeurIPS Datasets and Benchmarks Track},
  year      = {2023}
}
@inproceedings{zheng2024seeact,
  title     = {GPT-4V(ision) is a Generalist Web Agent, if Grounded},
  author    = {Zheng, Boyuan and Gou, Boyu and Kil, Jihyung and Sun, Huan and Su, Yu},
  booktitle = {ICML},
  year      = {2024}
}