merve/muse-glimmer-ft-clicking-v2

Muse-Glimmer-30B — QLoRA click grounding (v2)

A QLoRA adapter that teaches meta-models/Muse-Glimmer-30B to point at UI elements in web screenshots: given a screenshot and an instruction like "filter by MATEIN brand", it returns a click point as a percentage of image width/height.

The base model is normally sota on ScreenSpot-Pro but on MolmoWeb dataset the prompts are a bit vague so fine-tuning improves the performance on it. We ran these examples across 100 images and compared model fine-tuned on MolmoWeb format against base model zero-shot outputs converted to MolmoWeb format. We found that base model has 13% click accuracy + 35.0% within 5% diagonal while fine-tuned model has 41% + 68% within 5% diagonal.

The out-of-range row is the other half of what fine-tuning bought. The base model writes valid JSON every time, so a parse-rate check makes it look compliant — but 60% of its coordinates fall outside the 0-100 range the prompt asks for, with y values as high as 970 on screenshots 712 pixels tall. Those numbers do not correspond to original-image pixels or to the resized canvas the model was shown, and no rescaling rescues them: scored as percentages they give 4.0%, divided by 1000 6.3%, taken as raw pixels 4.0%. This adapter puts every one of 300 answers in range, and stops narrating: 290 generated tokens down to 99.

Distance bands are reported because strict in-box accuracy alone is misleading in both directions on this task. A 20%-of-diagonal band is a ~294px radius on these screenshots — about 15× wider than a typical target — so a high band rate can coexist with near-zero in-box hits. Read the bands as "roughly the right region", and only the first row as "would actually click the right thing".

Usage

import json, torch
from transformers import AutoModelForMultimodalLM, AutoProcessor, BitsAndBytesConfig
from peft import PeftModel

BASE = "meta-models/Muse-Glimmer-30B"
processor = AutoProcessor.from_pretrained(BASE)
model = AutoModelForMultimodalLM.from_pretrained(
    BASE,
    dtype=torch.bfloat16,
    device_map="auto",
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_compute_dtype=torch.bfloat16,
        bnb_4bit_use_double_quant=True,
        bnb_4bit_quant_type="nf4",
        # the vision tower shares weight names with the LLM; quantizing it degrades grounding
        llm_int8_skip_modules=["model.vision_tower", "model.vision_adapter", "lm_head"],
    ),
)
model = PeftModel.from_pretrained(model, "merve/muse-glimmer-ft-clicking-v2")
model.eval()

PROMPT = (
    "You are looking at a screenshot of a webpage. Follow this instruction: {question}\n\n"
    "Respond with a single JSON object of the form "
    '{{"thought": "<brief reasoning>", "action": {{"name": "click", "button": "left", '
    '"click_type": "single", "x": <0-100>, "y": <0-100>}}}}, '
    "where x and y are the click point as a percentage of the image width and height. "
    "Output only the JSON object."
)

def click(image, question):
    small = image.convert("RGB")
    small.thumbnail((512, 512))          # trained at max side 512; percentages are resize-invariant
    messages = [{"role": "user", "content": [
        {"type": "image", "image": small},
        {"type": "text", "text": PROMPT.format(question=question)},
    ]}]
    text = processor.apply_chat_template(
        messages, add_generation_prompt=True, tokenize=False, reasoning_strength="low"
    )
    inputs = processor(images=[[small]], text=[text], return_tensors="pt",
                       add_special_tokens=False).to(model.device)
    with torch.inference_mode():
        out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
    generated = out[0, inputs["input_ids"].shape[-1]:].tolist()

    # Read ONLY the `to=user` channel. The model also emits a `to=self` reasoning channel,
    # and numbers scraped from that are coordinates it already discarded. The tokenizer
    # carries a `response_template`, so let transformers do the parsing; `prefix` is the
    # assistant header the chat template pre-writes before generation starts.
    tok = processor.tokenizer
    answer = tok.parse_response(
        generated, prefix=tok("<|start|>assistant", add_special_tokens=False)["input_ids"]
    )["content"]
    action = json.loads(answer)["action"]
    return action["x"] / 100 * image.width, action["y"] / 100 * image.height  # original pixels

Two things will silently ruin your results if you skip them: call .eval() (in train mode LoRA dropout stays active and generation degrades badly), and parse only from the to=user channel.

Training

QLoRA on a frozen 4-bit NF4 base, bf16 compute, vision tower left unquantized and untrained.

LoRA	r=16, alpha=32, dropout 0.05, on q_proj/k_proj/v_proj/o_proj, exclude_modules=".*vision_tower.*"
trainable params	29,392,896 (0.099% of 29.8B)
data	22,364 click messages from 4,500 screenshots (≤5 instructions per screenshot)
schedule	350 steps × effective batch 64 (16 × 4 accum) ≈ 1 epoch, lr 1e-4, 7 warmup steps, adamw_8bit
loss	completion-only — masked to the assistant's answer, so image tokens are not loss targets
final train loss	0.529 (last logged 0.43, token accuracy 85.7%)
hardware	1×H200, 68 min

Two details mattered more than they look:

Completion-only loss is not optional. Without masking the prompt, image tokens make up the majority of the loss targets and the fine-tune degrades instead of improving.

This is one epoch, and that is the whole story of the result. An earlier version of this run used effective batch 16 for 500 steps — 8,000 samples, 0.36 of an epoch — over the same data pool, and scored 25.0%. Finishing the epoch at effective batch 64 with 2× the learning rate took it to 42.0% (+16pp on identical eval examples). The limit was undertraining, not data volume; a 95K-example pool (lifting the ≤5-per-screenshot cap) was tried and abandoned as not obviously worth 5× the compute.

Limitations

• 42% is not a usable click agent on its own. Strict in-box accuracy on small targets is hard; budget for verification or retries downstream.
• Trained only on MolmoWeb-mini desktop web screenshots at ≤512px max side. Mobile layouts, desktop applications, dense spreadsheets, and full-resolution inputs are out of distribution.
• Single-click grounding only. No typing, scrolling, dragging, or multi-step trajectories.
• Coordinates are percentages in [0, 100], not pixels. Multiply by the original image size.
• N=300 eval, so the 42.0% figure carries roughly ±5.6pp at 95% confidence.
• Zero-shot greedy decodes on this base model are not reproducible across GPU types — on identical inputs, only 17/100 baseline predictions matched between an A100 and an H200, because ~290 tokens of reasoning amplify small numeric differences. Compare a baseline and a fine-tune only when both were measured on the same hardware.

Provenance

Trained with TRL SFTTrainer and PEFT on Hugging Face Jobs. Supersedes merve/muse-glimmer-ft-clicking, whose reported score came from an eval pool that overlapped its own training images (an unseeded train_test_split); on a clean split that adapter scores 25.0%.