emirozcan16/AA-VL-4B-adapter

AA-VL-4B — LoRA Adapter (GUI Grounding, 4B)

~7K curated training rows · ~$8 compute budget · 100% verifiable per-sample predictions included

🧠 Merged FP16 Repo · 📦 GGUF Quants (llama.cpp) · 🎯 Interactive OSWorld-G Prediction Viewer

PEFT LoRA adapter for AA-VL-4B: loads on top of Qwen/Qwen3-VL-4B-Instruct to map a screenshot + natural-language instruction → click x=<int> y=<int> (0–999 normalized grid) or refuse.

• Base: Qwen/Qwen3-VL-4B-Instruct
• LoRA Config: r=32, alpha=64, targets q/k/v/o/gate/up/down_proj (initial adapter trained in 4-bit QLoRA, final 250 steps in fp16)
• Merged FP16 Weights: emirozcan16/AA-VL-4B
• GGUF Quantizations: emirozcan16/AA-VL-4B-GGUF

Usage (peft)

from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel
from PIL import Image

BASE = "Qwen/Qwen3-VL-4B-Instruct"
proc = AutoProcessor.from_pretrained(BASE, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    BASE, trust_remote_code=True, torch_dtype="auto", device_map="auto"
)
model = PeftModel.from_pretrained(model, "emirozcan16/AA-VL-4B-adapter")

img = Image.open("screen.png").convert("RGB")
msgs = [{"role": "user", "content": [
    {"type": "image", "image": img},
    {"type": "text",
     "text": "Instruction: Click the funnel icon right of the search bar. "
             "Reply with the single action line now, nothing else:"}
]}]
txt = proc.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
inp = proc(text=[txt], images=[img], return_tensors="pt").to(model.device)
out = model.generate(**inp, max_new_tokens=64, do_sample=False)
print(proc.batch_decode(out[:, inp["input_ids"].shape[1]:], skip_special_tokens=True)[0])
# -> click x=<int> y=<int>   (0-999 normalized scale)   or   refuse

Benchmark Results & Reference Context

Scores below are from the fp16 merged weights (this unmerged adapter reproduces them within ±1 sample merge rounding: 369/564 and 419/564 on OSWorld-G):

Benchmark	AA-VL-4B (Ours, 4B)	Correct / Total	Reference Context (Published Numbers)
OSWorld-G (Original)	65.25%	368 / 564	Official Leaderboard: GUI-Owl-7B (55.9), UI-Venus-1.0-7B (54.6), Jedi-7B (54.1), UI-TARS-1.5-7B (52.8), Qwen2.5-VL-7B (31.4). Recent 8B reports: MAI-UI-8B (60.1), UI-Venus-1.5-8B (69.7).
OSWorld-G (Refined)	74.47%	420 / 564	Published 7B–8B reports: UI-Venus-1.0-7B (61.7), UI-Venus-1.5-8B (74.1).
UI-Vision (Final Pooled)	29.93%	1640 / 5479	Official Leaderboard: UI-TARS-72B (25.50). Self-reported ≤8B: UI-Venus-1.0-7B (26.5), Phi-Ground (27.2), MAI-UI-2B (30.3), MAI-UI-8B (40.7), UI-Venus-1.5-8B (46.5).

Verification

Per-sample re-scoring artifacts ship with the merged model repo (emirozcan16/AA-VL-4B): VERIFY.md, osworld_g_predictions.jsonl (1,128 rows) and predictions.jsonl (5,479 rows).

Project

Built in ~3 months on a ~$8 compute budget. Contact: ozcannemir08@gmail.com