imajev-2b-web-visual-testing 🎯

imajev-2b-web-visual-testing is a specialized, typed-decision Vision-Language Model fine-tuned for Automated Web Visual Regression Testing.

Unlike conventional VLMs that generate free-form text or unstable JSON (introducing latency and pipeline flakiness), this model operates as a single-forward-pass semantic classifier built on top of Qwen/Qwen3.5-2B using LoRA and a calibrated Float32 decision readout head (decision_readout.safetensors).

It accepts two screenshot images (Reference/Golden Baseline vs. Current Screen) along with live browser telemetry (app_state), evaluates the diff, and outputs calibrated probabilities across 5 regression categories plus an explicit unknown state for human review.

---

📊 Benchmark & Evaluation Results

Tested on a held-out, balanced 300-row test dataset (dataset-v2 test split across real-world web sources including Applitools & dynamic web apps, 50 rows per class):

Metric	Base imajev-2b	imajev-2b-web-visual-testing (Ours)
Overall Accuracy (All 300)	36.7%	78.7% (+42.0%)
Accuracy (250 Known Classes)	40.4%	79.2% (+38.8%)
Pass Recall	100% (biased)	100%
Layout Recall	10%	52%
Missing Recall	30%	86%
Styling Recall	0%	90%
Asset Recall	62%	68%
Unknown Recall (Needs Review)	18%	76%
Decided at $\ge 0.90$ Conf	5% (67% right)	52% (87.8% right)
Decided at $\ge 0.75$ Conf	36% (57% right)	76% (84.3% right)

Key Takeaway: The base model suffered from extreme confirmation bias, calling 174 of 300 samples "Pass" and missing most true regressions. This fine-tune eliminated that bias, rescued 129 rows missed by the base model, and enables safe automated gating on over 75% of CI/CD test runs.

---

🏗️ Architecture & Model Files

This repository contains LoRA adapter weights and custom readout files designed to be loaded over Qwen/Qwen3.5-2B:

• adapter_model.safetensors & adapter_config.json: LoRA weights (Rank 16, Alpha 32).
• decision_readout.safetensors: Custom linear readout layer mapping last hidden state to decision logits.
• calibration.json: Temperature scaling configuration (T = 1.2334) for probability calibration.

---

🚀 Quickstart & Inference

1. Requirements

pip install -q transformers peft safetensors pillow torch

2. Python Inference Pipeline

import json
import torch
import torch.nn as nn
import torch.nn.functional as F
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

REPO_ID = "kinasr/imajev-2b-web-visual-testing"
BASE_MODEL_ID = "Qwen/Qwen3.5-2B"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

# 1. Load Processor & Base Model
processor = AutoProcessor.from_pretrained(REPO_ID)
base_model = AutoModelForImageTextToText.from_pretrained(
    BASE_MODEL_ID,
    torch_dtype=torch.float16 if DEVICE != "cpu" else torch.float32,
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, REPO_ID).eval()

# 2. Load Decision Readout Head
readout_path = hf_hub_download(repo_id=REPO_ID, filename="decision_readout.safetensors")
readout_state = load_file(readout_path)
num_codes, hidden_dim = readout_state["weight"].shape
has_bias = "bias" in readout_state

decision_head = nn.Linear(hidden_dim, num_codes, bias=has_bias, dtype=torch.float32).to(DEVICE)
decision_head.weight.data.copy_(readout_state["weight"].to(torch.float32))
if has_bias:
    decision_head.bias.data.copy_(readout_state["bias"].to(torch.float32))
decision_head.eval()

# 3. Load Calibration Temperature
cal_path = hf_hub_download(repo_id=REPO_ID, filename="calibration.json")
with open(cal_path, "r") as f:
    temp = json.load(f).get("temperature", 1.2334)

# 4. Run Visual Regression Check
def classify_visual_diff(ref_image_path: str, curr_image_path: str, app_state: dict):
    options = [
        "Pass (No visual regression)",
        "Layout shift, overlap, or misalignment",
        "Missing interactive element or CTA",
        "Broken CSS styling or unreadable contrast",
        "Broken image, icon, or media asset"
    ]
    
    img_ref = Image.open(ref_image_path).convert("RGB")
    img_curr = Image.open(curr_image_path).convert("RGB")
    img_ref.thumbnail((640, 640))
    img_curr.thumbnail((640, 640))
    
    opts_str = "\n".join([f"[{i}] {opt}" for i, opt in enumerate(options)])
    messages = [
        {"role": "system", "content": [{"type": "text", "text": "You are a typed decision model for web visual regression testing. Output a calibrated decision over the provided options or unknown."}]},
        {"role": "user", "content": [
            {"type": "image"}, {"type": "image"},
            {"type": "text", "text": f"App State: {json.dumps(app_state)}\nQuestion: Compare the current screenshot against baseline reference and classify the regression.\nType: choice\nOptions:\n{opts_str}\nDecision:"}
        ]}
    ]
    
    prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = processor(text=[prompt], images=[img_ref, img_curr], return_tensors="pt").to(DEVICE)
    
    with torch.no_grad():
        outputs = model(**inputs, output_hidden_states=True)
        last_hidden = outputs.hidden_states[-1][:, -1, :].to(torch.float32)
        active_logits = decision_head(last_hidden)[:, :len(options) + 1]
        probs = F.softmax(active_logits / temp, dim=-1).squeeze(0).cpu().tolist()
        
    pred_idx = int(torch.argmax(active_logits, dim=-1).item())
    verdict = options[pred_idx] if pred_idx < len(options) else "Unknown (Needs Human Review)"
    
    return {
        "verdict": verdict,
        "confidence": round(probs[pred_idx], 4),
        "unknown_prob": round(probs[-1], 4),
        "needs_review": (pred_idx == len(options) or probs[pred_idx] < 0.75)
    }