yp9211/julia-1-mind2web-finetuned

Julia-1-Mind2Web: A Lightweight Decision Engine for Autonomous UI Agents

Julia-1-Mind2Web is a fast, non-autoregressive 144M-parameter decision model fine-tuned specifically for Autonomous Web Agents, Robotic Process Automation (RPA), and Desktop OS Automation.

Unlike massive generative LLMs (7B–70B+ parameters) that slowly generate actions token-by-token and consume gigabytes of VRAM, this model acts as an ultra-low-latency routing and classification "brain." When paired with vision parsers like OmniParser or standard HTML DOM scrapers, it evaluates candidate UI elements in parallel and selects the exact button, link, or input field to interact with in milliseconds.

---

Model Details

Model Description

This model initializes a ModernBertForSequenceClassification backbone (jhu-clsp/mmBERT-small), injects the pretrained decision weights from SupersonicLabs/Julia-1, and fine-tunes the architecture on the flattened interactive action steps of the Mind2Web dataset.

• Developed by: Yatharth (yatharthp9211 / yp9211)
• Model type: ModernBertForSequenceClassification (Cross-Encoder Multiple-Choice Scorer)
• Parameters: ~144M
• VRAM Footprint: < 1.0 GB (Ideal for consumer hardware like a 6GB RTX 4050 running alongside heavy vision models)
• Language(s) (NLP): English (en)
• License: Apache 2.0
• Finetuned from model: SupersonicLabs/Julia-1 & jhu-clsp/mmBERT-small

Model Sources

• Hugging Face Repository: yp9211/julia-1-mind2web-finetuned

---

Uses

Direct Use

The model is designed to score a batch of candidate UI elements (<button>, <a>, <input>, etc.) against a user's high-level task intent and action history.

1. Web Scraping & Browser Automation: Selecting the correct DOM node from Selenium/Playwright snapshots.
2. Desktop OS Automation: Selecting the correct bounding box from OmniParser outputs converted into pseudo-HTML tags, followed by execution via PyAutoGUI.

Downstream Use

Can be plugged directly into multi-agent ecosystems as the fast local execution layer, leaving local VRAM completely free for OCR, object detection, or secondary local LLMs.

Out-of-Scope Use

• Text Generation: This is a sequence classification / logit-scoring model. It cannot chat, write code, or generate free-form text.
• Direct Pixel Processing: The model processes text and structural UI representations (tag, aria-label, text). It requires an upstream parser (like OmniParser or a DOM parser) to translate visual screens into text candidates first.

---

How to Get Started with the Model

Use the standalone Python script below to rank UI candidates and select the optimal element to click:

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

MODEL_ID = "yp9211/julia-1-mind2web-finetuned"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID).to(DEVICE)
model.eval()

# 1. Define the current task and state
task = "Book a cheapest one-way flight from New York to London."
last_action = "Typed 'London' into destination input box."
state_text = f"Task: {task}\nPrevious Action: {last_action}\nWhich element should be interacted with?"

# 2. Provide candidate UI elements (from DOM or translated from OmniParser bounding boxes)
candidates = [
    "<div_ container> Flight deals and promotions",
    "<button> aria-label='Search flights' Search",
    "<a> Cancel booking",
    "<input> Enter promo code"
]

# 3. Tokenize state against all options in parallel
inputs = tokenizer(
    [state_text] * len(candidates),
    candidates,
    padding=True,
    truncation=True,
    max_length=512,
    return_tensors="pt"
).to(DEVICE)

# 4. Score and select the winning UI element
with torch.no_grad():
    logits = model(**inputs).logits.squeeze(-1)
    probabilities = torch.softmax(logits, dim=0)
    best_idx = torch.argmax(probabilities).item()

print(f"🎯 Selected Element: {candidates[best_idx]} (Confidence: {probabilities[best_idx]*100:.2f}%)")

---

Training Details

Training Data

The model was trained on the osunlp/Mind2Web dataset.

Preprocessing

• Task Flattening: Multi-step web tasks were flattened into individual state-action pairs, preserving the step-by-step action_reprs history leading up to each interaction.
• Candidate Formatting: Raw DOM candidates were parsed from JSON attributes into standardized pseudo-HTML strings: <tag> {aria-label} {inner_text}.
• Distractor Sampling: Each positive ground-truth element was paired against 7 randomly sampled negative candidate distractors from the same webpage (NUM_OPTIONS = 8), shuffled dynamically during training to prevent positional bias.

Training Hyperparameters

• Training regime: fp16 mixed precision (torch.amp.autocast)
• Optimizer: AdamW (lr = 2e-5, weight_decay = 0.01)
• Scheduler: Linear warmup (10% of total steps) with linear decay
• Loss Function: Cross-Entropy Loss over the 8 candidate logits per step
• Micro-Batch Size: 2 (across 8 options = 16 sequences per forward pass)
• Gradient Accumulation Steps: 16 (Effective Batch Size = 32)
• Max Sequence Length: 512 tokens
• Epochs: 4 (Updated from initial 3-epoch baseline)

---

Evaluation

Testing Data & Metrics

The model was evaluated zero-shot on unseen tasks using the test_task split of the osunlp/Multimodal-Mind2Web benchmark (1,257 valid interactive action steps).

• Metric: Top-1 Element Accuracy (Percentage of steps where the model assigns the highest logit score to the exact ground-truth positive UI candidate out of 8 total options).

Results

Model / Training Stage	Parameters	Random Guess Baseline	Top-1 Element Accuracy (test_task)	Avg. Training Loss
Random Baseline (8 Options)	-	12.50%	12.50%	-
Julia-1-Mind2Web (3 Epochs)	144M	12.50%	51.79% (651 / 1257)	1.1983
Julia-1-Mind2Web (4 Epochs)	144M	12.50%	52.35% (658/1257)	1.0805

Summary

In just 4 epochs, fine-tuning boosted Top-1 accuracy to **52.35%**—more than 4x the random baseline—while operating inside a strict 512-token window and sub-1GB VRAM envelope.

---

Technical Specifications

Training Infrastructure

• Hardware: 2x NVIDIA Tesla T4 GPUs (30 GB Total VRAM via PyTorch nn.DataParallel)
• Software: PyTorch 2.x, Hugging Face transformers, datasets, safetensors

---

Bias, Risks, and Limitations

• Candidate Pool Size: During training, the model evaluated 8 candidates per pass. When deploying on cluttered screens with 50+ icons/nodes, developers should either batch candidates in groups or use a two-step hierarchical pass (e.g., selecting the screen region first, then the specific element).
• Execution Safety: Because this model can trigger clicks in live browsers or native OS environments, always implement human-in-the-loop confirmation before allowing an automated agent to click sensitive financial, system-deletion, or account-modification buttons.

---

Citation

If you use this model in your research or automation projects, please cite the repository and the underlying Mind2Web benchmark:

@misc{julia1_mind2web_2026,
  author = {Yatharth},
  title = {Julia-1-Mind2Web: A Lightweight Decision Engine for Autonomous UI Agents},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{[https://huggingface.co/yp9211/julia-1-mind2web-finetuned](https://huggingface.co/yp9211/julia-1-mind2web-finetuned)}}
}