dongjintao/GUI-Swift-4B

Single-Pass GUI Action Prediction
Examples · Results · Quick start · 简体中文

GUI-Swift-4B turns a screenshot, an instruction and action history into one desktop GUI action through a single native causal forward over 131 fixed value slots. It is a lightweight LoRA release built on GUI-Owl-1.5-4B-Instruct, with the original vocabulary head retained and no separate autoregressive text writer.

Release: Desktop131 SFT · step3368   /   Adapter: 33.03M parameters · 132 MB

This is a desktop research checkpoint. “Single-pass” describes the inference mechanism; a matched speed or task-success advantage has not been established.

Real validation examples

These are actual saved predictions from this checkpoint's 1,967-row AgentNet development validation set. The gallery is selected after evaluation and includes a text failure. Screenshots show the current input frame; action history was also part of the model input. These are next-action examples, not completed-task demonstrations.

GIMP · Find the Desktop folder ACTION CORRECT · OFFSET (2, 0) / 1000 Input screenshot · click for full size Model prediction {"action": "left_click", "coordinate": [155, 243]} Reference action {"action": "left_click", "coordinate": [153, 243]} Original task instruction & record Please use GIMP to open the "MMQ.png" image on your desktop, add the black text "sky" at the top of the image with a font size of 100 and make it bold. Historical action records: 4 · Strict action match: false Saved prediction & provenance (JSON)

Writer · Enter a search term EXACT ACTION MATCH Input screenshot · click for full size Model prediction {"action": "type", "text": "CAGR"} Reference action {"action": "type", "text": "CAGR"} Original task instruction & record Help me open the desktop document "144.odt", change the overall font size of the document to 16pt and the font color to green, use the find and replace function to replace "CAGR" with "WWAA", and then save it as "yrrte7" on the desktop after completion. Historical action records: 8 · Strict action match: true Saved prediction & provenance (JSON)

Impress · Insert a slide EXACT ACTION MATCH Input screenshot · click for full size Model prediction {"action": "key", "keys": ["ctrl", "m"]} Reference action {"action": "key", "keys": ["ctrl", "m"]} Original task instruction & record In Impress,add two new slides.Enter“Ancient History”and“Modern Art”in the title areas of slides 1 and 2 respectively.Enter the table of contents“・Egyptian・Greek・Roman”in the content area of slide 3.After completion,change the font to“Likhan”uniformly. Historical action records: 1 · Strict action match: true Saved prediction & provenance (JSON)

Calc · Rename a worksheet EXACT ACTION MATCH Input screenshot · click for full size Model prediction {"action": "type", "text": "7"} Reference action {"action": "type", "text": "7"} Original task instruction & record Create three worksheets named "5, 6, 7" and change the color labels of worksheet 567 to "red, yellow, blue", then hide them. Historical action records: 12 · Strict action match: true Saved prediction & provenance (JSON)

A limitation · Repeated text

ACTION CORRECT · TEXT MISMATCH

Input screenshot · click for full size

Model prediction
{"action": "type", "text": "・ Egyptianian GreekGreek Roman Roman"}

Reference action
{"action": "type", "text": "・Egyptian・Greek・Roman"}

Original task instruction & record

In Impress,add two new slides.Enter“Ancient History”and“Modern Art”in the title areas of slides 1 and 2 respectively.Enter the table of contents“・Egyptian・Greek・Roman”in the content area of slide 3.After completion,change the font to“Likhan”uniformly.

Historical action records: 13 · Strict action match: false

Saved prediction & provenance (JSON)

Inspect the GIMP click locations

The model predicts (155, 243) and the reference is (153, 243), on a 0–1000 coordinate scale. The action type agrees and the coordinate offset is (2, 0); the strict full-action match is false. The circles/cross are presentation overlays, not model outputs or evidence of executing the click.

Source: AgentNet / OpenCUA. Example manifest · Original screenshot license. The raw validation corpus is not bundled.

Results at a glance

Saved evaluation	Result	What it measures
OSWorld-derived selected suite	16.29% · 58 / 356	Environment-verified full task success among scored tasks
Development validation · action type	75.60% · 1,487 / 1,967	Whether the action type agrees
Development validation · position @50/1000	49.64% · 548 / 1,104	Action type agrees and both x/y errors are <= 50 on a 0–1000 scale
Development validation · text	30.08% · 120 / 399	Exact text action match on reference type rows

The position threshold was changed from 20 to 50 after inspecting the results, then recounted on the same saved predictions. It allows 5% of each screen dimension and measures the primary point, not widget hit rate or complete action correctness. The original @20 result is 41.30% (456/1,104); strict full-action matching is 24.05% (473/1,967). Metric definitions and original results.

The OSWorld-derived run selected 361 of 369 tasks: 356 scored, 5 unscored; mean reward among scored tasks was 0.1712077. This is not a complete official leaderboard score. Validation was repeatedly used during development, and this checkpoint was chosen after observing results. No new model inference or environment run was performed; the five examples above do not estimate average performance.

Next direction — reinforcement learning: use this SFT checkpoint as a starting point and train with environment-verifier feedback to improve end-to-end task completion. The 16.29% is the current measured SFT result; improvement through RL is a future objective and has not been established for this release.

Original saved metrics · Position recount · Training and evaluation details

Quick start

Use Python 3.10+ and an NVIDIA GPU. Download the adapter repository, then prepare the pinned backbone and GUI-Decisions dependency on a disk with enough space.

hf download dongjintao/GUI-Swift-4B --local-dir GUI-Swift-4B
cd GUI-Swift-4B
python -m pip install -r requirements.txt
python prepare_dependencies.py --destination /path/on/data-disk/gui-swift-deps
python infer.py \
  --base-model /path/on/data-disk/gui-swift-deps/GUI-Owl-1.5-4B-Instruct \
  --gui-decisions /path/on/data-disk/gui-swift-deps/GUI-Decisions \
  --request examples/request.json

The base download is approximately 8.9 GB. Matching existing local dependencies can be used instead. The script predicts an action; it does not operate your desktop. The bundled runnable request is a synthetic smoke example, separate from the saved validation gallery. Gallery JSON files document historical results and are not complete replay requests because historical images/actions are omitted.

Request format and loading notes

{"instruction": "Click the Save button.",
 "image": "screen.png", "history": []}

Image paths are relative to the request file. History records contain image and actions (actually executed canonical actions); the original runtime retains all supplied actions and the last two historical screenshots. Do not put the reference next action into the input. Use the custom infer.py entry point: a generic generate() call or attaching PEFT to the entire VLM does not reproduce the protocol. The frozen loader attaches LoRA to the language-model submodule.

Design & credits

We thank GUI-Decisions for its 256-bin coordinate representation and readout, our principal inspiration from that work. Single-forward complete-action prediction was already being explored in this project. The runtime also directly reuses the pinned upstream slotread.train.read and label/numerical utilities; design history and implementation reuse are credited separately.

We also thank GUI-Owl / Mobile-Agent for the backbone, OpenCUA / AgentNet for the validation examples, and PEFT / Transformers for adapter and model support. The external GUI-Decisions snapshot has no license file and is not bundled or relicensed here. Third-party sources and terms.

Research notes

• Current strengths and gaps: the saved cases show simple click, key and text prediction. Text repetition, dragging, misclicks and premature stopping remain limitations. No verified mobile compatibility or general latency advantage is claimed.
• Scope: SFT, desktop-only, 131 slots. This release is not Unified133 or the later RL model. Terra-assisted trajectories are not standalone model results.
• Reproducibility: 504 exported adapter tensors matched the checkpoint exactly. Separate-process checks matched inputs, logits and decoded actions on two saved inputs. One RTX 4090 used about 9.1 GiB peak allocated memory in that bounded check; this is not a maximum-context memory bound.

Full protocol and training recipe · Release verification · Provenance · File checksums