xxilu/gui-shift-implementation

GUI-Shift Implementation

Implementation of GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning (arXiv:2505.12493).

Overview

This repo provides a complete training pipeline for GUI-Shift using native TRL GRPOTrainer with Qwen2.5-VL.

Key Components

• build_data.py: Constructs K-step GUI Transition dataset from AndroidControl
• train_gui_shift.py: GRPO training with rule-based rewards (format + action correctness)
• run_train.sh: Launch script with hyperparameters matching the paper

Workflow

1. Build dataset: python build_data.py

   • Downloads AndroidControl (ckg/AndroidControlParsedWithImages-20k)
   • Extracts K-step state pairs (default k=1)
   • Saves 2000 transition samples as JSONL

2. Train: bash run_train.sh

   • Loads Qwen2.5-VL-7B-Instruct
   • Freezes vision encoder + projector
   • Trains with GRPO (format_reward + action_reward)
   • Pushes to Hugging Face Hub

Hyperparameters (from paper Appendix A)

Parameter	Value
learning_rate	1e-6
num_generations	8
num_train_epochs	4
max_prompt_length	1024
max_completion_length	256
per_device_train_batch_size	2
gradient_accumulation_steps	8
beta (KL)	0.04
epsilon (clip)	0.2
temperature	0.9
bf16	true

Action Space

The unified action space covers 8 types:

• click, long_press (with x,y coordinates)
• scroll (with direction)
• open_app (with app_name)
• input_text (with text content)
• navigate_back, navigate_home, wait

Reward Design

• Format reward: +1 if output contains <answer>...</answer> tags
• Action reward: +1 if parsed action matches ground truth
  • Click/long_press: point falls within ground-truth bounding box
  • Scroll/open_app/input_text: exact parameter match
  • navigate_back/navigate_home/wait: action type match only

References

• Paper: https://arxiv.org/abs/2505.12493
• Dataset: ckg/AndroidControlParsedWithImages-20k
• Base Model: Qwen/Qwen2.5-VL-7B-Instruct