Orienter

Official code release for “Look Before You Leap: Context-Sensitive GUI Grounding for Boosting Automated Extended Reality (XR) Testing.”

Paper · Project page · Pipeline guide · Environment · Required assets · Model manifest · Release checklist

Orienter detects context-sensitive, user-interactable GUI elements in XR screenshots. The public pipeline has three explicit stages:

1. an LMM combines application context and the current screenshot to mine interactable element descriptions;
2. APE-L(_D) grounds those descriptions, with the optional PII.5–PII.7 reflection loop described in the paper;
3. the evaluation tools convert, merge, filter, and score predictions.

The repository contains source code, selected evaluation ground truth, and the APE-L(_D) checkpoint needed by the main detector. Dataset image payloads, private data, API credentials, experiment logs, generated predictions, and previous experimental results are deliberately excluded.

Release status

The offline control path and utilities are unit-tested. Maintainer validation has exercised one-image live paths for the default OpenRouter VLM, APE-L(_D), and the PII.5–PII.7 advisor loop without retaining generated artifacts in this public copy. Historical experiment outputs and validation result tables are not bundled.

A fresh-clone paper-scale inference reproduction still requires the external assets in docs/ASSETS.md, especially the dataset image payload, provider access, and a validated CUDA/APE environment.

The paper is published in PACMSE, Volume 3, Issue FSE, Article FSE127 (DOI 10.1145/3808134). Do not publish this directory until the repository license, third-party revision audit, and final public model/data links are complete. Machine-readable citation metadata is available in CITATION.cff.

Repository layout

Path	Purpose
approach/run_vlm.py	configurable screenshot-context and candidate generation
approach/run_ape.py	configurable APE detection and optional reflection
approach/reflection.py	paper-aligned PII.5, PII.6, and PII.7 loop
approach/config.py	OpenRouter and paper model profiles
approach/method*.py	legacy experiment entrypoints retained for provenance
approach/ovod/	vendored APE, Grounding DINO, and related detector code
evaluation/tools/	prediction conversion and validated shard merge
evaluation/	detection, semantic, context, and interaction evaluation
dataset/	dataset conversion and split-generation scripts
tests/	offline regression tests for release-critical logic

Installation

The supported public environment targets Python 3.10 or later so it can use a currently supported Pillow release with the latest security fixes:

python -B -m venv .venv
source .venv/bin/activate
python -B -m pip install --upgrade pip
python -B -m pip install -r requirements.txt

requirements.txt covers the release utilities and provider adapters. APE-L(_D) has a separate CUDA/PyTorch stack under approach/ovod/APE; the checkpoint is bundled at approach/ovod/APE/ape_d_model_final.pth and recorded in docs/MODEL_MANIFEST.md. The public tree intentionally excludes compiled APE extensions, build directories, and egg-info metadata, so build/install the APE extension inside the target CUDA environment before running Stage 2. See docs/ENVIRONMENT.md and environment.reference.yml for the tested historical environment, a public compatibility starting point, build commands, and verification boundary. The reference YAML is not a cross-platform lockfile or a security attestation. The historical combined dependency snapshot remains at approach/requirements.txt, but it is not a clean lockfile.

Model configuration

The default provider is OpenRouter. In this project workspace, use openrouter-run so the key is injected by the local credential wrapper and never written to source files, logs, or shell history:

openrouter-run python -B -m approach.run_vlm --help

For a public fresh clone outside this workspace, inject OPENROUTER_API_KEY through the runner's secret manager or process environment. Do not commit .env, shell history, logs, or generated manifests containing credential values.

The default profile uses openai/gpt-5.6-sol, the current flagship model configured for this release. Override it without editing code:

export OPENROUTER_MODEL=openai/gpt-5.6-sol

Paper-oriented profiles remain available through OpenRouter:

Profile	Model
paper_openai / gpt4v	openai/gpt-4o-2024-08-06
paper_claude / claude35sonnet	anthropic/claude-3.5-sonnet
paper_gemini / gemini15pro	google/gemini-pro-1.5

Profiles are defined in approach/config.py. Library callers can override the catalog with a YAML or JSON profile file; CLI users select a named profile and may override the default model through OPENROUTER_MODEL. The paper profiles preserve the evaluated identifiers for GPT-4o-2024-08-06, Claude 3.5 Sonnet, and Gemini 1.5 Pro. Provider catalogs may retire historical routes; if one is unavailable, use a custom profile or direct vendor access and record the substitution instead of silently changing the paper profile. The client records only the credential variable name in run manifests; it never writes the credential value.

Quick start

The commands below show the complete public interface. They require the external files listed in docs/ASSETS.md. Keep screenshots, metadata, prompts, predictions, visualizations, and evaluation files outside the repository:

export ORIENTER_DATA_DIR=/absolute/path/to/orienter_data
export ORIENTER_RUN_DIR=/absolute/path/to/orienter_run
mkdir -p "$ORIENTER_RUN_DIR"

Build a question manifest from mounted screenshots:

python -B scripts/generate_questions.py \
  --images-dir "$ORIENTER_DATA_DIR/images" \
  --output "$ORIENTER_RUN_DIR/questions.jsonl" \
  --prompt "Identify interactable elements." \
  --metadata-template "$ORIENTER_RUN_DIR/app_metadata.json"

Screenshot filenames must match <numeric_app_id>_<numeric_frame>.<supported_ext>, for example 123_4.jpg; additional underscores are not supported. The frame must be in [0, 999] and is zero-padded to three digits when deriving image_id. Fill every generated app_name and app_description field in $ORIENTER_RUN_DIR/app_metadata.json before Stage 1; blank or missing records fail closed.

Validate the mounted inputs and the metadata coverage:

python -B scripts/verify_assets.py \
  --repo-root . \
  --questions "$ORIENTER_RUN_DIR/questions.jsonl" \
  --images-dir "$ORIENTER_DATA_DIR/images" \
  --app-metadata-cache "$ORIENTER_RUN_DIR/app_metadata.json"

Generate candidate descriptions:

openrouter-run python -B -m approach.run_vlm \
  --questions "$ORIENTER_RUN_DIR/questions.jsonl" \
  --images-dir "$ORIENTER_DATA_DIR/images" \
  --output "$ORIENTER_RUN_DIR/candidates.jsonl" \
  --profile default \
  --app-metadata-cache "$ORIENTER_RUN_DIR/app_metadata.json"

--app-metadata-cache accepts JSON or JSONL records keyed by app_id and avoids live Steam page drift. When supplied, missing app IDs fail closed. Omit it only for exploratory runs where live Steam lookup is acceptable.

Data privacy

Stage 1 sends each source screenshot to the selected OpenRouter/provider model. When reflection is enabled, PII.5 additionally sends the bounding-box crop and PII.6 sends the annotated full scene. Do not run private, personally identifying, confidential, or otherwise unauthorized screenshots through a remote provider. Review the selected provider's retention and data-processing terms, obtain the necessary permission, and use an approved local/provider deployment when screenshots cannot leave your environment. Generated candidates, predictions, traces, and error files may also reveal image labels or local filenames; keep them outside the release repository.

Ground candidates with APE-L(_D):

python -B -m approach.run_ape \
  --questions "$ORIENTER_RUN_DIR/questions.jsonl" \
  --candidates "$ORIENTER_RUN_DIR/candidates.jsonl" \
  --images-dir "$ORIENTER_DATA_DIR/images" \
  --output "$ORIENTER_RUN_DIR/predictions.json" \
  --visualization-dir "$ORIENTER_RUN_DIR/ape_visualizations" \
  --ape-root approach/ovod/APE \
  --ape-checkpoint ape_d_model_final.pth

The released checkpoint is SHA-256 verified before loading. A different checkpoint is rejected unless --trust-custom-checkpoint is supplied explicitly after independently verifying its source; PyTorch checkpoints are executable pickle-based artifacts and must not be loaded from an untrusted source.

Enable the paper reflection loop explicitly because it incurs additional LMM calls:

openrouter-run python -B -m approach.run_ape \
  --questions "$ORIENTER_RUN_DIR/questions.jsonl" \
  --candidates "$ORIENTER_RUN_DIR/candidates.jsonl" \
  --images-dir "$ORIENTER_DATA_DIR/images" \
  --output "$ORIENTER_RUN_DIR/predictions.json" \
  --visualization-dir "$ORIENTER_RUN_DIR/ape_visualizations" \
  --ape-checkpoint ape_d_model_final.pth \
  --enable-reflection \
  --reflection-profile default \
  --max-reflection-iterations 10

PII.5 sends the original scene and the bbox crop as separate images. The crop is neither enlarged nor flipped. PII.6 sends the full scene with current boxes, and PII.7 returns structured verification/refinement feedback. Rejected false positives are removed before the next detector call; verified detections are retained.

Ranges, shards, resume, and merge

Use either a range or deterministic modulo shards; the two selection modes are intentionally mutually exclusive. Output filenames receive a selection suffix automatically, so parallel jobs cannot overwrite each other.

python -B -m approach.run_ape ... --start-index 0 --end-index 100
python -B -m approach.run_ape ... --num-shards 4 --shard-index 0
python -B -m approach.run_ape ... --num-shards 4 --shard-index 1

Add --resume to continue from an existing output. Progress sidecars record completed images even when a valid inference produces zero detections.

Merge shards only after validating overlap and coverage:

python -B -m evaluation.tools.merge_predictions \
  --inputs "$ORIENTER_RUN_DIR/predictions.shard00-of-04.json" \
           "$ORIENTER_RUN_DIR/predictions.shard01-of-04.json" \
           "$ORIENTER_RUN_DIR/predictions.shard02-of-04.json" \
           "$ORIENTER_RUN_DIR/predictions.shard03-of-04.json" \
  --questions "$ORIENTER_RUN_DIR/questions.jsonl" \
  --output "$ORIENTER_RUN_DIR/predictions.merged.json"

The merge command fails on overlapping shard progress, duplicate detections, missing sidecars, or incomplete expected image coverage.

Evaluation

Convert prediction fields explicitly for each task:

python -B -m evaluation.tools.to_pred \
  --task semantics \
  --input "$ORIENTER_RUN_DIR/predictions.merged.json" \
  --output "$ORIENTER_RUN_DIR/predictions.semantics.json"

--task accepts semantics, interactable, interaction, or all; all writes separate outputs and never overwrites one task with another. The remaining metric scripts and expected ground-truth layouts are documented in PIPELINE_README.md.

Tests

Run the release-critical offline suite from the repository root:

python -B scripts/smoke_control_path.py
python scripts/run_tests.py

The deterministic control-path smoke uses a generated image plus injected provider/detector functions; it does not call a paid API or load APE weights. The unit tests are also offline. The final release gate additionally requires a real one-image inference, reflection smoke test, and evaluation smoke test after the external materials are supplied.

Legacy experiment scripts

approach/method.py, method_fastuse.py, method_claude.py, and method_gemini.py preserve historical branches and commented stage variants so experiment provenance is not lost. Their active APE paths now produce valid accumulated JSON and accept range/shard controls; method.py and method_fastuse.py also accept explicit data/checkpoint/output paths and reflection flags.

For new runs, prefer run_vlm.py and run_ape.py. They make stage selection, paths, model profiles, app metadata caches, ranges, and shards configurable instead of requiring source edits.

Data, licenses, and citation

Dataset images, annotations, bundled weights, and vendored third-party code may use different licenses. Publish every external asset with its stable URL, version, SHA-256, license, and expected local path. See docs/THIRD_PARTY.md and docs/MODEL_MANIFEST.md.

The paper is published in Proceedings of the ACM on Software Engineering, Volume 3, Issue FSE, Article FSE127, pages 2858–2881, in 2026:

@article{li2026look,
  author  = {Shuqing Li and Binchang Li and Yepang Liu and Cuiyun Gao and
             Jianping Zhang and Shing-Chi Cheung and Michael R. Lyu},
  title   = {Look Before You Leap: Context-Sensitive GUI Grounding for
             Boosting Automated Extended Reality (XR) Testing},
  journal = {Proceedings of the ACM on Software Engineering},
  year    = {2026},
  volume  = {3},
  number  = {FSE},
  pages   = {2858--2881},
  articleno = {FSE127},
  doi     = {10.1145/3808134}
}

CITATION.cff contains the same preferred paper citation. The top-level code license and final Hugging Face repository links remain publication decisions; licenses for the paper, data, model, and vendored source are separate and must not be inferred from one another.