Gaze2HOI checkpoints

Inference weights for Gaze2HOI: Gaze-Aware Intention Disambiguation for Egocentric Hand-Object Interaction Forecasting.

Paper · Project page · Code · Example data

Gaze2HOI predicts structured intentions from RGB observations and gaze, grounds these intentions in contact geometry, and generates coupled hand-object motion. The complete pipeline additionally requires observed 3D hand/object states and scene geometry. It is not a raw-video-only reconstruction system.

Contents

Component	File	Purpose
Intention adapter	intention/adapter_model.safetensors	LoRA parameters and the learned TGSM gaze modulation module
Contact grounding	acg.pt	Online anticipatory contact, support, and placement fields
Motion generation	motion.pt	EMA diffusion parameters with the required model buffers
Fixed model assets	assets/	Normalization, vocabularies, object templates, contact priors, and bottom priors

The package contains inference tensors and runtime architecture settings. Optimizer states, duplicate parameter copies, training logs, base-model weights, and MANO model files are omitted. PyTorch files are plain tensor state dictionaries and are loaded with weights_only=True.

Use

After installing the inference code, download the checkpoints and examples from the code checkout:

gaze2hoi download \
  --model-repo gaze2hoi/Gaze2HOI \
  --data-repo gaze2hoi/EgoAIM-Demo

The default layout places both packages beside the code checkout:

workspace/
├── gaze2hoi-release/
├── gaze2hoi-checkpoints/
├── egoaim-demo/
└── mano/

Follow the code README to install dependencies and prepare local MANO files. Then, from the code repository:

gaze2hoi check --samples all --mano-dir ../mano --hashes
gaze2hoi intention --output outputs/intentions
gaze2hoi forecast --mano-dir ../mano --output outputs/homepage
gaze2hoi render --mano-dir ../mano --results outputs/homepage --output outputs/gallery

The base model and processor are loaded separately from Qwen/Qwen3-VL-8B-Instruct. For a local copy, pass --base-model /path/to/Qwen3-VL-8B-Instruct --local-files-only. The adapter includes a custom gaze module and must be loaded through the released Gaze2HOI implementation.

The default inference run uses the four qualitative and three failure examples with their recorded draw numbers. --samples all additionally includes the four overview examples. Model outputs are generated without temporal smoothing or motion refinement.

Sampling and verification

Default motion sampling uses 50 DDIM steps, guidance scale 1.5, eta 0, and base seed 999. Per-example draw numbers and noise profiles are stored in the example package. VLM inference uses constrained greedy decoding.

All 11 released examples were regenerated: structured intentions matched the corresponding references, and the largest trajectory translation difference was below 1.20 × 10⁻⁷ metres in the verified environment. Fixed seeds support reproduction in a matched setup; hardware, precision, software versions, and device-based noise streams can affect numerical agreement. See the code repository's reproduction notes for the exact verification scope.

The base model's BF16 weights occupy approximately 16.3 GiB, excluding runtime overhead. Multi-GPU placement is supported. Contact and motion inference run after the VLM is released from device memory. MANO is obtained separately under its own license.

manifest.json lists the sizes and SHA-256 hashes of the packaged files. Do not modify those files without regenerating the manifest.

Intended use and scope

The release supports research on gaze-aware intention inference and egocentric interaction forecasting. The supplied examples illustrate both successful forecasts and failure cases; they do not constitute the complete EgoAIM benchmark. Forecasts can confuse purpose, contact part, or active hand, as illustrated on the project page.

See LICENSE.md and NOTICE.md for component-specific terms and source attribution. The code license does not replace third-party data or model terms.

Citation

@article{ma2026gaze2hoi,
    title   = {{Gaze2HOI}: Gaze-Aware Intention Disambiguation for Egocentric Hand-Object Interaction Forecasting},
    author  = {Ma, Chuanyang and Wang, Chenye and Wang, Yixuan and Zhou, Huiling and Qi, Xingqun and Sun, Muyi and Zhang, Man and Shan, Caifeng and Sun, Zhenan},
    journal = {arXiv preprint arXiv:2610.27877},
    year    = {2026}
}