ActWorld: From Explorable to Interactive World Model via Action-Aware Memory

NeurIPS 2026 · Paper (arXiv:2606.17730) · Project Page

ActWorld is an interactive world model built on Helios (Wan2.1-14B). In a single chunk-autoregressive rollout it supports both mid-rollout object interaction (pick up, open, pour, place, ...) and keyboard/mouse viewpoint control.

This repository contains the Stage-1 base model that produced the paper's I-Bench results. It uses 50-step sampling with CFG and includes every component described in the paper:

• per-chunk captions;
• Plücker-ray FiLM and the text-camera branch;
• EAFR and ACHA;
• the persistent event/object memory bank.

The few-step distilled model is not included.

Files

Path	Size	Contents
merged/	58.5 GB	Full ActWorld transformer (fp32): Helios-Base with the LoRA merged in, plus every new module (Plücker FiLM, multi-term-memory patch embeddings, per-head history amplifier, ACHA, memory-bank modules, EAFR scorer)
lora/pytorch_lora_weights.safetensors	2.50 GB	The same model as LoRA weights (rank 128, fp32) on top of the Helios-Base transformer
lora/pytorch_extra_weights.safetensors	1.30 GB	The new modules listed above (fp32), used together with the LoRA
action_embeds_cache.pt	0.34 GB	UMT5 embeddings of the 81 keyboard/mouse command templates, used by the text-camera branch. Generated with tools/offload_data/precompute_action_embeds.py from the Helios-Base text encoder

merged/ and lora/ contain the same weights: merged/ equals Helios-Base + LoRA + extras exactly. infer.sh loads merged/ when it is present and otherwise falls back to lora/ (a 3.8 GB download instead of 58.5 GB). The two give the same results up to bf16 rounding, because lora/ applies the LoRA at runtime.

Usage

pip install -U "huggingface_hub[cli]"
hf download BestWishYsh/Helios-Base --local-dir checkpoints/Helios-Base
hf download xzhexiao/ActWorld --include "merged/*" "action_embeds_cache.pt" --local-dir checkpoints/ActWorld

Helios-Base provides the VAE, the text encoder and the scheduler. Then run, from the ActWorld code repository:

bash infer.sh                          # the I-Bench samples in example/
INPUT_DIR=my_inputs NUM_GPUS=8 bash infer.sh

An input folder holds prompts.csv (id, prompt, image_name, chunk_actions_json), first_frames/, and optionally annotations/<id>.json (per-chunk captions and interaction labels) and cameras/<id>/ (camera trajectory). See the code repository's README for the full format.

Note. The persistent memory bank stays disabled for this checkpoint: infer.sh does not pass --event_memory_enabled or --object_memory_enabled. The bank weights are included so that the checkpoint matches the full architecture.

Citation

@inproceedings{xiong2026actworld,
  title     = {ActWorld: From Explorable to Interactive World Model via Action-Aware Memory},
  author    = {Xiong, Zhexiao and Song, Yizhi and Kang, Hao and Yan, Qing and Jiang, Liming and Yang, Jenson and Fu, Zhoujie and Fotiadis, Stathi and Wang, Angtian and Liu, Zichuan and Liu, Bo and Yang, Yiding and Lu, Xin and Jacobs, Nathan},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026},
  eprint    = {2606.17730},
  archivePrefix = {arXiv}
}

Acknowledgements

ActWorld builds on Helios and Wan2.1. It is released under the Apache-2.0 license, the same as Helios-Base.