zhuyuze/SC2_WM

SC²-WM

Official checkpoints for SC²-WM: A Self-Correcting World Model with Closed-Loop Feedback for Vision-and-Language Navigation in Continuous Environments (ICML 2026).

📄 Paper: https://arxiv.org/abs/2608.07548 💻 Code: https://github.com/sunrise-yuze/SC2_WM

Authors: Xuan Yao, Yuze Zhu, Junyu Gao, Zongmeng Wang, Changsheng Xu

Overview

Vision-and-Language Navigation in Continuous Environments (VLN-CE) requires agents to make fine-grained navigation decisions under partial observability. However, most existing methods rely on open-loop execution, lacking mechanisms to detect and correct internal state drift during inference. We propose SC²-WM, a self-correcting world model framework that introduces internal feedback for closed-loop decision making in VLN-CE. Our method derives feedback from world-model foresight to perform state-level plan refinement before action execution. To handle challenging scenarios, we further introduce conditional world-aware adaptation, which enables model-level correction by selectively updating the world model at test time when feedback indicates model capacity insufficiency. The framework is built on Habitat-Sim and is evaluated on the R2R-CE and RxR-CE benchmarks.

Checkpoints

This repository hosts the released weights for the R2R-CE setting.

File	Description
ckpt.46600.pth	R2R-CE main checkpoint (fine-tuned SC²-WM model)
ViT-B-16.pt	CLIP ViT-B/16 vision encoder
pretrained/model_step_100000.pt	VLN-BERT (R2R variant)
pretrained/model_step_82500.pt	VLN-BERT (default)
pretrained/cwp_predictor.pth	Candidate waypoint predictor
pretrained/NeRF_p16_8x8.pth	NeRF rendering module
pretrained/segm.pt	Image segmentation module
pretrained/resnet18-f37072fd.pth	ResNet-18 backbone

Results

R2R-CE (val_unseen)

Method	SR	SPL	NE ↓
Baseline (VLN-3DFF)	44.9	30.4	5.95
SC²-WM (Ours)	50.9	37.2	5.37

RxR-CE (val_unseen)

Method	SR	SPL	NE ↓
Baseline (VLN-3DFF)	25.5	18.1	8.79
SC²-WM (Ours)	35.8	27.2	8.36

Usage

Please refer to our GitHub repository for installation, data preparation, and evaluation instructions. Download the checkpoints into the directory layout described in the repository README.

You can download the weights with the huggingface_hub library:

from huggingface_hub import snapshot_download

snapshot_download(repo_id="zhuyuze/SC2_WM", local_dir="SC2_WM")

Citation

If you find this project useful in your research, please consider citing:

@inproceedings{yao2026sc2wm,
    title     = {{$SC^2$-WM}: A Self-Correcting World Model with Closed-Loop Feedback for Vision-and-Language Navigation in Continuous Environments},
    author    = {Yao, Xuan and Zhu, Yuze and Gao, Junyu and Wang, Zongmeng and Xu, Changsheng},
    booktitle = {Proceedings of the 43rd International Conference on Machine Learning (ICML)},
    year      = {2026}
}

License

Released under the MIT License.