ReentryMem

Out of View, Still in Motion:
Dynamic Entity Re-entry in Video World Models

🌐 Project Page  |  πŸ’» Code  |  Base Model

ReentryMem addresses dynamic entity re-entry in camera-controlled video world models: recovering a moving subject after it leaves the field of view and later returns. It reads entity presence and spatial regions from text cross-attention on noisy video latents. The shared readout guides entity-weighted flow-matching training and localized overshooting with re-noising during inference.

Explore the project page for synchronized video comparisons, the method, and more qualitative results.

Model Overview

Property Specification
Released variant single-entity
Base model Wan2.2-TI2V-5B
Parameters 5.01B
Weights Complete merged DiT with camera-conditioning parameters
Precision BF16
Training checkpoint Step 1496
Inputs Observed video, caption, subject phrase, and planned camera controls
Continuation 76 frames at 832Γ—480, 15 fps

The single-entity checkpoint was fine-tuned using entity-weighted flow matching with internally predicted entity regions. The released transformer is accompanied by the calibrated region settings and inference defaults. Its tokenizer, UMT5 text encoder, and VAE are obtained separately from Wan2.2.

How ReentryMem Works

ReentryMem method overview

  • Localize during denoising. Selected cross-attention heads estimate entity presence and shape directly from noisy latents.
  • Emphasize entity supervision. The predicted regions weight the flow-matching objective during training.
  • Correct the entity region. At inference, localized overshooting and re-noising use the same region readout; other positions retain the Euler update.

localized and euler are two sampling choices for this same checkpoint. The former is the full ReentryMem inference configuration; the latter corresponds to ReentryMem-Train in the controlled comparison.

Getting Started

Use the ReentryMem inference repository with Python 3.10 or later and CUDA-enabled PyTorch:

git clone https://github.com/Larry-077/ReentryMem.git
cd ReentryMem
conda create -n reentrymem python=3.11 -y
conda activate reentrymem
pip install -r requirements.txt

python tools/download_models.py \
  --repo-id larry077/ReentryMem \
  --checkpoint-dir checkpoints/ReentryMem

Prepare a 77-frame observed video at 15 fps. The subject phrase must appear in the caption:

python tools/create_case.py \
  --video /path/to/observed.mp4 \
  --prompt "A girl walks through a stylized town." \
  --subjects "girl" \
  --preset turn-return \
  --output examples/case.json

python infer.py \
  --case examples/case.json \
  --checkpoint checkpoints/ReentryMem \
  --base-model pretrained/Wan2.2-TI2V-5B \
  --output-dir outputs/demo

The default uses localized sampling, 50 steps, CFG 5, sigma shift 5, and correction strength 8. Add --sampler euler for ordinary sampling or --save-masks to save entity-region fields. The output includes the complete video, the generated continuation, and a settings receipt. See the code documentation for camera inputs and additional options.

Evaluation

The controlled comparison reported on the project page is:

Method Rea-Score (%) ↑ SC ↑ TCM ↑
Baseline 96.1 0.8204 0.9154
ReentryMem-Train 99.2 0.8380 0.9231
ReentryMem 97.2 0.8378 0.9318

The protocol separately reports entity reappearance, subject consistency, and temporal consistency match. Refer to the project page for the full comparisons and qualitative examples.

Usage Notes

This release targets single-entity re-entry with the stated observed/future clip lengths and camera controls. Use the custom ReentryMem inference implementation to load the camera-conditioned transformer. GPU memory use depends on the attention backend and runtime settings.

The research results above come from the project's evaluation runs. The standalone package has passed CPU tests, strict full-weight loading, and a small forward check with synthetic text features; full-resolution CUDA generation has not yet been re-run for this package. Validation details accompany the code release.

License and Acknowledgments

The released weights and inference code use Apache-2.0. ReentryMem builds on Wan2.2 and BiWM. See NOTICE for upstream attribution and sampling provenance.

Downloads last month
27
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for larry077/ReentryMem

Finetuned
(145)
this model