ReentryMem
Out of View, Still in Motion:
Dynamic Entity Re-entry in Video World Models
π Project Page | π» Code | Base Model
ReentryMem addresses dynamic entity re-entry in camera-controlled video world models: recovering a moving subject after it leaves the field of view and later returns. It reads entity presence and spatial regions from text cross-attention on noisy video latents. The shared readout guides entity-weighted flow-matching training and localized overshooting with re-noising during inference.
Explore the project page for synchronized video comparisons, the method, and more qualitative results.
Model Overview
| Property | Specification |
|---|---|
| Released variant | single-entity |
| Base model | Wan2.2-TI2V-5B |
| Parameters | 5.01B |
| Weights | Complete merged DiT with camera-conditioning parameters |
| Precision | BF16 |
| Training checkpoint | Step 1496 |
| Inputs | Observed video, caption, subject phrase, and planned camera controls |
| Continuation | 76 frames at 832Γ480, 15 fps |
The single-entity checkpoint was fine-tuned using entity-weighted flow matching with internally predicted entity regions. The released transformer is accompanied by the calibrated region settings and inference defaults. Its tokenizer, UMT5 text encoder, and VAE are obtained separately from Wan2.2.
How ReentryMem Works
- Localize during denoising. Selected cross-attention heads estimate entity presence and shape directly from noisy latents.
- Emphasize entity supervision. The predicted regions weight the flow-matching objective during training.
- Correct the entity region. At inference, localized overshooting and re-noising use the same region readout; other positions retain the Euler update.
localized and euler are two sampling choices for this same checkpoint. The former is the full ReentryMem inference configuration; the latter corresponds to ReentryMem-Train in the controlled comparison.
Getting Started
Use the ReentryMem inference repository with Python 3.10 or later and CUDA-enabled PyTorch:
git clone https://github.com/Larry-077/ReentryMem.git
cd ReentryMem
conda create -n reentrymem python=3.11 -y
conda activate reentrymem
pip install -r requirements.txt
python tools/download_models.py \
--repo-id larry077/ReentryMem \
--checkpoint-dir checkpoints/ReentryMem
Prepare a 77-frame observed video at 15 fps. The subject phrase must appear in the caption:
python tools/create_case.py \
--video /path/to/observed.mp4 \
--prompt "A girl walks through a stylized town." \
--subjects "girl" \
--preset turn-return \
--output examples/case.json
python infer.py \
--case examples/case.json \
--checkpoint checkpoints/ReentryMem \
--base-model pretrained/Wan2.2-TI2V-5B \
--output-dir outputs/demo
The default uses localized sampling, 50 steps, CFG 5, sigma shift 5, and correction strength 8. Add --sampler euler for ordinary sampling or --save-masks to save entity-region fields. The output includes the complete video, the generated continuation, and a settings receipt. See the code documentation for camera inputs and additional options.
Evaluation
The controlled comparison reported on the project page is:
| Method | Rea-Score (%) β | SC β | TCM β |
|---|---|---|---|
| Baseline | 96.1 | 0.8204 | 0.9154 |
| ReentryMem-Train | 99.2 | 0.8380 | 0.9231 |
| ReentryMem | 97.2 | 0.8378 | 0.9318 |
The protocol separately reports entity reappearance, subject consistency, and temporal consistency match. Refer to the project page for the full comparisons and qualitative examples.
Usage Notes
This release targets single-entity re-entry with the stated observed/future clip lengths and camera controls. Use the custom ReentryMem inference implementation to load the camera-conditioned transformer. GPU memory use depends on the attention backend and runtime settings.
The research results above come from the project's evaluation runs. The standalone package has passed CPU tests, strict full-weight loading, and a small forward check with synthetic text features; full-resolution CUDA generation has not yet been re-run for this package. Validation details accompany the code release.
License and Acknowledgments
The released weights and inference code use Apache-2.0. ReentryMem builds on Wan2.2 and BiWM. See NOTICE for upstream attribution and sampling provenance.
- Downloads last month
- 27
Model tree for larry077/ReentryMem
Base model
Wan-AI/Wan2.2-TI2V-5B