luuuulinnnn/mcsdf-dl3dv-i2v14b

MCSDF — DL3DV I2V-14B run (LoRA + memory modules)

Checkpoint + training visualizations for a Memory-Conditioned Scene Diffusion Forcing (MCSDF) run: Wan2.1-I2V-14B backbone + geometry world-memory modules, trained with the anchor-mixture Diffusion-Forcing objective (noise level = knownness) on DL3DV.

Files

• ckpt_last.pt (1.1 GB) — trained weights only: {step, "lora": <DiT LoRA rank-32 deltas on q,k,v,o,ffn>, "mem_mods": <geometry memory modules>}. The base Wan2.1-I2V-14B-480P DiT is frozen and NOT included — load it separately and apply these on top.
• viz_<scene>_step<N>.png — filmstrips (frames 0 / 20 / 40 / 60 / 80) of the model's generated video at each 500 steps, for 2 clips. Frame 0 = known/near start; later frames = extrapolated.

Training config

• Backbone: Wan2.1-I2V-14B-480P (frozen); trained = LoRA rank 32 (q,k,v,o,ffn.0,ffn.2) + memory modules (MemoryRead every 5 blocks).
• Objective: anchor-mixture Diffusion Forcing — per-token noise level set by knownness k; known tokens SDEdit from the memory latent z_mem, unknown from noise; target velocity points at the true x0 (learn to correct the memory, not copy it); unknown tokens up-weighted.
• Data: DL3DV, leak-free scene split (train 1410 / val 20; eval50 + revisit + DL3DV-140 scenes held out). 832×480, 81-frame out-and-back clips + world-memory cache.
• Run: 8×H200 DDP, 3000 steps, ~18.8h, final EMA loss ≈ 0.26 (converged, no NaN).

Honest status

The objective trained cleanly (loss converged), but the extrapolated frames still show splat / ghost artifacts (see *_step3000.png: frame 0 is sharp, frames 20–80 degrade). i.e. the rank-32 LoRA run does not yet resolve the "splat-cleaner" failure — the model reuses known content well but does not yet generate clean content in unknown regions. Next: diagnose memory over-injection vs sampler vs capacity; a full-finetune (FSDP) of the backbone is the leading candidate.

Load sketch

pipe = load_wan_i2v14b()                      # frozen base
add_lora(pipe, rank=32); attach_mem(pipe)     # same modules as training
ck = torch.load("ckpt_last.pt", map_location="cpu")
pipe.dit.load_state_dict(ck["lora"], strict=False)
pipe._mem_mods.load_state_dict(ck["mem_mods"])