primecai/fixer-vace-self-forcing

Fixer VACE — Self-Forcing student

VACE student at SF iteration 800, after 1,000 causal flow-matching warm-start updates. This is an intermediate checkpoint, representing 160 student updates during the Self-Forcing stage. It is the latest complete VACE checkpoint selected for this export on 2026-11-02.

The model repairs a draft video through its already-normalized Wan latents. Its main backbone takes a 16-channel noisy latent and predicts a 16-channel flow; its VACE control branch receives 96 channels made from the archived inactive latent, the draft latent, and an all-ones mask. Inference is causal, one latent frame per chunk, with four SDE denoising steps and KV-cache updates in both the main and VACE branches. Text conditioning is the supplied archived empty context, with CFG 1.

These are custom Fixer/FastGen student weights, not a stock Diffusers pipeline. The export includes the complete student state dict, including the main backbone and VACE control branch. This is an inference package; it does not contain the fake-score model, discriminator, teacher weights, or optimizer states needed to resume training.

Contents

• student.safetensors: student state dict with its original FP32 tensor values and full transformer.* names.
• Wan2.1-VACE-1.3B-diffusers/transformer/config.json: matching VACE architecture with 16 input/output channels and 96 conditioning channels.
• empty_prompt_context.pt: exact archived empty text embedding used for training and evaluation.
• vace_inactive.pt: the corpus's exact vace_inactive tensor, stored under the same key. This is not a tensor of latent zeros.
• runtime/: matching Fixer adapter source and its local import dependency.
• load_student.py: strict loader, VACE conditioning builder, and four-step latent repair helper.
• expected_results/expected_results.mp4: a 28-second reel of 2×2 comparisons generated from this exported student on four fixed held-out examples, showing the input draft, reference target, 50-step teacher, and four-step student.
• expected_results/all_examples_grid.mp4: a 16-second overview with one example per row and the same four comparison columns.
• expected_results/examples/<content_id>/: individual comparison videos and source/output assets.
• expected_results/evaluation_report.json: example identities, seeds, sampling settings, and measurements.

The comparisons are qualitative examples of this intermediate checkpoint, not a claim that the student matches the teacher on a broad benchmark. Draft and target panels show cached latents decoded with the original Wan VAE. Teacher times measure denoising; student times also include the packaged helper's conditioning/noise preparation and conditioning verification. Both exclude model loading, VAE decoding, and video encoding.

Loading

Use the FastGen environment at commit c40fbea17b9a28a38528ee5254471dac4f2f9f12 from NVlabs/FastGen, with PyTorch 2.8.0, Diffusers 0.35.1, Transformers 4.49.0, and PEFT 0.17.1. Install the pinned FastGen dependencies and make its package available on PYTHONPATH.

import sys
from pathlib import Path
import torch
from huggingface_hub import snapshot_download

folder = snapshot_download("primecai/fixer-vace-self-forcing")
sys.path.insert(0, folder)
from load_student import load_student, repair_latents

student = load_student(folder)  # CUDA / bfloat16 for inference
torch.backends.cuda.matmul.allow_tf32 = False
torch.backends.cudnn.allow_tf32 = False
# First supplied example: already-normalized Wan latents [1,16,21,60,104].
example = Path(folder) / "expected_results/examples/000f54795d59dd97"
draft = torch.load(example / "draft_latents.pt", map_location="cpu", weights_only=True)["draft"]
repaired = repair_latents(student, draft, folder, seed=20261025)

The helper constructs condition["vid_context"] as the channel concatenation [vace_inactive, draft, ones_mask], with shape [B,96,21,60,104]. It also loads the archived empty embedding from empty_prompt_context.pt["context"]. Do not replace that embedding with the corpus's context_empty, or pass the 16-channel draft directly as VACE conditioning.

Do not apply Wan latent normalization a second time to cached training/evaluation latents. The original Wan VAE is a separate dependency for encoding/decoding RGB; it is not bundled here. The 21 latent frames correspond to the pilot's 81-frame, 832×480 videos. This package uses draft conditioning, not depth extraction.

Keep the supplied Wan2.1-VACE-1.3B-diffusers directory name: the pinned FastGen loader identifies the model family from that path. The helper sets a default HF_HOME when the environment variable is absent and loads weights strictly against the supplied architecture.

Sampling and training settings

Sampling uses four-step SDE sigmas [0.999,0.937,0.833,0.624,0.0], context noise 0, VACE context scale 1, and one latent frame per causal chunk. For the expected-results recipe, initial noise is generated on CPU in FP32 using the example seed, then moved to CUDA and cast to BF16. The independent SDE stream uses seed + 100000; generating the initial noise directly on CUDA would not reproduce the same sample.

Warm start used 1,000 causal flow-matching updates at LR 5e-5. Self-Forcing uses DMD2 plus GAN at LR 5e-6, with one student update per five outer SF iterations. Each run uses 32 H200 GPUs across four nodes, global batch 64, microbatch 2, and no gradient accumulation. The teacher initialization is the trained VACE Fixer step-5873 teacher.