Wan-Animate-2

Wan-Animate-2 by the Alibaba Wan Team animates a reference character image with the motion of a driving video. The driving video is processed in fixed-length segments: each segment runs a reference-extraction pass that caches the driving segment’s K/V in every transformer layer, denoises against that cache, and is decoded inside the loop because the next segment conditions on the previous segment’s decoded tail frames.

Two presets are available: the base checkpoint samples with classifier-free guidance, and the distilled checkpoint samples in few steps without it (its guider is pinned to guidance_scale=1.0).

import torch
from diffusers import ModularPipeline
from diffusers.utils import export_to_video, load_image, load_video

pipe = ModularPipeline.from_pretrained("Wan-AI/Wan2.2-Animate-2-14B-Diffusers")
pipe.load_components(dtype=torch.bfloat16)

# The transformer weights and the per-segment reference KV cache do not co-reside on one 80 GB
# card at the default resolution, so stream the transformer's blocks. Compiling the blocks is
# required as the in-context attention runs on the flex backend
pipe.transformer.enable_group_offload(
    onload_device=torch.device("cuda"),
    offload_device=torch.device("cpu"),
    offload_type="block_level",
    use_stream=True,
)
pipe.text_encoder.to("cuda")
pipe.image_encoder.to("cuda")
pipe.vae.to("cuda")
pipe.transformer.compile_repeated_blocks(fullgraph=False)

# The first demo from the official repository: https://github.com/Wan-Video/Wan-Animate-2
demo = "https://raw.githubusercontent.com/Wan-Video/Wan-Animate-2/main/examples/demo1"
image = load_image(f"{demo}/reference.png")
driving_video, driving_video_fps = load_video(f"{demo}/template.mp4", return_fps=True)
prompt = "人物外观描述：一只银灰色虎斑纹的小猫，拥有圆润的脸庞、竖立的耳朵和巨大的圆形眼睛。它身穿一套深蓝色的制服套装，包括一件带有金色纽扣的西装外套和一条百褶裙。外套里面搭配着白色衬衫，领口处系着一个红色的蝴蝶结，袖口露出白色的衬衫边缘。背景描述：背景为纯白色，光线均匀明亮，无其他杂物或装饰。"

videos = pipe(
    image=image,
    driving_video=driving_video,
    driving_video_fps=driving_video_fps,
    prompt=prompt,
    output="videos",
)
export_to_video(videos[0], "output.mp4", fps=24)

For the distilled checkpoint, load Wan-AI/Wan2.2-Animate-2-14B-Distilled-Diffusers the same way — nothing else changes. Each preset carries its own sampling defaults (40 steps for the base checkpoint, 10 for the distilled one), and no guidance_scale argument exists anywhere: guidance is owned by the pipeline’s guider component (classifier-free guidance at 3.0 for the base preset, disabled for the distilled one).

height and width (defaults 800 and 640) set the target area of the generated video; the actual frame size keeps the reference image’s aspect ratio, and the driving frames are letterboxed to it. Inputs that already sit at the target letterbox size pass through the preprocessing untouched, so preprocessing can also be done entirely outside the pipeline.

WanAnimate2ModularPipeline

class diffusers.WanAnimate2ModularPipeline

< source >

( blocks: diffusers.modular_pipelines.modular_pipeline.ModularPipelineBlocks | None = Nonepretrained_model_name_or_path: str | os.PathLike | None = Nonecomponents_manager: diffusers.modular_pipelines.components_manager.ComponentsManager | None = Nonecollection: str | None = Noneworkflow: str | None = Nonemodular_config_dict: dict[str, typing.Any] | None = Noneconfig_dict: dict[str, typing.Any] | None = None**kwargs )

A ModularPipeline for Wan-Animate-2 character animation.

> This is an experimental feature and is likely to change in the future.

WanAnimate2DistilledModularPipeline

class diffusers.WanAnimate2DistilledModularPipeline

< source >

A ModularPipeline for the distilled Wan-Animate-2 model, which samples in few steps without classifier-free guidance.

> This is an experimental feature and is likely to change in the future.

WanAnimate2Blocks

class diffusers.WanAnimate2Blocks

< source >

( )

Modular pipeline blocks for Wan-Animate-2 character animation: a reference character image and a driving video produce a video of the character following the driving motion.

Components: text_encoder (UMT5EncoderModel) tokenizer (AutoTokenizer) image_processor (WanAnimate2VideoProcessor) image_encoder (CLIPVisionModel) video_processor (WanAnimate2VideoProcessor) vae (AutoencoderKLWan) transformer (WanAnimate2Transformer3DModel) scheduler (SchedulerMixin) guider (ClassifierFreeGuidance)

Inputs: prompt (str): The prompt or prompts to guide image generation. negative_prompt (str, optional): The prompt or prompts not to guide the image generation. prompt_ref (str, optional, defaults to 人物动作的参考视频): The reference prompt for the driving video context max_sequence_length (int, optional, defaults to 512): Maximum sequence length for prompt encoding. image (Image | list): The reference image holding the character to animate. height (int, optional, defaults to 800): Together with width, the target area of the generated video; the aspect ratio comes from image. Overwritten with the resolved frame height. width (int, optional, defaults to 640): See height. Overwritten with the resolved frame width. driving_video (list): The driving video that provides the motion, in any format accepted by VideoProcessor.preprocess_video. driving_video_fps (float, optional): The frame rate driving_video was captured at — load_video(..., return_fps=True) reports it. When set, the driving frames are resampled from it to fps; when None they are used as-is. fps (int, optional, defaults to 24): The frame rate the model generates at segment_frame_length (int, optional, defaults to 81): The number of frames in each inference segment prev_segment_conditioning_frames (int, optional, defaults to 1): The number of conditioning frames carried over from the previous segment generator (Generator, optional): Torch generator for deterministic generation. num_inference_steps (int, optional, defaults to 40): The number of denoising steps. *denoiser_input_fields (None,optional): conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc. output_type (str,optional*, defaults to np): The output type of the decoded videos

Outputs: videos (list): The generated videos.

WanAnimate2DistilledBlocks

class diffusers.WanAnimate2DistilledBlocks

< source >

( )

Modular pipeline blocks for distilled Wan-Animate-2 character animation, sampling in few steps without classifier-free guidance.

Inputs: prompt (str): The prompt or prompts to guide image generation. negative_prompt (str, optional): The prompt or prompts not to guide the image generation. prompt_ref (str, optional, defaults to 人物动作的参考视频): The reference prompt for the driving video context max_sequence_length (int, optional, defaults to 512): Maximum sequence length for prompt encoding. image (Image | list): The reference image holding the character to animate. height (int, optional, defaults to 800): Together with width, the target area of the generated video; the aspect ratio comes from image. Overwritten with the resolved frame height. width (int, optional, defaults to 640): See height. Overwritten with the resolved frame width. driving_video (list): The driving video that provides the motion, in any format accepted by VideoProcessor.preprocess_video. driving_video_fps (float, optional): The frame rate driving_video was captured at — load_video(..., return_fps=True) reports it. When set, the driving frames are resampled from it to fps; when None they are used as-is. fps (int, optional, defaults to 24): The frame rate the model generates at segment_frame_length (int, optional, defaults to 81): The number of frames in each inference segment prev_segment_conditioning_frames (int, optional, defaults to 1): The number of conditioning frames carried over from the previous segment generator (Generator, optional): Torch generator for deterministic generation. num_inference_steps (int, optional, defaults to 10): The number of denoising steps. *denoiser_input_fields (None,optional): conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc. output_type (str,optional*, defaults to np): The output type of the decoded videos

Outputs: videos (list): The generated videos.

Update on GitHub