paralaif/anima-motion-module

Anima Motion Module (experimental, v0.1)

An AnimateDiff-style motion module for Anima (Cosmos-Predict2 2B Text2Image), plus a ComfyUI node and a "freeze or animate anything" workflow.

This is the first experiment, trained on only 20 clips. It is useful today for one effect: freeze part of an image and animate the rest, or the other way round. It is not a general video model yet. See Next plans.

Frozen character, flowing night beach: Anima + charref keyframe, background redrawn by the motion module.

Files

File	What it is
anima_motion_r2v_v0.1.safetensors	Recommended. Trained with the Anima Ref Adapter v2 active (reference-to-video): every frame reads the character sheet. 2000 steps.
anima_motion_i2v_v0.1.safetensors	First run, without a reference (I2V/T2V). 1500 steps. Kept for comparison.
ComfyUI-Anima-Motion/	ComfyUI node Anima Motion Module (Experimental). Copy into custom_nodes/.
workflows/anima_freeze_animate.json	Freeze-or-animate workflow (keyframe → WeVisOne mask → motion module → video).
scripts/	Data generation (MiniMax H3), latent encoding, training and evaluation scripts, as used (local paths; edit before running).
examples/	Frame strips and clips from the tests below.

How it works

• Anima's DiT is a video architecture (Cosmos), but the released weights were trained on images only. Running it on several frames gives haze and no motion.
• The module folds the frames into the batch, so each frame is a plain Anima image at RoPE t = 0 (full image quality, no haze). It adds a zero-initialized temporal attention after each of the 28 DiT blocks: attention across frames at every spatial position, with 1D RoPE on the frame index.
  • 117.5M trainable parameters (inner dim 512, 8 heads); Anima stays frozen.
  • Zero init: an untrained module reproduces Anima exactly.
• Per-frame timesteps. A frame kept by the noise mask (a keyframe) gets timestep 0, as Cosmos-Predict2 I2V does.
• R2V. After Anima Ref Adapter (v2) in the graph, the node takes over its reference: every frame attends to the cached character-sheet keys/values, with the adapter LoRA on.
• Decode each latent as an image (LatentCutToBatch with t, slice 1). Decoding as video with the causal Wan VAE smears and blotches.

Training

Data	20 clips from MiniMax H3 image-to-video, each from an Anima keyframe of a different character, with a fixed-camera prompt and one gesture (wave, stretch, turn, laugh…)
Frames	16 frames at 8 fps (every 3rd frame of 24 fps), 384×576, each encoded as a separate image
Text	Anima tags + gesture: masterpiece, best quality, score_7, safe, {character tags}, {motion}, anime screencap
Objective	Flow matching as ComfyUI CONST (shift 3, logit-normal σ), MSE on ε − x₀
Optimizer	AdamW, lr 1e-4, grad clip 1.0, batch 1, bf16 autocast, gradient checkpointing
I2V v0.1	1500 steps, 50% of steps with a clean keyframe (frame 0 at σ = 0); RTX 5060 Ti, 5.8 s/step
R2V v0.1	2000 steps from zero init, no keyframe, Ref Adapter v2 with each character's sheet; RunPod RTX 5090, 2.3 s/step (~77 min, about USD 1.50)

Final loss: about 0.088 (I2V) and 0.089 (R2V), plateaued from step ~1200. With 20 clips that is the memorization limit.

Results

Metrics are on 4 test cases (2 characters seen in training, 2 unseen), 16 frames each. Frame-to-frame is the mean absolute pixel difference between consecutive frames (lower = steadier). Contrast is the pixel std; a keyframe is about 55–75 and haze lowers it.

Setup	Frame-to-frame	First → last frame	Contrast	Notes
Anima alone, video latent	3–6	5–6	28–59	Hazy still image, no motion
I2V, 500 steps	41–47	34–54	49–64	Sharp, but every frame is a different image
I2V, 1500 steps	16–36	28–45	41–64	Temporal coherence appears; ignores the keyframe
R2V, 1000 steps	23–27	31–48	53–71	Identity locked to the character sheet
R2V, 2000 steps	16–23	27–43	43–66	Steadier, varied framing, small motion

Pairs of rows: I2V v0.1 (first frame is the keyframe), then R2V v0.1, for three characters. I2V drifts in outfit and identity (bottom pair: the pink-haired officer becomes someone else); R2V keeps the character sheet.

Masked: freeze the rest, animate a selection (training-free on top of the module)

Module: R2V v0.1. The keyframe is repeated over 16 latent frames and only a WeVisOne matte (threshold 0.3, grow 16) is denoised (SetLatentNoiseMask); everything else stays pixel-identical.

Mask, denoise	Frame-to-frame	Result
character (grow 32), 1.0	11–20	New pose; leftover silhouette fills with blotches
character, 0.9	11–18	Larger gestures, breaks the keyframe pose
character, 0.8	9–15	Faithful pose; expression and small hand motion
character, 0.6	9–14	Facial expression only
character, 0.4	8–13	Blinks and breathing (cinemagraph)
face, 0.8	0.6–1.8	Expression only (laughs, blinks, winks); nothing else moves

Character matte, denoise 0.8. The first column is the keyframe.

Face matte, denoise 0.8.

Inverted: freeze the character, let the world flow

The same matte, inverted: the character is frozen and the background is redrawn with an AnimateDiff-like flow (waves, moon, sky). Two things matter:

• Run the background pass without the charref and without character tags (no humans + scene). With the reference, the redrawn area grows a giant copy of the character.
• Use a tight, soft mask (grow 4, blur 8). With grow 16 a ring of the old background stays stuck around the character.

Left: grow 16 (halo). Right: grow 4 + blur 8.

Usage (ComfyUI)

Custom nodes (in ComfyUI/custom_nodes/):

• Anima Motion Module: copy the ComfyUI-Anima-Motion/ folder from this repo.
• Anima Ref Adapter: git clone https://github.com/ArcAnima/ComfyUI-Anima-Ref-Adapter
• WeVisOne-VL, my ComfyUI port of Tencent WeVisOne, which the workflow uses for the selection mask: GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/paralaif/ComfyUI-WeVisOne-VL (skips the weights stored in that repo; download one file below).

Models:

File	Folder	From
anima_motion_r2v_v0.1.safetensors	models/anima_motion/	this repo
wevisone_vl_4b_int8_convrot.safetensors (5.7 GB) or wevisone_vl_4b_bf16.safetensors (9.7 GB)	models/text_encoders/	paralaif/ComfyUI-WeVisOne-VL
anima_ref_adapter_v2_r64.safetensors	models/loras/	ElectronGirl/anima-ref-adapter (the node also downloads it on first use)
split_files/diffusion_models/anima-turbo-v1.0.safetensors	models/diffusion_models/	circlestone-labs/Anima
split_files/text_encoders/qwen_3_06b_base.safetensors	models/text_encoders/	circlestone-labs/Anima
split_files/vae/qwen_image_vae.safetensors	models/vae/	circlestone-labs/Anima

Then load workflows/anima_freeze_animate.json. The same download list is in the workflow's note.

Workflow controls

• Keyframe (two switches):
  • generate with the character sheet: Anima + charref; the sheet is the video's reference;
  • text only (T2V): plain Anima; the generated keyframe becomes the video's charref reference;
  • your own image: it keeps its aspect ratio (scaled to the Megapixels value, sides in multiples of 16) and is the video's charref reference.
• query: what to select (character, face, sky, sea, hair…).
• Freeze/animate switch:
  • true: freeze the selection and animate the rest, with no charref and the THE REST prompt.
  • false: animate only the selection, with charref and the THE SELECTION prompt.
• denoise (video KSampler): 0.4 breathes, 0.8 recommended, 1.0 mutates.
• With your own image, describe its background in the THE REST prompt, or the background turns into whatever that prompt says.

Top: switch true (frozen character). Bottom: switch false (frozen background).

Minimal graph without masks: Anima Ref Adapter (v2, ref_guidance 1) → Anima Motion Module → KSampler on a 16-frame latent → LatentCutToBatch (t, 1) → VAEDecode → CreateVideo (8 fps).

Limitations

• 20 training clips. Motion is small (gestures, expressions). The module memorizes frequent objects: a book from the data shows up in early checkpoints and in some unmasked generations.
• Trained at 384×576 portrait. Other sizes work (the 16:9 beach is 832×480) but are untested for quality.
• It never saw landscapes moving, so in the inverted mode the background morphs (AnimateDiff-like) instead of moving physically.
• The R2V module was trained without a keyframe, so without a mask it does not start from your image.
• Each frame is decoded as an independent image: some fine-line shimmer remains.
• 16 frames (2 s at 8 fps). Longer clips are untested.

Next plans

1. More data: about 300 clips. About 200 characters with varied gestures and about 100 fixed-camera landscapes with ambient motion (waves, clouds, lanterns, falling petals), all from Anima keyframes animated with MiniMax H3.
2. Keyframe conditioning in R2V training. A clean frame 0 in about 70–90% of the steps, so the module learns to continue an existing image instead of redrawing it.
3. Train the way it is used. Start some samples from a noised keyframe (partial denoise) and with spatial masks, to match the masked workflow.
4. Bigger run on a cloud GPU (L40S, batch 4, 3000–5000 steps). Mixed batches: character clips with the reference, landscape clips without it.
5. Longer and wider clips: 32 frames (4 s) and 16:9 / 9:16 buckets.
6. Re-check frame-to-frame stability, keyframe adherence and object memorization against this v0.1.

License and credits

• License: the CircleStone Labs Non-Commercial License, as for Anima, since the module is trained on and runs with Anima. Training clips were generated with MiniMax H3; check its terms for your use.
• Anima by CircleStone Labs.
• Anima Ref Adapter by ElectronGirl.
• WeVisOne-VL by Tencent (ComfyUI port: paralaif/ComfyUI-WeVisOne-VL).
• Method inspired by AnimateDiff.