tt-animatediff

A port, not a checkpoint. This repository ships no model weights. It is an implementation of AnimateDiff that runs the SD 1.4 UNet on Tenstorrent Blackhole hardware through TTNN, with cross-frame temporal attention for motion coherence. The weights it needs — SD 1.4 and the AnimateDiff MotionAdapter — are resolved from their upstream repositories the first time you generate.

Version 0.9.0 · GitHub · Docs · How it was built

Usage

trust_remote_code=True is required: this is a custom pipeline, so loading it executes pipeline.py from this repo.

from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "episod/tt-animatediff",
    custom_pipeline="episod/tt-animatediff",
    trust_remote_code=True,
)

# mode="auto": Blackhole if the ttnn runtime is importable, CPU otherwise.
frames = pipe("a swirling nebula, teal and gold, cinematic").frames
frames[0].save("out.gif", save_all=True, append_images=frames[1:], duration=125, loop=0)

print(pipe.resolved_mode)  # "blackhole" or "cpu" — what actually ran

Loading is offline-safe and opens no device; nothing is fetched until you call the pipeline.

model_index.json's base_model, motion_adapter, and lightning_repo are declarative metadata, not configurable inputs: they record the upstream weights this pipeline resolves, but __call__ never reads them, so passing a different value at from_pretrained() time (e.g. base_model="other/model") is accepted, persists in pipe.config, and changes nothing about what actually runs.

On a Blackhole box (tt-metal built and its python_env active), mode="blackhole" requires the hardware rather than falling back, so a missing runtime is an error instead of a silent 100× slowdown:

frames = pipe("a swirling nebula", mode="blackhole", num_frames=8, num_steps=25).frames

On any machine, no hardware, distilled 4-step Lightning weights make CPU tolerable:

frames = pipe(
    "a swirling nebula", mode="cpu", use_lightning=True, lightning_steps=4,
    num_frames=4, guidance_scale=1.0,
).frames

The delegate package is imported if installed, and otherwise fetched from this repo automatically. To install it explicitly:

pip install 'animatediff-ttnn @ git+https://github.com/tenstorrent/tt-animatediff'

A note on the model tree

The base_model entries in this repo's metadata make the Hub show stable-diffusion-v1-4 and the AnimateDiff MotionAdapter in its model tree. Read that as "this pipeline resolves those weights at runtime", not as "these weights were fine-tuned into a new checkpoint" — nothing here is trained, and this repo ships no weights of its own. The Hub has no relation type that expresses "reimplementation for different hardware", which is what this actually is.

Implementation phases

Phase	What runs where	Motion mechanism
1	CPU, diffusers AnimateDiffPipeline	Full MotionAdapter
2.5	TTNN UNet on Blackhole ← this pipeline's default	Cross-frame temporal attention (temporal_alpha)
3	TTNN UNet + MotionAdapter injected into the denoising loop	Full MotionAdapter, 7 injection points

This pipeline runs Phase 2.5 on Blackhole and Phase 1 on CPU. Phase 3 is reached through this repo's CLI (--motion-adapter), not through this pipeline.

Measured performance

Configuration	Hardware	Throughput	Provenance
mode="blackhole", 25 steps	Blackhole P300C, 1 chip	~1.94 s/frame (15.5 s for 8 frames)	measured 2026-09-07
mode="blackhole", 8 steps	Blackhole P300C, 1 chip	~0.82 s/frame (6.5 s for 8 frames)	measured 2026-09-07
mode="cpu"	CPU (any machine)	~2 min/frame	earlier estimate, not re-verified
mode="cpu", Lightning 4-step	CPU (any machine)	~20 s/frame	earlier estimate, not re-verified

Blackhole figures are 8 frames at 512×512 on a single P300C chip (QB2 board, 4 × P300C), warm — TTNN JIT already compiled — median of 3 runs, timed around generate_animation(). CPU figures are the reference path, not a target.

These Blackhole numbers replace a claim that was 6.4× too slow. The table used to say ~12.5 s/frame at 25 steps, alongside a second row distinguishing a PNDM scheduler from Euler. Both were stale: generate_animation(mode="blackhole") and the ASGI server call the same function (generate_frames_temporal), which uses EulerDiscreteScheduler with timestep_spacing="trailing" and takes no scheduler argument, so there is one Blackhole configuration and not two. The re-measurement validated itself against a known point before being trusted: 8 frames at 8 steps came in at 6.54 s here against the 7.363 s median committed in docs/measurements/serving-benchmark.json, and that ~0.8 s gap is the HTTP and base64-GIF overhead the benchmark's own limitations section attributes it to — so the two agree once measured at the same layer.

Limitations

• The TTNN path needs a Blackhole board and a local tt-metal build (ttnn is not on PyPI). Most users of this repo will only ever exercise the CPU path.
• Frame count must be a multiple of the chip count on multi-chip boards (mesh frame sharding); 8 works on 1, 2, and 4 chips.
• The LCM distillation track is closed — all four runs failed and no distilled weights ship here. use_lightning=True on CPU uses ByteDance's published checkpoint.
• mode="sim" (ttsim virtual Blackhole) is bit-exact but 10–100× slower per op, and needs the simulator binary: pass sim_so="/path/to/libttsim_bh.so", or leave it unset only if the binary is at ~/sim/libttsim_bh.so.

Licensing — read this before redistributing output

The code in this repository is Apache-2.0. The weights it downloads at runtime are not, and this repository cannot grant you their terms:

Artifact	License
This repo's code	Apache-2.0
CompVis/stable-diffusion-v1-4	CreativeML Open RAIL-M
ByteDance/AnimateDiff-Lightning	CreativeML Open RAIL-M
guoyww/animatediff-motion-adapter-v1-5-2	Undeclared upstream

Using this pipeline means accepting the RAIL-M use restrictions on the weights it fetches, even though this repository does not carry them. The MotionAdapter's terms are not stated by its publisher; if that matters to your use, resolve it with the upstream author rather than inferring permission from this repo's Apache-2.0 header.