🎬 Video Model Harness

A registry-driven marimo notebook for running open-weight text-to-video diffusion models locally on your own GPU.

Pick a model, write a prompt, set a duration, press Generate video. One generate_video function drives LTX-Video, Wan and CogVideoX alike: it inspects each pipeline's __call__ signature and forwards only the arguments that pipeline actually accepts, so adding a checkpoint to the registry is enough to support it.

Models in the registry

Key	Checkpoint	Pipeline	Download
`ltx-video-0.9.5`	`Lightricks/LTX-Video-0.9.5`	`LTXPipeline` / `LTXImageToVideoPipeline`	~25.7 GB
`wan2.2-ti2v-5b`	`Wan-AI/Wan2.2-TI2V-5B-Diffusers`	`WanPipeline` / `WanImageToVideoPipeline`	~34.2 GB
`wan2.1-t2v-1.3b`	`Wan-AI/Wan2.1-T2V-1.3B-Diffusers`	`WanPipeline`	~28.9 GB
`cogvideox-2b`	`THUDM/CogVideoX-2b`	`CogVideoXPipeline`	~13.8 GB

Two rendering paths

Short clips — up to one chunk of 8k+1 frames renders in a single diffusion pass.
Long clips — anything longer is rendered as a chain of chunks where each chunk is image-to-video conditioned on the previous chunk's final frame, then stitched into one continuous take. Presets with no I2V class fall back to text-only chunks, which still goes long-form but cuts between chunks.

Nothing in the harness asks a single diffusion pass to cover every frame of a half-minute render; that is what keeps long output coherent.

The lesson builder

build_lesson_video renders a narrated, captioned instructional video end to end from a declarative LESSON config:

each segment is generated as its own clip (so one segment can be re-shot without re-rendering the whole lesson);
every frame is upscaled to 1280×720 and composited with a lesson card drawn using real fonts — diffusion models cannot spell, so every word a learner reads is drawn with PIL, not generated;
narration is synthesized with edge-tts and padded per segment to that segment's exact video length, which keeps the voice locked to the captions it belongs to;
frames and audio are muxed to H.264/AAC with ffmpeg.

Running it

pip install marimo
marimo edit video_model_harness.py

The notebook carries its own inline PEP 723 dependency block, so uv-aware runners install it automatically:

accelerate>=1.15.0, diffusers==0.42.1, edge-tts==7.3.1, ftfy>=6.3.1,
huggingface-hub>=0.34.0,<2, imageio-ffmpeg>=0.8.0, numpy==2.5.4,
pillow==13.0.0, sentencepiece>=0.2.2, torch>=2.11,<2.12

Hardware. A CUDA GPU is required in practice — the presets are 13.8–34.2 GB of weights. The notebook reports the detected device, VRAM, dtype and library versions at the top so you know what you are actually rendering on. Weights are downloaded on first use and cached by huggingface_hub; a pipeline instance is reused for the rest of the session (load_seconds is 0.0 on a cache hit).

Outputs

Every run writes an .mp4 plus a JSON sidecar recording the exact settings — model, prompt, negative prompt, frames, fps, resolution, steps, guidance, seed, render seconds and peak VRAM — to outputs/video/. The notebook shows the most recent clip with autoplay and a table of the last 15 runs.

Field notes

Learned the hard way while building the lesson renderer, and kept in the code as comments:

The 2B model renders rooms well and people badly. Wide shots of a colourful classroom with the speaker small in frame measured ~13× sharper and ~18× more saturated than a soft-lit close-up of a person, which came out a blown-out, colourless "ghost". Keep prompts wide, well-exposed and colourful.
Never ask for "soft light" or "shallow depth of field"; under-specified prompts drift pale and desaturated.
grade_frame pulls only clearly out-of-range frames back into range — brightness down if mean luminance exceeds 205, saturation up if mean saturation is under 0.35.

License

Not specified — the author should add a LICENSE file and a license field to this card before others reuse it.

Downloads last month: -