UDGAM LABS · Video model · 212M-parameter generator, text-to-video and image-to-video

Drishya Tiny

Drishya Tiny is a 212M-parameter video generator trained from scratch, using the frozen Wan2.2 VAE and Flan-T5-Base text encoder. It turns a text prompt, or a text prompt plus a first frame, into a 2-second 256×256 clip.

It is a research release, and a fun experiment: how far does a small generator get when it learns video from random weights, on a few rented GPUs, about $57 all-in? The answer is below, with the parts that worked, the parts that didn't, and what each one cost. The clips are short, small and often wrong. Waves, fire and snowy skies work best. People do not.

Drishya (दृश्य) means "a scene, something seen".

Try it: in your browser, on your own device · on Hugging Face Spaces (GPU) · technical report · code, logs and every stage on GitHub

Quick start

pip install git+https://github.com/udgamlabs/drishya-tiny

from drishya import Drishya, save_mp4

pipe = Drishya.from_pretrained("udgamlabs/Drishya-Tiny")   # also fetches Flan-T5-Base and the Wan2.2 VAE

# Text-to-video
prompt = "waves crashing on rocks at sunset"
frames = pipe(prompt, image=None, steps=50, cfg=5.0, shift=5.0, seed=0)
save_mp4(frames, "clip.mp4", prompt)

# Image-to-video: the image becomes the first frame (centre-cropped to 256×256) and is kept as given
from PIL import Image
prompt = "a boat drifting on a calm lake"
frames = pipe(prompt, image=Image.open("lake.jpg"), seed=0)
save_mp4(frames, "boat.mp4", prompt)

From the command line:

python -m drishya "waves crashing on rocks at sunset" --out clip.mp4
python -m drishya "a boat drifting on a calm lake" --image lake.jpg --out boat.mp4

It picks CUDA, then Apple-silicon MPS, then CPU. The output is 17 frames at 8 fps (about 2 seconds), 256×256.
save_mp4 burns a small "AI-generated" label into each frame and writes provenance metadata (generator, model, prompt, seed). Please keep both when you share clips (see the use policy).
Settings: 50 Euler steps, CFG 5 and shift 5 are the defaults and what the VBench numbers below use. CFG 3 gives better FVD (373 against 432 FVD-512) with slightly weaker prompt-following. Prompts are cut at 77 Flan-T5 tokens.
Prompts that work: nature and scenery (water, waves, sky, snow, boats, fire). One subject, no text, no faces.

Run it in your browser

onnx/ holds a browser version that runs on your own device with WebGPU (desktop Chrome or Edge; other browsers fall back to a slower CPU path). It works from text or from a photo; nothing you type or add leaves your device. It downloads 665 MB once:

File	What it is
`onnx/drishya_tiny_dit_cond.onnx`	the 212M generator, guidance built in, with the image-to-video input (fp16 weights, fp32 maths)
`onnx/taew2_2_encoder.onnx`	TAEW2.2's tiny encoder, for the photo in image-to-video (3 MB, loaded only when you add a photo)
`onnx/flan_t5_base_encoder.onnx`	the Flan-T5-Base encoder (google/flan-t5-base, Apache-2.0; from the Xenova/flan-t5-base export, weights stored in fp16)
`onnx/taew2_2_decoder.onnx`	TAEW2.2, a tiny decoder for Wan2.2 latents (madebyollin/taehv, MIT)

The browser uses TAEW2.2 instead of the full Wan2.2 VAE, so its clips are softer than the Python package's. (onnx/drishya_tiny_dit.onnx is the earlier text-only export, kept for reference.) The page and its code are in the GitHub repo's browser/ folder.

The model

Part	Parameters	Trained by us?
Generator: VideoDiT "s210" (width 768, depth 16, 12 heads, patch 1×2×2)	211.92M	Yes, from random initialisation
Text encoder: Flan-T5-Base encoder (including the token embedding)	109.63M	No, frozen
Wan2.2 VAE (from `Wan-AI/Wan2.2-TI2V-5B-Diffusers`, `vae/`)	704.69M (encoder 149.63M, decoder 555.05M)	No, frozen

Whole system at inference: text-to-video uses the generator, the T5 encoder and the VAE decoder, about 877M parameters; image-to-video adds the VAE encoder, about 1.03B. Only the 212M generator was trained by us.
This repository holds the generator only: model.safetensors (EMA weights, step 180,080) and config.json. The frozen parts download from their owners' pages under their own licences.
Inside: the generator works on Wan2.2 latents of 48 channels × 5 frames × 16 × 16 (the VAE compresses 16× in space and 4× in time), which become 320 tokens after patching. Each block is adaLN-zero self-attention over all 320 tokens, cross-attention to the Flan-T5 tokens, and an MLP. Sampling is rectified flow with Euler steps and classifier-free guidance.
Image-to-video by first-frame replacement: the clean latent of the given frame replaces the first noisy latent frame at every step, so the model learns to continue it.

Evaluation

The pre-registered gate (Stage 3, 20 October 2026): pass

Held-out VidGen-1M clips (zips 2040-2047, never trained on). EMA weights, 50 Euler steps, shift 3, CFG 5. Measured by Udgam Labs.

Check	Result	Reference
Text-to-video FVD-2048 (lower is better)	235.7	real clips frozen on frame 0: 660.4
Motion ratio (against real clips)	0.73	rule: 0.5-2.0
CLIP gap (own caption minus a shuffled caption; OpenCLIP ViT-B/32 `laion2b_s34b_b79k`, not OpenAI CLIP)	20.15	real clips: 25.73, so 78% of real
Held-out flow loss, end of run vs 25% of the run	0.454 vs 0.471	must fall
Image-to-video FVD-512	240.3 (motion 0.89, CLIP gap 23.7)

FVD harness audit: our FVD code matches StyleGAN-V's official FVD code to a relative 1e-10 on identical frames (21.5059 vs 21.5059; 279.374 vs 279.374).
FVD-512 is biased upward against FVD-2048, so the two are never compared.

VBench (21-22 October 2026)

These are our own sampling and scoring with the official VBench code, not a certified leaderboard entry. VBench at commit fd18b3d055cb; CFG 5, shift 5, 50 steps, chosen by a sampling sweep we fixed in advance.

Suite	Total	Quality	Semantic / I2V
Text-to-video, standard prompts (946 prompts × 5 videos)	66.92	75.59	semantic 32.26
Text-to-video, VBench long prompts (GPT-enhanced)	not computed (temporal flickering not run, by design)	-	semantic 33.03
Image-to-video, 1:1 crops (1,118 × 5)	82.62	75.00	I2V 90.24

Text-to-video per dimension (%):

Quality: subject consistency 87.54, background consistency 94.52, temporal flickering 98.78, motion smoothness 96.87, dynamic degree 80.56, aesthetic 34.77, imaging 51.49.
Semantic: object class 24.97, multiple objects 2.91, human action 20.00, colour 84.39, spatial relationship 6.03, scene 6.03, appearance style 20.40, temporal style 12.15, overall consistency 14.53.

Image-to-video per dimension (%): i2v subject 93.64, i2v background 95.92, camera motion 24.80, subject consistency 85.77, background consistency 93.63, motion smoothness 96.49, dynamic degree 69.35, aesthetic 50.87, imaging 63.43.

The VBench-I2V input images we used are byte-identical (sha256) to the official suite: 355 of 355, checked 25 October 2026.

Compared with published 256px models

Published numbers, not ours: the VBench team's own sampling and scoring on the official leaderboard, at 256×256 and 16 frames (about 2 s), checked against the leaderboard on 25 October 2026. Every one of these models has Stable Diffusion in its lineage.

Model	Parameters	Total	Quality	Semantic
Drishya Tiny (our own scoring)	211.92M generator, trained from scratch; frozen Wan2.2 VAE and Flan-T5-Base	66.92	75.59	32.26
ModelScope T2V (published)	1.7B system (UNet about 1.35B), initialised from SD 2.1	75.75	78.05	66.54
VideoCrafter-0.9 (published)	not stated; spatial layers from SD	73.02	74.91	65.46
InstructVideo (published)	a reward fine-tune of ModelScope	76.61	81.56	56.81

Image-to-video model	I2V total
DynamiCrafter-256 (published)	85.25
ConsistI2V, from SD 2.1-base (published)	84.07
Drishya Tiny (our own scoring)	82.62
VideoCrafter-I2V (published)	82.57

In one line: VBench video quality 75.6, level with the 256px VideoCrafter-0.9 (74.9) and close to ModelScope (78.1), both built on Stable Diffusion, with a generator trained from scratch (on the frozen Wan2.2 VAE and Flan-T5-Base) and about a sixth the size of ModelScope's 1.35B UNet; prompt-following (semantic 32.3) is about half theirs (65-67), and the total score is lower.

How it was trained, stage by stage (failures included)

Every stage's results, logs and machine bills are in the GitHub repository.

Day-0 checks ($0.03). The Wan2.2 VAE round trip, an overfit test and the whole pipeline on a Mac and a Colab probe, before renting anything.
Stage 1: does the recipe learn? (18 October, $7.14). The same 211.9M generator, unconditional, on SkyTimelapse: FVD-2048 167.7 (real frames frozen still: 215.1; real against real: 72.2), motion 0.89, 2.78M clip-samples in 120 minutes on one RTX 5090. The recipe learned. What went wrong: the model ran out of data, not compute: its held-out loss bottomed at about half the run. For context only (different harness, not a matched comparison), Latte-XL reports 59.82 and StyleGAN-V 79.52 on this benchmark.
Stage 2: three techniques against the plain recipe, at equal compute (19 October, $7.20). 60 GPU-minutes each. What went wrong: none of them beat the plain recipe (noise margin ±25.1 FVD). Joint image+video training (182.4) and TREAD token routing (171.2) tied the control (160.7); TREAD gave 39% more throughput but no quality gain. Our MeanFlow implementation diverged; that is a bug or a setup problem on our side, not a verdict on the method. We kept the plain recipe.

Stage 3: the release model (20 October, $24.71). The Stage 1 recipe, unchanged, on VidGen-1M:

What	Value
Hardware	1× NVIDIA L40S 48 GB (Shadeform, Massed Compute)
Training time	900 minutes (15 h), 02:41-18:21 IST
Steps	180,080 at batch 64
Clips seen	11.53M (about 24 passes over 475,157 windows)
Objective	Rectified flow (flow matching); 25% of steps train image-to-video by first-frame replacement; 10% caption dropout for classifier-free guidance
Optimiser	MuonAdamW (AdamW learning rate 2e-4, Muon 0.02), 1,000 warm-up steps, learning-rate decay over the final 25%, EMA 0.9999

The release weights are the run's final EMA weights. All four pre-registered gate checks passed. What went wrong: $3.55 of this stage went on five failed data-step attempts, failed main-run starts, a dry run and a speed probe.

Stage 4: evaluation (21-22 October, $8.48). The FVD harness audit, the sampling sweep and VBench.
Stage 5a: can better data fix prompt-following? (22-23 October, $9.90). A controlled test: three arms, each trained 18,000 more steps from the Stage 3 model, scored on six VBench semantic dimensions (516 prompts × 2 seeds, paired bootstrap 95% CI). None of these arms is the release model.
- A caption fix (128 tokens and 20% short captions): +3.01 (CI +0.93 to +5.26), at no cost in held-out FVD-512 (413 against 414).
- The caption fix plus 30% CC12M photo steps: +4.34 against the control (CI +1.99 to +6.72), but held-out FVD-512 got 104 worse (518.7 against 414.4).
- What the photos add on top of the caption fix: +1.65, in object class (+6.0) and scene (+3.9); multiple objects and spatial relationship did not improve. (+3.01 and +4.34 are paired, per-prompt statistics.) What went wrong: neither fix came close. Photos teach what things are, not how to compose them, and they cost video quality. At this size, data fixes move prompt-following by 3-4 points, not the 30 it would need.

Cost

About $57 all-in ($57.46), every failed attempt included; about $21 for the release model's own path (the Stage 3 data step on one RTX 5090 for about 6.2 hours, $7.40, and the training run on one L40S, $13.76 for 15 h 40 min of machine time including evaluations).

Stage	GPU cost	What it bought
Day-0 checks (Mac, Colab probe, research)	$0.03	feasibility
Stage 1 (SkyTimelapse method check)	$7.14	the recipe works
Stage 2 (three techniques vs the plain recipe)	$7.20	keep the plain recipe
Stage 3 (data + training of the release model)	$24.71	the release model
Stage 4 (sampling sweep + VBench)	$8.48	the evaluation
Stage 5a (three controlled arms + scoring)	$9.90	the ablation
Total	$57.46

Not included: the pretraining of the borrowed Wan2.2 VAE and Flan-T5-Base (not ours), people's time, Mac electricity and storage.

Training data

VidGen-1M (Tan et al., 2024, arXiv 2408.02629), whose clips come from HD-VILA-100M (Xue et al., CVPR 2022, Microsoft). We used about 300K clips (zips 0-599), cut into 475,157 two-second training windows: one from the start of each clip and, for long clips, one from the middle.
Captions: VidGen-1M's own, about 80 words on average; the model reads the first 77 tokens.
Held out: zips 2040-2047 (3,918 windows; the gate scored 2,048 clips), never trained on.
No photos, no synthetic data and no other video set are in the release model.
Research use only: HD-VILA-100M is provided under Microsoft's Research Use of Data Agreement. That is why the weights are non-commercial.
Clips and captions are not redistributed. The clip IDs (VidGen vid and window start) and the preparation scripts are in the GitHub repository.

Where it falls short

Prompt-following is about half the peers' (VBench semantic 32.3 against about 65-67 for the 1-1.7B 256px peers). It almost never draws two named objects (multiple objects 2.9%), rarely gets spatial relations (6.0%) or named scenes (6.0%), and gets the object class right about a quarter of the time (25.0%).
Faces and hands smear. Text melts.
Short and small: 2 s, 256×256, 8 fps. Motion is small.
Image-to-video drifts: later frames lose detail and blur over the 2 s; faces and text melt.
Visual quality is modest: aesthetic 34.8%, imaging 51.5%.
Data fixes alone won't close the gap at this size: Stage 5a moved prompt-following by 3-4 points.
Not certified: the VBench numbers are our own sampling and scoring.
Non-commercial weights: the training clips come from HD-VILA-100M, whose terms allow research use only.
It works best on nature and scenery (water, waves, sky, snow, boats, fire); people degrade. This is our impression from picked clips, not a measurement.

Intended use

For: research, teaching and personal experiments with small video models: studying how a generator learns motion, fine-tuning, ablations, and reproducing or extending the recipe.

Not for: paid products or services, depicting real people, news or factual footage, or anything where a wrong or misleading clip could cause harm. The model has no safety filter of its own; add your own moderation for any public-facing use.

Use policy (USE_POLICY.md). In short, please don't use Drishya Tiny, or anything built from it, to break the law, make sexual content involving minors, make intimate content of a real person without consent, impersonate real people, pass generated video off as real footage, harass or target anyone, spread election or public-safety disinformation, or make decisions about people's rights, jobs, credit, health or legal status. Label outputs as AI-generated and keep the provenance metadata. The policy is guidance and a request, not an extra licence term. Report problems in this page's Community tab.

Licence

Weights: CC BY-NC 4.0 (Creative Commons Attribution-NonCommercial 4.0 International). Copyright 2026 Udgam Labs LLP. You may use, study, fine-tune, share and adapt them for non-commercial purposes. Credit:

Drishya Tiny by Udgam Labs (https://udgamlabs.com/models/drishya-tiny), licensed under CC BY-NC 4.0.
Code: Apache-2.0 (GitHub), so you can retrain the generator on clips you have commercial rights to.
Third-party parts (not covered by the weights licence, not redistributed here, downloaded from their owners):
- Wan2.2 VAE: Wan Team, Alibaba, Wan-AI/Wan2.2-TI2V-5B-Diffusers (vae/), Apache-2.0.
- Flan-T5-Base text encoder: Google, google/flan-t5-base (Chung et al., 2022), Apache-2.0.
Why non-commercial: the training clips come from HD-VILA-100M, whose agreement allows research use only and bars using models trained on it in a commercial offering.
No warranty: the weights are provided as is (Section 5 of the legal code). Outputs are AI-generated and may be inaccurate, low-quality or offensive.

Credits

Wan2.2 VAE (Wan Team, Alibaba) and Flan-T5-Base (Google): the frozen video autoencoder and text encoder that made a tiny generator possible.
DiT (Peebles and Xie, "Scalable Diffusion Models with Transformers", 2023): the adaLN-zero transformer design we re-implemented.
Image-to-video by first-frame replacement: STIV and SANA-Video.
FVD: Unterthiner et al. (2018), with the evaluation settings and I3D detector of StyleGAN-V (Skorokhodov et al., 2022), against which we audited our code.
VBench and VBench++ (Huang et al.): the benchmark.
Data: VidGen-1M (Tan et al., 2024) and HD-VILA-100M (Xue et al., 2022, Microsoft); SkyTimelapse (Xiong et al.,
1. for Stage 1; CC12M (Changpinyo et al., 2021) for one Stage 5a arm.
TAEW2.2 (Ollin Boer Bohan, MIT): the tiny decoder behind the browser version.
Optimiser: MuonAdamW from nanochat (Andrej Karpathy) and modded-nanogpt (Keller Jordan).
A cheaper open peer: MiniDance trains a text-to-video generator from scratch on the same frozen Wan2.2 VAE for even less (about $2), on nature scenery only. Worth a look if scenery is all you need.

Citation

@misc{udgamlabs2026drishyatiny,
  title  = {Drishya Tiny: a 212M-parameter text- and image-to-video generator on the frozen Wan2.2 VAE and Flan-T5-Base},
  author = {Prasad, Jitesh K},
  year   = {2026},
  howpublished = {Udgam Labs},
  url    = {https://huggingface.co/udgamlabs/Drishya-Tiny}
}

Drishya Tiny by Udgam Labs · AI research from first principles

About Udgam Labs. Udgam Labs is an independent AI research lab. We work from first principles: before we build, we understand why things work, whether we are training a model, fine-tuning one or applying AI to a new problem. Udgam (उद्गम) is Sanskrit for origin. udgamlabs.com

Downloads last month: -

Safetensors

Model size

0.2B params

Tensor type

F32

Space using udgamlabs/Drishya-Tiny 1

Collection including udgamlabs/Drishya-Tiny

Drishya

Collection

212M video generator trained from scratch (frozen Wan2.2 VAE, Flan-T5-Base). 2-second 256×256 clips from text, or text plus a first frame. • 2 items • Updated about 3 hours ago