helloworld-interactor-lora-r32-2000

⚠️ This LoRA is a fine-tuned Derivative of LTX-2 and is distributed under the LTX-2 Community License (including its use-based restrictions, Attachment A). A complete copy of the license is included in this repository.

Rank-32 LoRA for Lightricks LTX-2.3 that adds camera-trajectory following from warp-video history conditioning — the self-distillation recipe of "HelloWorld: Enabling Socially Interactive Characters in Video World Models" (Ouyang, Liu, Chu, Zhang, Sato — Alaya Lab / The University of Tokyo), reimplemented independently before the authors' official code release. Paired with the training-free F-press cross-attention mask from the same paper, you get characters that perform at the camera inside a chosen time window while the shot follows a camera path you draw.

• Code, warp pipeline, Eq. 4 mask, training config, reproduction guide: https://github.com/scrappylabsai/helloworld-interactor
• What it is: LoRA rank 32, alpha 32, on all self-attention projection matrices of the video branch (attn1.*), trained 2000 steps @ LR 1e-4, flow matching, bf16, on 156 self-generated 1280×704×121 clips with recovered-trajectory warp videos as reference conditioning; camera text excluded from captions (paper §3.2, §5.2).
• Training cost: ~12 h on one rented RTX PRO 6000 (96 GB), peak 72.3 GB; whole project ~$43.50.

Results (mini-bench, 40 generated characters, base vs LoRA)

Metric	Judge	Base LTX-2.3	+ this LoRA	Paper
Performs in window (n=9 pairs)	Gemini, true-video	1/9 (11%)	7/9 (78%)	TimeAcc 81.7%
TimeAcc (n=40)	Qwen3.6-35B-A3B, frame grids	21%	44%	81.7%
Performs at all (n=40)	Qwen3.6-35B-A3B, frame grids	24/40	34/40	—

Qwen3.6-35B-A3B is the paper's judge model; our serving feeds it frame grids rather than raw video, which attenuates absolute scores for both columns (motion between sampled frames is invisible) — the true-video Gemini calibration on the same clips lands at the paper's number within small-n error. Caveats: n=40/n=9 (paper: 400); trained at 121 frames vs the paper's 241; our geometric CamCtrl metric is pending a pose-surveyor bugfix (qualitative camera control — dolly-in, orbit — confirmed side-by-side at fixed seed). Where the trained model missed the directed interaction (2/9 true-video pairs, both subtle low-amplitude gestures — a wink, a sliding object), the untrained baseline missed the same clips: misses degrade to baseline behavior rather than below it, and trained clips showed fewer artifacts (1/9 vs 3/9).

Example pairs (same seed, same prompt, same warp input)

Full-quality mp4s in examples/ — <id>__base.mp4 is stock LTX-2.3, <id>__lora.mp4 is +this LoRA. All characters fully generated.

Character	What was directed (interaction · window · camera)	Base	+LoRA
Brewer (taproom)	Raises a full glass toward the viewer in a toast, saying "Cheers!" · 1.5–2.5 s · dolly-in	base	LoRA
Baker (window)	Holds up a steaming loaf toward the viewer and winks · 1.5–2.5 s · static	base	LoRA
Clay snowman	Tips his tiny top hat toward the camera · 1.5–2.5 s · dolly-in	base	LoRA
Tin robot (workbench)	Winds its own key, then salutes the viewer stiffly · 1.5–2.5 s · dolly-in	base	LoRA
Plush octopus (bathtub)	Wiggles two tentacles toward the viewer in greeting · 1.5–2.5 s · orbit-left	base	LoRA
Mechanic (garage)	Wipes his hands on a rag and gives the viewer a thumbs-up · 3.0–4.0 s · pan-right	base	LoRA

Usage

With the repo's inference driver (LTX-2 packages installed + the repo's vendor patch applied — see the repo's docs/REPRODUCE.md):

python -m helloworld_ltx.infer_warp \
    --distilled-checkpoint-path models/ltx-2.3/ltx-2.3-22b-distilled-1.1.safetensors \
    --gemma-root models/gemma-3-12b \
    --spatial-upsampler-path models/ltx-2.3/ltx-2.3-spatial-upscaler-x2-1.1.safetensors \
    --image inputs/shot/first_frame.png 0 1.0 \
    --warp-npz inputs/shot/warp.npz \
    --scene "A lighthouse keeper on the gallery rail at dusk." \
    --interaction "He turns and waves at the viewer." \
    --quality "Photoreal, natural lighting." \
    --window 24 48 \
    --height 704 --width 1280 --num-frames 121 --frame-rate 24 \
    --lora helloworld-interactor-lora-r32-2000.safetensors 1.0 \
    --seed 42 --output-path out.mp4

--warp-npz comes from the repo's warp/warp.py (choose a trajectory) or warp/annotate.py (recover one from a video). --window is the F-press window in pixel frames at the output rate. Timing control is training-free; this LoRA adds the camera-following.

Dependency license note (Pi3X)

Generating the warp inputs uses the Pi3X / π³ geometry model (arXiv:2507.13347): its weights are CC BY-NC 4.0 (non-commercial) and are NOT included here or in the code repo. The LoRA weights themselves have no Pi3X dependency, but the as-shipped warp pipeline does — commercial users must substitute a permissively-licensed geometry backbone for warp generation.

Credit

All credit for the method to the HelloWorld authors (Alaya Lab / The University of Tokyo) — this is an unofficial reproduction from the paper text, published with full attribution. Watch https://github.com/AlayaLab/HelloWorld for their official release.

Built by ScrappyLabs.

Try it on your own productions — if you direct your own characters with this, we'd love to see the results: open a Discussion on the GitHub repo with your clips and what you directed.