elismasilva/ltx2.3_video_custom_blocks

LTX 2.3 Video custom modular blocks

Custom Modular Diffusers blocks for joint video/audio generation, with separate base and distilled block sets. Conditions are explicit records rather than monolithic-pipeline arguments.

prompt                        -> video + generated audio
prompt + image condition      -> image-guided video + generated audio
prompt + audio condition      -> audio-conditioned video + generated audio
prompt + image + audio        -> jointly conditioned video + generated audio

This repository contains code only. Model weights, input media and DDO are separate. The manifest selects the eight-step LTX2VideoDistilledBlocks workflow.

Loading & running

Requires the Diffusers fork providing diffusers.modular_pipelines.ltx2 (tested with 0.41.0.dev0), not an arbitrary stock Diffusers release. Use the compatible Torch/Transformers environment prepared for the selected checkpoint.

import torch
from diffusers import ModularPipelineBlocks
from diffusers.utils import encode_video

blocks = ModularPipelineBlocks.from_pretrained(
    "elismasilva/ltx2.3_video_custom_blocks",
    trust_remote_code=True,
    # revision="<reviewed-commit>",  # Pin executable code for reproducibility.
)
pipe = blocks.init_pipeline(
    pretrained_model_name_or_path="OzzyGT/LTX-2.3-Distilled-1.1",
)
pipe.load_components(
    names=["text_encoder", "tokenizer", "connectors", "transformer",
           "vae", "audio_vae", "vocoder", "scheduler"],
    torch_dtype=torch.bfloat16,
)
pipe.vocoder.to(dtype=torch.float32)
pipe.to("cuda")

state = pipe(
    prompt="a musician plays the guitar in an old western town",
    height=704,
    width=1280,
    num_frames=121,
    frame_rate=24.0,
    num_inference_steps=8,
    generator=torch.Generator(device="cuda").manual_seed(43),
)
encode_video(
    state.get("videos")[0], fps=24.0, output_path="ltx23_video.mp4",
    audio=state.get("audio")[0].float().cpu(),
    audio_sample_rate=pipe.vocoder.config.output_sampling_rate,
)

This minimal full-pipeline example assumes enough memory for all components. For an 8 GiB GPU, use staged component loading/offloading instead of moving the entire pipeline to CUDA. The blocks do not choose an allocator or residency policy.

Image-to-video

Use condition classes from the same trusted module as the loaded blocks:

from importlib import import_module
from diffusers.utils import load_image

code = import_module(type(blocks).__module__)
image = load_image("input.png")
state = pipe(
    prompt="the musician starts playing and singing",
    image_conditions=[code.LTX2ImageCondition(image, frame=0, strength=1.0)],
    height=704, width=1280, num_frames=121, frame_rate=24.0,
    num_inference_steps=8,
    generator=torch.Generator(device="cuda").manual_seed(43),
)

Audio conditioning

Load a waveform with shape [channels, samples] and resample it to pipe.audio_vae.config.sample_rate. Pass it as an explicit timeline condition:

import torchaudio

waveform, rate = torchaudio.load("conditioning.wav")
target_rate = pipe.audio_vae.config.sample_rate
if rate != target_rate:
    waveform = torchaudio.functional.resample(waveform, rate, target_rate)
state = pipe(
    prompt="a musician sings in sync with the supplied voice",
    image_conditions=[code.LTX2ImageCondition(image, frame=0)],
    audio_conditions=[code.LTX2AudioCondition(waveform, start_time=0.0, end_time=5.0)],
    height=704, width=1280, num_frames=121, frame_rate=24.0,
    num_inference_steps=8,
)

One source audio timeline is shared across the output batch; independent audio per prompt is not implemented. 121 frames at 24 fps follows the causal 8n+1 frame grid used for the nominal five-second workload.

Blocks

The distilled workflow is composed of ten top-level modular stages:

1. inputs validates dimensions and supported workflow controls.
2. text_encoder reads prompt hidden states without LM-head logits.
3. connectors adapts text embeddings for video/audio conditioning.
4. video_encoder encodes optional image/video timeline anchors.
5. audio_encoder encodes optional audio timeline conditions.
6. reference_encoder encodes optional compatible IC-LoRA references.
7. prepare creates latents, masks, timesteps and positional coordinates.
8. denoise runs joint video/audio denoising with separate guiders/schedulers.
9. trim removes reference tokens before decoding.
10. decode produces video frames and audio waveforms.

Each stage can be extracted from blocks.sub_blocks and initialized separately for memory-sensitive execution. The base LTX2VideoBlocks class is exported by the same trusted entry module; configure its guiders explicitly for the selected base checkpoint. Neutral defaults do not select a checkpoint-specific recipe.

Components

The blocks expect these components from the chosen LTX-2.3 checkpoint:

• text_encoder, tokenizer, connectors
• independent local-code transformer
• vae, audio_vae, vocoder
• scheduler, plus video/audio guider components

Code comes from this repository; weight configuration comes from the checkpoint. No duplicate hardcoded model index or model tensors are stored here.

Notes

The local transformer preserves checkpoint keys and includes opt-in activation chunking, phase-local modulation, compact timesteps, residual/RoPE reuse and experimental compute callbacks. These are LTX execution features, not generic DDO FP8 support. They do not silently enable themselves during Hub loading. Optional feed-forward FP8 conversion reuse is a bounded compute experiment; TorchAO/Diffusers owns quantization math, and DDO owns physical sources, transfers and lifetime when the host explicitly selects its native route.

The staged Windows host has selected BF16/FP8 generation, conditioned ten-second and local-transformer migration evidence. This Hub packaging has CPU loading checks only until separately approved CUDA validation; do not transfer historical performance/quality claims to arbitrary models, recipes or environments.

Diffusers merge compatibility

Guidance imports support both the pinned legacy modular location and the new diffusers.guiders.LTX2Guidance export. The local transformer accepts optional video_keyframes_mask with the upstream embedding rule, only when the model configuration enables keyframe embeddings. LTX 2.3 defaults and positional forward arguments remain unchanged. This is not a DFR/LTX-2.5 workflow. Shared loader/revision changes remain in Diffusers, not duplicated here. CPU compatibility checks do not grant CUDA or performance acceptance for a new Diffusers version.

Video-only inference, sliding-window/legacy audio timing, step callbacks, two-stage refinement and API media handling are not supported workflows here. IC-LoRA references require a compatible adapter already loaded by the host. Reference strength, audio identity guidance and streaming decode are explicit experimental features, not full parity with every monolithic pipeline option.

trust_remote_code=True executes repository code: review/pin a revision. local_files_only=True requires that exact code revision and weights already cached; first use must download code. Preserve the Apache-2.0 copyright notices in the Diffusers-derived modules. Model weights retain their own licenses.