yuvraj108c/LTX-2.5-22b-IC-LoRA-BBox-Control

LTX 2.5 22B IC-LoRA — Bounding Box Control

An In-Context LoRA (IC-LoRA) for LTX 2.5 which adds support for bounding box control and regional prompting.

Use in ComfyUI via ComfyUI-LTX-BBox-Animator

Examples

Prompt

8 BBoxes

Prompt

9 BBoxes

Prompt

9 BBoxes

Prompt

10 BBoxes

Prompt

15 BBoxes

Contents

• Highlights
• Model Details
• Prompting Guide
• Control Signal
• How Regional Conditioning Works
• Training Dataset
• Bounding Box Dataset Generation Pipeline
• Research Process and Experiments
• Training Configuration
• Validation and Expected Behavior
• Limitations
• Future Work: Occlusion and Depth Ordering
• Support
• Acknowledgments

Highlights

• Animated spatial control: Objects follow keyframed bounding-box positions and sizes.
• Independent regional prompts: Every object has its own description and animated spatial mask.
• Prompt swapping: Exchange object descriptions while leaving the control video unchanged.
• Shared scene conditioning: A separate global prompt controls lighting, environment, camera framing, and scene context.
• Interactive ComfyUI workflow: Draw, animate, and describe objects with the companion ComfyUI-LTX-BBox-Animator custom nodes.
• Beyond the training object count: The architecture supports more than three regions, although reliability decreases as scenes become denser and VRAM requirements increase.

Model Details

Property	Value
Base model	Lightricks/LTX-2.5, 22B
Adaptation	In-context LoRA with animated reference-video conditioning
Training framework	Lightricks LTX-2 trainer, extended with regional denoising conditioning
LoRA rank	32
Trainable parameters	163,577,856
Training resolution	768 × 448
Training clip length	121 frames
Training dataset	152 manually reviewed videos
Dataset source	Pexels; free-to-use stock footage under the Pexels License
Unique annotated object tracks	171
Multi-object training videos	15
Overlapping training examples	0
Precision	bfloat16
Training steps	3,000 steps; learning rate of 2e-4; Linear
Approximate dataset passes	19.74 passes at 3,000 steps
Reference latent downscale	1
Regional/global prompt weights	0.85 / 0.15
Regional/full-frame loss weights	0.60 / 0.40
Multi-object/single-object sampling	0.35 / 0.65

Prompting Guide

Follow the same prompt structure used during training:

style: [visual style, lighting, camera framing, and colors]

scene: [environment], where [subjects, their positions, and their shared action].

The where clause connects the environment to the people, animals, or objects occupying the scene. Mentioning subject count and approximate initial positions can improve placement and reduce unwanted additional characters.

Global Style and Scene

style: High-contrast cinematic street photography with crisp afternoon sunlight,
sharp shadows, steady eye-level framing, and natural colors.

scene: A modern glass office plaza with reflective skyscraper windows and polished
concrete, where a businessman walks on the left side and a businesswoman walks
on the right side.

Regional Prompt: Object 1

A businessman wearing a tailored charcoal-gray three-piece suit, a white shirt,
and polished black leather shoes, walking on the left side of the frame.

Regional Prompt: Object 2

A businesswoman wearing a flowing scarlet-red silk dress and matching red
high-heeled shoes, walking on the right side of the frame.

To swap the subjects, exchange the regional prompts while keeping the boxes and control video unchanged. If global text contains position-specific identities, update those identities as well so the global and regional prompts do not contradict each other.

Prompting Recommendations

• Keep lighting, visual style, environment, and overall action in the global prompt.
• Mention the intended number and general type of objects in the where clause.
• Keep detailed clothing, materials, colors, and identity inside the corresponding regional prompt.
• Describe initial positions or motion when they match the animated trajectory.
• Avoid duplicating detailed object descriptions globally; unrestricted object details can leak between regions.
• Choose box dimensions that resemble the intended subject's overall size and aspect ratio.

Control Signal

The IC-LoRA receives a reference video with:

• A solid black background.
• Hollow white bounding boxes with 2 px borders.
• A rounded white center marker with a 2 px radius.
• A 5-frame, 1 px motion trail clipped to the interior of each box.

All boxes use the same appearance. Colors displayed by the ComfyUI editor are only interface aids and are not present in the generated control video.

Why the Controls Look This Way

The control is intentionally sparse: it should communicate an object's spatial extent and movement without competing with the prompt or prescribing the object's appearance.

Visual element	Design	Purpose
Background	Solid black	Keeps the conditioning signal clean and prevents unintended background colors, textures, or scene content from becoming part of the control.
Bounding box	Hollow white rectangle with a 2 px border	Specifies position, approximate size, and aspect ratio while leaving the interior visually unconstrained for the generated object.
Shared box color	White for every object	Avoids accidental correlations between control colors and clothing or object colors; identity is determined by regional prompts and masks instead.
Center dot	Rounded white marker with a 2 px radius	Marks the tracked object's current center explicitly, providing a compact positional anchor even when the surrounding box changes size.
Motion trail	1 px line covering up to five recent frames	Encodes the recent path behind the center dot, giving a short visual cue for movement direction and approximate speed.
Trail clipping	Restricted to the current bounding box	Prevents one object's trajectory from spilling into neighboring controls or creating unrelated white marks outside the object region.
Fixed stroke sizes	Constant pixel dimensions	Keeps control geometry visually consistent across clips rather than making borders and markers grow with bbox size.

The center marker and trail answer slightly different questions: the dot indicates where the object is now; the trail indicates where it came from. A still object therefore has a stationary center marker, while a moving object leaves a short directional trace.

A longer displacement between recent center positions produces a longer visible trace; a short displacement produces a compact trace. This is an intentionally lightweight motion hint, not a separate velocity label, optical-flow map, or proven guarantee of better temporal consistency. Its primary practical requirement is that training and inference use the same control rendering.

Why Not Filled or Colored Boxes?

Filled controls make the object's entire area an artificial visual input, increasing the chance that control colors or block shapes influence the generated subject. Hollow outlines preserve the layout signal while exposing less synthetic structure inside the region.

Earlier colored controls, checkerboard/striped identifiers, caption tags, and balanced color permutations did not establish reliable semantic binding on their own. The successful mechanism was spatially restricted regional conditioning, so the final control removes unnecessary identity coding and uses one consistent white outline for every object.

For object i at frame t, its bounding box is defined as:

$$b_{i,t}=(x_{i,t}^{(1)},y_{i,t}^{(1)},x_{i,t}^{(2)},y_{i,t}^{(2)})\mathrm{.}b\_\{i,t\}\; =\; (x\_\{i,t\}^\{(1)\},\; y\_\{i,t\}^\{(1)\},\; x\_\{i,t\}^\{(2)\},\; y\_\{i,t\}^\{(2)\}).$$

Its full regional mask is

The control video and regional masks play different roles:

$$\underset{\text{white bbox reference}}{\underbrace{C_t}}⟶\text{where objects move},\underset{\text{regional prompt + mask}}{\underbrace{(p_i,M_{i,t})}}⟶\text{which object appears there}\mathrm{.}\backslash underbrace\{C\_t\}\_\{\backslash text\{white\; bbox\; reference\}\}\; \backslash longrightarrow\; \backslash text\{where\; objects\; move\},\; \backslash qquad\; \backslash underbrace\{(p\_i,\; M\_\{i,t\})\}\_\{\backslash text\{regional\; prompt\; +\; mask\}\}\; \backslash longrightarrow\; \backslash text\{which\; object\; appears\; there\}.$$

Using the LoRA without the regional-conditioning node can still encourage objects to occupy the boxes, but it does not reliably bind a particular object description to a particular box.

How Regional Conditioning Works

Independent Text Encoding

The global scene prompt and each object prompt are encoded separately:

$$E_g=\mathrm{TextEncoder}(p_g),E_i=\mathrm{TextEncoder}(p_i)\mathrm{.}E\_g\; =\; \backslash operatorname\{TextEncoder\}(p\_g),\; \backslash qquad\; E\_i\; =\; \backslash operatorname\{TextEncoder\}(p\_i).$$

The weighted text context passed into the denoising transformer is

$$E=[{\lambda }_g{E}_g\mathrm{\parallel }{\lambda }_r{E}_1\mathrm{\parallel }\cdots \mathrm{\parallel }{\lambda }_r{E}_N],E\; =\; \backslash left[\backslash lambda\_g\; E\_g\backslash ;\backslash Vert\backslash ;\backslash lambda\_r\; E\_1\backslash ;\backslash Vert\backslash ;\backslash cdots\backslash ;\backslash Vert\backslash ;\backslash lambda\_r\; E\_N\backslash right],$$

The global prompt weight is 0.15, the regional prompt weight is 0.85, and the vertical separators denote text-token concatenation.

The global prompt intentionally contains only the shared style: and scene: description. Detailed object descriptions are not repeated globally, because doing so would allow object-specific information to leak into unrelated regions.

Spatially Restricted Cross-Attention

For a generated video latent token at position q = (t, x, y), global text tokens remain visible everywhere. Object-prompt tokens are visible only when q belongs to that object's downsampled mask:

The transformer therefore computes masked text cross-attention as

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{Q{K}^{\mathrm{\top }}}{\sqrt{d}}+A)V\mathrm{.}\backslash operatorname\{Attention\}(Q,K,V)\; =\; \backslash operatorname\{softmax\}\backslash left(\backslash frac\{QK^\backslash top\}\{\backslash sqrt\{d\}\}\; +\; A\backslash right)V.$$

Here, M_tilde_i represents the object mask resampled to the video latent grid. Spatial downsampling uses conservative pooling so that a latent cell remains assigned to an object whenever its footprint intersects that object's mask.

This conditioning is applied to the actual denoising forward pass. It is not a detached classifier, post-processing step, color-matching heuristic, or separate auxiliary accuracy objective.

Region-Balanced Denoising Loss

Let v_pred(q) be the predicted video target at latent position q, v(q) the training target, Omega the valid generated-video positions, and R the union of all object regions.

The ordinary full-frame loss is

$${\mathcal{L}}_{\mathrm{f}\mathrm{u}\mathrm{l}\mathrm{l}}=\frac{1}{\mathrm{\mid }\mathrm{\Omega }\mathrm{\mid }}\sum _{q\in \mathrm{\Omega }}{\parallel {\widehat{v}}_{\theta }(q)-v(q)\parallel }_2^2\mathrm{.}\backslash mathcal\{L\}\_\{\backslash mathrm\{full\}\}\; =\; \backslash frac\{1\}\{|\backslash Omega|\}\backslash sum\_\{q\backslash in\backslash Omega\}\; \backslash left\backslash |\backslash hat\{v\}\_\backslash theta(q)-v(q)\backslash right\backslash |\_2^2.$$

The region-normalized loss is

$${\mathcal{L}}_{\mathrm{r}\mathrm{e}\mathrm{g}\mathrm{i}\mathrm{o}\mathrm{n}}=\frac{1}{\mathrm{\mid }R\cap \mathrm{\Omega }\mathrm{\mid }}\sum _{q\in R\cap \mathrm{\Omega }}{\parallel {\widehat{v}}_{\theta }(q)-v(q)\parallel }_2^2\mathrm{.}\backslash mathcal\{L\}\_\{\backslash mathrm\{region\}\}\; =\; \backslash frac\{1\}\{|R\backslash cap\backslash Omega|\}\backslash sum\_\{q\backslash in\; R\backslash cap\backslash Omega\}\; \backslash left\backslash |\backslash hat\{v\}\_\backslash theta(q)-v(q)\backslash right\backslash |\_2^2.$$

The final training objective is

$$\boxed{{\displaystyle {\mathcal{L}}_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}=0.60{\mathcal{L}}_{\mathrm{r}\mathrm{e}\mathrm{g}\mathrm{i}\mathrm{o}\mathrm{n}}+0.40{\mathcal{L}}_{\mathrm{f}\mathrm{u}\mathrm{l}\mathrm{l}}}}\backslash boxed\{\; \backslash mathcal\{L\}\_\{\backslash mathrm\{total\}\}\; =\; 0.60\backslash ,\backslash mathcal\{L\}\_\{\backslash mathrm\{region\}\}\; +\; 0.40\backslash ,\backslash mathcal\{L\}\_\{\backslash mathrm\{full\}\}\; \}$$

Normalizing each term independently prevents smaller object regions from disappearing into a background-dominated frame average, while the full-frame component preserves scene coherence and overall visual quality. Both losses come from the same denoising prediction; no second transformer forward pass is required.

Balanced Sample Selection

Only 15 of 152 training videos contain multiple objects. Uniform sampling would expose the model to multi-object examples only

$$\frac{15}{152}=9.87\mathrm{\%}\backslash frac\{15\}\{152\}\; =\; 9.87\backslash \%$$

of the time. A class-balanced sampler increases their target sampling probability to 35%:

This produces approximately

$$\frac{0.35}{15\mathrm{/}152}\approx 3.55\backslash frac\{0.35\}\{15/152\}\backslash approx\; 3.55$$

times more multi-object training exposure than uniform sampling, without copying or modifying the original dataset.

Why Mostly Single-Object Training Still Scales

The dataset is 90.13% single-object clips, yet inference can compose substantially more independent regions. This is possible because the LoRA and the regional-conditioning mechanism learn complementary responsibilities:

1. Single-object examples teach the IC-LoRA the reusable primitive: a hollow moving box indicates where a subject should appear and how its position changes over time.
2. The 15 multi-object examples, sampled at a target rate of 35%, expose the model to multiple simultaneous boxes and independent object descriptions.
3. At inference, the same regional attention operation is repeated for each additional prompt/mask pair. Its structure does not depend on a fixed number of colors, object tags, or training classes.
4. LTX's pretrained knowledge supplies the underlying ability to compose people, animals, props, and environments into a shared scene.

The result is compositional generalization: the model learns box following primarily from simple examples, while the inference pipeline applies that learned behavior independently across multiple regions. Layouts with up to ten objects have shown useful results despite the training set containing at most three annotated objects in any single video.

This is observed generalization, not a guarantee. Dense arrangements, small boxes, unusual object shapes, and overlapping trajectories remain more difficult than the training distribution.

Training Dataset

The dataset contains 152 original, manually reviewed, non-overlapping stock video clips sourced from Pexels under the Pexels License.

Each clip is exactly 121 frames, for a total of 18,392 analyzed frames. There are no flipped duplicates, unmatched coordinate files, invalid annotations, or overlapping object boxes.

Why Exclude Overlap?

The training dataset intentionally excludes overlapping bounding boxes. Earlier experiments showed that ambiguous overlaps made it harder to learn stable object-to-region associations and increased color/identity bleeding.

At inference, boxes may still cross or overlap, but foreground/background ordering and occlusion are not explicitly supervised. The base model must resolve those interactions itself; correct ordering is not guaranteed.

Dataset Construction

The dataset was assembled from Pexels stock videos containing animals, dancing, running people, and small groups. Each candidate was paired with tracked per-frame bounding-box coordinates, standardized to 768 × 448 and 121 frames, rendered as an annotated review overlay, and manually inspected.

Videos were excluded when their coordinates were missing, tracking quality was poor, they contained fewer than 121 frames, or their bounding boxes overlapped. The retained clips were converted into hollow-white control videos, full per-object animated masks, a scene-level global caption, and one independent caption per annotated object.

No flipped duplicates were included. A small number of frames—160 out of 18,392, or 0.87%—contain no visible box because an object temporarily leaves the frame.

Bounding Box Dataset Generation Pipeline

The released dataset is the final, deliberately simplified output of a broader computer-vision and video-processing pipeline. Earlier research versions additionally included depth estimation, occlusion-aware rendering, color balancing, horizontal flips, and substantially larger manifests; those experiments informed the final design but are not properties of the 152-video release dataset.

1. Licensed Stock Video Collection

Landscape stock videos were collected from the Pexels video library using targeted searches for animals, dancing, running people, and small groups. Pexels permits free use and modification under the Pexels License;

An automated ingestion and cleanup stage rejected corrupt clips, portrait-oriented footage, unusable durations, and clips without enough frames. Early research datasets were initially normalized around 1280 × 720 before the final retained examples were resized to 768 × 448 and trimmed to exactly 121 frames.

2. Object Detection and Track Initialization

Source videos were scanned using YOLO26x to identify candidate subjects. Instead of initializing from an arbitrary first frame, the tracking pipeline selected a strong pivot frame containing clearly visible subjects, filtered duplicate or unwanted nested detections, and retained the most useful objects.

The resulting boxes seeded Segment Anything Model 3 (SAM 3) video tracking. Tracks were propagated both forward and backward from the pivot frame so that an object preserved the same track identity throughout its visible trajectory.

Additional filtering removed duplicate tracks, brief transient detections, unstable boxes, and poor subject coverage. This produced per-frame object coordinates associated with persistent track IDs.

3. Manual Bounding Box Review

The tracked boxes and IDs were rendered over the source footage and reviewed in a dedicated browser-based interface. Each clip could be approved or rejected after inspecting frame-to-frame stability, object coverage, identity continuity, and unwanted overlaps.

Rejected samples were removed consistently across video, coordinate, caption, and overlay directories. For the final release dataset, clips containing any overlapping bounding boxes were also excluded, prioritizing clean object-to-region supervision over explicit occlusion training.

4. Accurate Coordinate Remapping

After source videos were resized, every bounding box was remapped using the exact horizontal and vertical resize ratios:

$$x^{\mathrm{\prime }}=x\cdot \frac{768}{W_{\mathrm{s}\mathrm{o}\mathrm{u}\mathrm{r}\mathrm{c}\mathrm{e}}},y^{\mathrm{\prime }}=y\cdot \frac{448}{H_{\mathrm{s}\mathrm{o}\mathrm{u}\mathrm{r}\mathrm{c}\mathrm{e}}}\mathrm{.}x\text{'}\; =\; x\; \backslash cdot\; \backslash frac\{768\}\{W\_\{\backslash mathrm\{source\}\}\},\; \backslash qquad\; y\text{'}\; =\; y\; \backslash cdot\; \backslash frac\{448\}\{H\_\{\backslash mathrm\{source\}\}\}.$$

The same transformation is applied to both corners of every box. Coordinates are clipped to the visible image bounds when rendered, while the underlying animated trajectory can still represent subjects entering or leaving the frame.

The final coordinate files retain object identities and bounding-box positions only. Earlier experimental depth annotations are not required by the released training pipeline.

5. Hollow-White Control Rendering

Each approved trajectory is rendered onto a black canvas as a thin, hollow white rectangle. A small center marker and clipped motion trail provide additional temporal direction cues without extending outside the corresponding box.

All boxes intentionally use the same white appearance. Object identity does not depend on color, numeric IDs, caption order, or trigger words: the control specifies where, while regional text conditioning specifies what.

6. Per-Object Animated Mask Generation

For every tracked object, the same coordinates are converted into a separate full-size binary video mask. Pixels inside the object's bounding box are white; all other pixels are black.

The hollow control and filled mask are complementary: the control teaches the diffusion model the desired layout, while the filled mask restricts the object's independently encoded prompt to its own spatial region.

7. Structured Captioning and Semantic Review

Earlier captioning runs used Qwen3-Omni-30B-A3B-Thinking through schema-guided inference to generate visual descriptions and validate their agreement with tracked subjects. The final release format separates each sample into:

• A global caption containing only the shared style: and scene: description.
• One object caption for each persistent track ID.
• The corresponding animated regional mask for that same track.

Legacy color tags such as [obj_1] or [red_box] were removed. Track IDs are retained as dataset bookkeeping so the correct description can be paired with the correct mask, but they are not exposed as prompt tokens required for inference.

8. Synchronized Manifest and Precomputation

The final manifest includes a sample only when all required files exist and agree: source video, control video, scene caption, object captions, tracked coordinates, and one mask for every object.

Preprocessing then caches target-video latents, control-video reference latents, global text embeddings, independent object-prompt embeddings, and latent-resolution regional masks. These cached assets are consumed directly by the weighted regional-denoising training strategy.

Earlier Depth-Aware Research Pipeline

Earlier, larger experimental datasets also used Depth Anything V2 to estimate relative foreground/background order, propagated reliable depth values across occluded frames, applied temporal smoothing to reduce ordering flicker, rendered depth-aware overlapping controls, balanced ten bbox colors, and generated horizontally flipped copies.

Those steps were useful for investigating occlusion and identifying failure modes, but were intentionally excluded from the final release dataset after repeated experiments showed that ambiguous overlaps and color-coded identity could destabilize prompt-to-object binding.

Research Process and Experiments

The released approach was reached through a sequence of experiments across control rendering, caption design, dataset balancing, auxiliary losses, attention routing, regional inference, and denoising-time conditioning. Many intermediate approaches produced convincing proxy metrics or plausible-looking frames but failed the controlled prompt-swap test.

Experiment Summary

Experiment	What was tested	Observed result
Default flexible IC-LoRA	One global caption and a bbox control video	Learned to fill boxes, but object-to-box assignment remained ambiguous.
Positional caption wording	Descriptions containing left/right and motion direction	Improved placement through spatial language, without establishing tag-based identity binding.
Color-coded tags	[obj_1], [red_box], [blue_box], and per-box colors	Tags could be removed with little change; scene priors and clothing/color correlations dominated.
Tag permutations	Reassigned tags and regenerated corresponding controls	Balanced tag identities but did not make globally encoded captions spatially binding.
Caption-order permutations	Reordered object descriptions and balanced caption positions	Reduced ordering shortcuts without consistently enabling prompt swaps.
Hollow versus filled controls	Compared outlined boxes against solid colored regions	Changed visual conditioning and bleed behavior, but did not solve object binding.
Pattern-coded controls	Checkerboard and diagonal-striped bbox identifiers	Made controls visually distinct; identity still required spatial prompt restriction.
Numeric and combined identifiers	Explored numbered labels and combined trigger-style concepts	Did not address the absence of explicit token-to-region visibility.
Boosted bbox loss	Increased denoising loss inside bbox regions	Strengthened local reconstruction but did not specify which prompt belonged there.
Overlap-only loss	Applied additional reconstruction weight inside overlapping areas	Sometimes improved occlusion in overlap-trained prototypes, but could destabilize identity.
Bbox plus overlap loss	Combined region and overlap weighting	Did not reliably prevent identity swaps or appearance bleeding.
Depth-filtered datasets	Compared changing-depth, fixed-depth, and overlap-filtered subsets	Some early checkpoints handled crossings well, but behavior was inconsistent over longer training.
Occlusion rendering variants	Hidden, faded, grayscale, colored, and depth-resolved overlaps	Exposed tradeoffs between visible geometry, missing controls, and cross-object color bleed.
Swapped-caption ranking	Two transformer passes for correct and swapped global captions	Produced measurable ranking gaps but increased step time and did not reliably improve samples.
Three-pass regional reconstruction	Separate full-frame, object-1, and object-2 denoising passes	Increased training time substantially; lower regional losses did not guarantee correct identity.
Random single-pass regional training	Randomly selected global, object-1, or object-2 reconstruction per step	Reduced compute but still lacked reliable prompt-to-region binding.
Tag-to-control alignment	Aligned isolated tag embeddings with control-region features	Auxiliary accuracy reached 100% while generated objects still appeared in the wrong boxes.
Object-to-control alignment	Aligned full object descriptions with control-region features	Solved the proxy classification more easily than the actual denoising task.
Joint counterfactual binding	Compared joint tag/object features against matched and mismatched regions	Several sampled blocks showed high accuracy without reliable visual swaps.
Global-caption binding	Extracted object information from the actual global caption embedding	Improved measured alignment but left generation-dependent attention insufficiently constrained.
All-block attention routing	Applied routing supervision across all 48 transformer blocks	Accuracy improved, but the separate routing objective could remain weak or decoupled from generation.
Routing loss-balance diagnostics	Logged default loss, routing loss, weighted contribution, and block accuracy	Revealed that apparently useful accuracy did not demonstrate causal control of the final output.
Tagless object routing	Removed explicit tags and routed isolated object descriptions directly	Removed unnecessary tag dependence but still relied on auxiliary supervision.
Variable-object routing	Extended routing from two objects to arbitrary object counts	Revealed single-object, tiny-mask, ambiguous-token, and overlap-sampling edge cases.
Regional validation-only inference	Patched inference to expose each object prompt only inside its mask	Produced the first repeatable prompt swaps and established the effective binding mechanism.
Regional denoising with full global captions	Applied masked object prompts while retaining object details in global text	Global leakage caused duplicated subjects, bleeding, and incorrect swaps.
Visible-only versus full object masks	Compared depth-subtracted masks with complete bbox masks	Full masks preserved object identity and trajectory more consistently.
Scene-only global regional denoising	Restricted global text to style/scene and applied masked object prompts	Matched the inference structure and improved object assignment.
Weighted regional denoising	Combined masked prompts, regional/full-frame loss, and balanced sampling	Became the final released training strategy.

Color-Coded Boxes and Caption Tags

Early experiments assigned different colors to different boxes and described their subjects with tags such as [obj_1], [red_box], and [blue_box].

The generated videos initially looked convincing: objects appeared inside plausible boxes. Controlled prompt-swapping tests revealed that the model was not consistently using the tags. Instead, it relied on scene priors, object size, positional language, and sometimes correlations between box colors and clothing colors.

Removing the tags often produced nearly identical outputs. This showed that apparent bbox adherence did not establish a true caption-to-box association.

Balanced Tag and Caption Permutations

The dataset was expanded with permutations of object tags, caption order, and box/color mappings to remove positional shortcuts and balance tag distributions.

Although the distributions became nearly uniform, the model still processed the caption as global text and could ignore the intended token-to-box relationship. Swapping descriptions or reordering prompts did not consistently swap the generated subjects.

Alternative Control Encodings

Several control representations were evaluated:

• Colored hollow bounding boxes.
• Filled colored bounding boxes.
• Diagonal-striped and checkerboard patterns.
• Numeric or combined trigger-style identifiers.
• Hidden, faded, or color-filled overlap regions.

Distinct patterns sometimes improved the visibility of individual controls, but they did not reliably solve the underlying problem: a global caption does not enforce which object description belongs to which spatial region.

Reconstruction and Occlusion Loss Experiments

The first loss-based approaches modified the default denoising objective by increasing the weight of bbox pixels, overlap pixels, or both. Additional experiments used separate reconstruction passes for the complete frame and each object region, followed by a cheaper randomized one-pass variant.

These methods could improve local visual fidelity or produce isolated successful occlusions. However, a spatially weighted reconstruction loss still does not identify which sentence describes the object being reconstructed. If all object descriptions remain globally visible, the model can minimize loss while assigning subjects according to scene priors rather than the intended boxes.

Occlusion-focused versions were trained on earlier datasets that contained overlapping boxes. Their strongest examples are research prototypes and are not evidence that the released overlap-free model learned explicit depth ordering.

Swapped-Caption Contrastive Ranking

A counterfactual training strategy performed separate denoising forwards for the correct caption and a caption with object descriptions swapped. The goal was to make the correct assignment produce a lower reconstruction error:

$${\mathcal{L}}_{\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{k}}=\max (0,m+{\mathcal{L}}_{\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}}-{\mathcal{L}}_{\mathrm{s}\mathrm{w}\mathrm{a}\mathrm{p}\mathrm{p}\mathrm{e}\mathrm{d}})\mathrm{.}\backslash mathcal\{L\}\_\{\backslash mathrm\{rank\}\}\; =\; \backslash max\backslash left(0,\backslash ;\; m\; +\; \backslash mathcal\{L\}\_\{\backslash mathrm\{correct\}\}\; -\; \backslash mathcal\{L\}\_\{\backslash mathrm\{swapped\}\}\backslash right).$$

Although training logs reported differences between the correct and swapped losses, the objective roughly doubled the cost per step and did not consistently produce correct swaps at inference. Global captions still allowed the model to exploit non-binding shortcuts.

Tag-to-Control and Object-to-Control Alignment

Separate auxiliary objectives compared isolated tag embeddings or complete object-caption embeddings against features pooled from the corresponding control regions.

Tag-to-control alignment often reached near-perfect training accuracy, followed by object-to-control alignment reaching similarly high accuracy. Generated samples still failed prompt swaps. The learned discriminator could identify matched pairs without forcing the actual denoising prediction to depend on that association.

Joint Counterfactual and Global-Caption Binding

Joint counterfactual binding combined object and tag information and compared correct versus incorrect region assignments at selected transformer blocks, including early, middle, and late layers.

Global-caption binding then switched from separately encoded tag-only features to object features grounded in the actual full-caption context. Both approaches produced encouraging block-level losses, accuracy, and margins, but those proxy measurements did not guarantee that the generated video used the intended assignment.

Transformer-Wide Attention Routing

Attention-routing supervision was subsequently expanded across all 48 transformer blocks. Diagnostics reported routing accuracy, block-specific margins, sampled overlap coverage, and the contribution of routing loss relative to the ordinary diffusion loss.

Further versions removed tags entirely, supported arbitrary numbers of objects, oversampled multi-object examples, skipped single-object samples when contrastive routing was undefined, and handled masks that disappeared or became ambiguous at latent resolution.

This improved observability and sometimes improved regional inference, but the routing objective remained auxiliary: a model could optimize routing scores without sufficiently changing the denoising behavior that determines the output video.

Regional Inference Proof of Concept

The decisive experiment modified inference itself. Instead of sending one unrestricted caption to every spatial position, it encoded a global scene prompt and each object prompt separately, then used animated masks to decide which text each video token could attend to.

Holding the same control video and seed fixed while swapping two regional prompts produced the first repeatable subject swaps. The experiment also showed that object tags were unnecessary: the binding was defined directly by the object prompt and its mask.

Early integrations also established practical requirements: use the true generated-video latent grid, exclude concatenated reference tokens from the regional mask, process object embeddings through LTX's text connectors, and preserve globally visible scene context. Incorrect grid sizes or unprocessed text embeddings caused severe visual artifacts despite apparently correct masks.

Training and Inference Alignment

After regional inference succeeded, the same conditioning structure was incorporated into the actual training forward pass. An intermediate version still retained the original full caption globally, allowing object descriptions to leak outside their masks.

The final correction separated a scene-only global caption from independently encoded object descriptions and used full animated bbox masks. This matched the inference-time conditioning path and avoided relying on a separate proxy classification loss.

Final Approach

The effective change was to make the actual denoising forward pass use independently encoded object prompts restricted to their corresponding animated masks, during both training and inference.

This directly aligns the training condition with the inference condition:

Global scene prompt → visible across the entire frame
Object prompt 1     → visible only inside animated mask 1
Object prompt 2     → visible only inside animated mask 2
Object prompt N     → visible only inside animated mask N
White control video → supplies shared bbox positions and motion

Once regional prompt visibility was enforced inside the denoising transformer, prompt swaps became observable in generated videos without requiring colored boxes or object tags.

Training Configuration

lora:
  rank: 32

optimization:
  learning_rate: 2.0e-4
  steps: 3000
  batch_size: 1
  gradient_accumulation_steps: 1
  scheduler_type: linear

acceleration:
  mixed_precision_mode: bf16

training_strategy:
  name: flexible
  binding_conditions_dir: binding_conditions

  regional_probability: 1.0

  regional_prompt_weight: 0.85
  global_prompt_weight: 0.15

  regional_loss_weight: 0.60
  full_frame_loss_weight: 0.40

  multi_object_sampling_weight: 0.35
  single_object_sampling_weight: 0.65

With regional_probability: 1.0, every training batch uses the same global-plus-masked-object conditioning structure used for regional inference. This removes the training/inference mismatch of training with one unrestricted caption and expecting isolated object prompts only at inference.

Recommended Inference Settings

Setting	Starting value
Resolution	768 × 448; matches the training bucket
Frame count	121
Frame rate	24 fps
LoRA strength	1.0; adjust to preference
IC-LoRA reference downscale	1
Regional prompt weight	0.85
Global prompt weight	0.15
Per-object regional strength	1.0

Validation and Expected Behavior

The most direct validation is a counterfactual prompt swap:

1. Generate a video with object prompt A assigned to box 1 and object prompt B assigned to box 2.
2. Keep the seed, control video, scene, and box trajectories unchanged.
3. Exchange only the regional object prompts.
4. The corresponding subjects should exchange locations while their trajectories remain attached to the original boxes.

Successful results depend on the combined system: the bbox IC-LoRA supplies trajectory control, while animated regional conditioning binds each independent object prompt to its spatial region.

Example Scenarios

Useful examples include:

1. One moving object following a keyframed bbox.
2. Two independently controlled subjects.
3. The same two subjects with their regional prompts swapped.
4. A non-human replacement, such as changing a person into a robot.
5. A dense scene with several independently prompted regions.
6. An earlier experimental crossover demonstrating the potential for future occlusion support.

Limitations

• No explicit occlusion training: All training clips are overlap-free. Crossing objects can exhibit incorrect foreground ordering, identity bleed, or temporary appearance changes.
• Limited multi-object supervision: Only 15 training clips contain multiple objects, and the dataset contains no examples with more than three annotated tracks. Larger layouts are extrapolations.
• Bounding-box geometry matters: A short, wide chair-shaped box may resist prompts requesting a tall, narrow fire hydrant; resize the box to fit the intended subject.
• Small objects are harder: Very small masks occupy fewer latent tokens and can weaken regional prompt adherence.
• Prompt conflicts can cause leakage: Contradictory global and regional descriptions, repeated colors across objects, or crowded scenes can reduce consistency.
• No guaranteed persistence during occlusion: A moving box does not constitute a persistent instance-segmentation or tracking identity.
• VRAM scales with object count: Every additional regional prompt contributes its own processed text context.
• Best fidelity near the training bucket: 768 × 448, 121-frame clips are closest to the training distribution.
• Reference images in the editor are visual guides only: They are not supplied to the generation model.

Future Work: Occlusion and Depth Ordering

The released LoRA was deliberately trained on a dataset with zero overlapping bounding boxes. It should therefore not be presented as an occlusion-trained model, and explicit foreground/background control is not a supported feature of this release.

Earlier experimental LoRAs used different datasets containing overlapping objects, per-frame depth annotations, overlap masks, and occlusion-weighted reconstruction losses. Some of those earlier checkpoints successfully rendered crossings with the expected foreground/background relationship.

Why Bounding Box Overlap Is Ambiguous

A bounding box describes a rectangular extent, not an object's true visible silhouette. When two boxes intersect, the overlapping rectangle can contain foreground body pixels, background body pixels, empty space, or a mixture of all three.

For two tracked boxes, the overlap region is:

$$O_{ij}(t)=B_i(t)\cap B_j(t)\mathrm{.}O\_\{ij\}(t)\; =\; B\_i(t)\; \backslash cap\; B\_j(t).$$

If object i is in front of object j, an ideal depth-resolved ownership rule would preserve foreground pixels and remove only the genuinely occluded background pixels:

$$M_i^{\mathrm{v}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{b}\mathrm{l}\mathrm{e}}(t)=M_i(t),M_j^{\mathrm{v}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{b}\mathrm{l}\mathrm{e}}(t)=M_j(t)\setminus O_{ij}(t)\mathrm{.}M\_i^\{\backslash mathrm\{visible\}\}(t)\; =\; M\_i(t),\; \backslash qquad\; M\_j^\{\backslash mathrm\{visible\}\}(t)\; =\; M\_j(t)\; \backslash setminus\; O\_\{ij\}(t).$$

This approximation is imperfect because rectangular boxes are not segmentation masks. For example, a person's hand can enter another subject's box even when most of their body remains behind that subject. Erasing the entire rectangular overlap can therefore remove too much background information, while leaving both masks unrestricted can cause appearance bleed or identity swaps.

Occlusion Methods Explored

• Depth-ordered rendering: Draw background controls before foreground controls using estimated per-frame object depth.
• Hidden background overlap: Remove the background box's intersecting region to indicate which object should appear in front.
• Faded or grayscale overlap: Preserve overlap visibility while reducing the background control's visual strength.
• Foreground-colored overlap: Fill intersections with the front object's color; this introduced visual leakage in some scenes.
• Visible-only regional masks: Subtract the overlap from background masks; this can weaken object identity when a subject becomes heavily occluded.
• Full regional masks: Preserve each complete object region; this improves trajectory continuity but does not resolve depth ownership by itself.
• Overlap-weighted denoising: Increase reconstruction emphasis inside intersections; this occasionally improved crossings but could destabilize identity.
• Depth smoothing and propagation: Stabilize depth estimates across near-equal ordering and contaminated occluded frames.

The key unresolved challenge is to combine reliable regional identity binding with explicit foreground/background ownership without deleting important control geometry or introducing prompt bleed.

The video above demonstrates an earlier research prototype trained with overlapping examples; it is not an output or guaranteed capability of the released overlap-free LoRA.

Future work could combine the current reliable regional prompt-to-box binding with overlap-aware training data, stable per-object masks, explicit depth ordering, and carefully balanced overlap reconstruction losses. The main challenge is improving occlusion without reintroducing object-identity swaps, appearance bleeding, or missing control geometry.

Support

If you find this project useful and would like to support future releases:

Acknowledgments

Built on the LTX 2.5 foundation model and the Lightricks LTX-2 training framework.

Training footage sourced from Pexels under the Pexels License.

ChatGPT and Gemini