TC -Bench: Benchmarking Temporal Compositionality in Conditional Video Generation (2025)

Abstract: Video generation has many unique challenges beyond those of image generation. The temporal dimension introduces extensive possible variations across frames, over which consistency and continuity may be violated. In this work, we evaluate the emergence of new concepts and relation transitions as time progresses in a video, which we refer to as Temporal Compositionality. We propose TC-Bench, a benchmark of meticulously crafted text prompts, ground truth videos, and new evaluation metrics. The prompts articulate the initial and final states of scenes, effectively reducing ambiguities for frame development. In addition, by collecting corresponding ground-truth videos, the benchmark can be used for text-to-video and image-to-video generation. We develop new metrics to measure the completeness of component transitions, which demonstrate significantly higher correlations with human judgments than existing metrics. Our experiments reveal that contemporary video generators are still weak in prompt understanding and achieve less than 20% of the compositional changes, highlighting enormous improvement space. Our analysis indicates that current video generation models struggle to interpret descriptions of compositional changes and synthesize various components across different time steps.

## introduction conditional video generation is the task of synthesizing realistic videos based on controlling inputs such as text prompts (text-to-video, t2v) or images (image-to-video, i2v). significant advancement in dataset scale and model design has led to several large-scale, high-quality video generation models, such as cogvideo @xcite , videocrafter @xcite , stable video diffusion @xcite , and others @xcite @xcite . the additional time dimension in videos makes it essential to accurately assess and benchmark the alignment between the generated temporal variations and the condition inputs. while several studies have proposed fine-grained and comprehensive evaluation protocols @xcite @xcite , compositionality in the temporal dimension remains an underaddressed yet crucial aspect of video generation tasks. the principle of compositionality specifies how constituents are arranged and combined to make a whole @xcite @xcite . ideal generative systems should produce outputs that reflect the compositions described by the prompts @xcite @xcite . in image generation, prior work has focused on improving faithful compositionality in attributes, numbers, and spatial arrangement @xcite @xcite . in video generation, compositional faithfulness is much more challenging-the output must consistently reflect the required combination of concepts, even as it changes through time. in this work, we investigate this temporal compositionality problem in video generation models by focusing on prompts describing scenarios where object attributes or relations change over time. while image generation prompts involving spatial compositionality @xcite and video prompts describing actions or motions @xcite @xcite have been used for assessing t2v models, they have two drawbacks: first, these prompts describe invariant compositions in time, and second, they lead to synthesized videos that manipulate existing metrics. for instance, while fig. 1 (left) depicts "a horse running on the beach" motion, there are no compositional variations in the visual entities along the time axis. such omissions can lead to flaws that, while noticeable to human users, are not captured by current benchmarks. in contrast, fig. 1 (right) involves more specific compositional changes in position and color, marked by the vertical "edges" and the gap between horizontal edges in the spatiotemporal image representing attribute or object binding changes. to this end, we propose temporal compositionality benchmark (tc-bench), which addresses three scenarios of compositional changes: attribute transition, object relations, and background shifts. we craft realistic prompts that clearly specify an object's initial and final states, thereby requiring changing compositional characteristics in a correctly synthesized video. these prompts span a wide range of topics and scenes and present distinct challenges to different modules of t2v models. on the one hand, the text encoding stage needs to aggregate different groups of constituents from the prompt to guide the generation of different frames. on the other hand, the generation module must synthesize seamless transitions between frames while maintaining object consistency. to broaden applicability to i2v, we collect ground truth videos corresponding to the prompts, which allows us to benchmark models capable of performing generative frame interpolation @xcite . to facilitate the use of tc-bench, we propose two evaluation metrics, tcr and tc-score, that first produce frame-level compositionality assertions and check them throughout the video using vision language models (vlms). tcr and tc-score measure compositional transition completion and overall text-video alignment, which are better correlated to human judgments than existing metrics. we extensively benchmark multiple baselines across three categories of methods, ranging from direct t2v models @xcite @xcite to multi-stage t2v @xcite and i2v models @xcite . our comprehensive experiments demonstrate that most of the video generation models accomplish less than ∼20% of the test cases, implying enormous space for future improvement. our contribution can be summarized as three points: ## tc-bench prompts prompt types in this work, prompts with specific temporal compositionality follow one of the three types of scene graph transitions, as shown in fig. 2 (left). we use o i to denote an object, a i to denote an attribute, and arrows or line segments to represent possession or other relationships between them. based on the scene graph change, the prompts are categorized into three types: attribute transition, object relation change, and background shifts with one example for each in fig. 2 ## evaluation metrics we design an evaluation pipeline that uses llms/vlms to generate and verify assertions on the generated videos. for i2v models, we additionally investigate frame consistency based on latent features to make the metrics more robust. a pink chameleon turns green. a cartoon monkey moves a yellow boat from a train trestle bridge down to the lake below. a cityscape transitioning from sunset to evening. ## method we introduce a simple and effective baseline to improve the transition completion rate over text-tovideo generation models. based on the prompt p , we first instruct an llm to generate the text description of the initial scene p 0 and the end scene p k . then we utilize a diffusion-based text-to-image generation model f @xmath0, i k . however, simply using p 0 , p k to guide the generation process overlooks the consistency across the frames. therefore, we apply the same noise map z t as the initialized noise pattern of both diffusion paths and substitute the selfattention maps of i k with maps from i 0 's diffusion trajectory for the first half of the timesteps. as p 0 and p k share similar semantics except in some attributes or object positions, we discover that such a simple method can end up with i 1 and i k sharing similar image structures. then, the generated frames are used to guide the process of video generation so that the temporal transition can be completed under the guidance of i k . we use an off-the-shelf video generation model seine @xcite for the generative transition from i 1 to i k . we refer to this baseline as sdxl+seine as we adopt sdxl @xcite for start and end frame generation. metrics for the proposed tcr and tc-score, we adopt gpt-4 turbo, cogvlm2-19b, and llava-next-7b to assess all the assertions. the results reported in the main paper are based on gpt-4 turbo, and the other results are reported in ap-pendix a and gen-3 alpha demonstrates apparent superiority in background transitions with a 27.45% tcr, while dream machine achieves the best results in object relation transitions with a 12.77% tcr. as no single model/system dominates over all three types of prompts, we conjecture that the results implicitly reflect the differences in data curation between different models/systems. the overall results also evidence the difficulty of tc-bench prompts, even though they are mostly single-hop transitions in concept. our rank of t2v models also aligns with established benchmarks such as vbench @xcite or evalcrafter @xcite . multi-stage t2v models, including free-bloom @xcite , lvd @xcite , and our sdxl+seine, generates framewise prompts, layouts, and images as intermediate steps. while these methods effectively enhance the overall tcr, the unbalanced fluctuations across types reveal limitations using explicit mid-level representations. for instance, lvd fails to address attributes or backgrounds because these transitions cannot be represented using bounding boxes. sdxl+seine underperforms in object relation because t2i models struggle to control object positions in two diffusion paths of similar structures. the results suggest the necessity of fundamentally dream machine gen-3 alpha free-bloom sdxl+seine (ours) figure 4: qualitative comparison between different models in attribute and object binding transitions. i2v models as shown in ## conclusion in this work, we propose a new video generation benchmark tc-bench, featuring temporal compositionality. tc-bench characterizes three different types and a wide range of topics. we show that simple transitions that can happen in several seconds remain extremely challenging to existing t2v methods. we also propose assertion-based evaluation metrics and investigate consistency evaluation using flow-based methods or latent features. our benchmark, experimental results, and analysis unveil the weaknesses of existing t2v and i2v models in temporal compositionality, suggesting crucial directions for future improvement. future work should investigate techniques to 1) automatically mine videos with specific temporal compositionality and generate detailed captions, 2) evaluate text-video alignment more efficiently, and 3) improve text-to-video models in addressing temporal compositionality.