V id C ap B ench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generation (2025)

Abstract: The training of controllable text-to-video (T2V) models relies heavily on the alignment between videos and captions, yet little existing research connects video caption evaluation with T2V generation assessment. This paper introduces VidCapBench, a video caption evaluation scheme specifically designed for T2V generation, agnostic to any particular caption format. VidCapBench employs a data annotation pipeline, combining expert model labeling and human refinement, to associate each collected video with key information spanning video aesthetics, content, motion, and physical laws. VidCapBench then partitions these key information attributes into automatically assessable and manually assessable subsets, catering to both the rapid evaluation needs of agile development and the accuracy requirements of thorough validation. By evaluating numerous state-of-the-art captioning models, we demonstrate the superior stability and comprehensiveness of VidCapBench compared to existing video captioning evaluation approaches. Verification with off-the-shelf T2V models reveals a significant positive correlation between scores on VidCapBench and the T2V quality evaluation metrics, indicating that VidCapBench can provide valuable guidance for training T2V models. The project is available at https://github.com/VidCapBench/VidCapBench.

## introduction controllable text-to-video (t2v) generation leverages text prompts to guide video synthesis @xcite , enabling the instant visualization of designs and facilitating applications in creative content and entertainment. advances in generative model's backbones @xcite @xcite further innovate the video generation process to adhere to textual instructions, exhibit aesthetic appeal, and conform to physical laws. video captioning, the crucial supporting infrastructure to t2v generation, has also progressed. coarse-grained or detail-lacking captions significantly hinder both the comprehension and reconstruction of visual information @xcite . hence, prevalent t2v models are devoted to strengthening the alignment between the generated content and the detailed prompts/captions @xcite . with the objective to optimize this alignment, t2v models present high fidelity in subjects' motion @xcite @xcite , temporal changes @xcite @xcite , and event progression @xcite . meanwhile, the quality of datasets used for training captioning models @xcite has also improved considerably, such as openvid @xcite and sharegpt4v @xcite . to direct the optimization of t2v models, video captions should be accurate, comprehensive, diverse, concise, and abundant. while caption formats vary across different models @xcite , the core elements in captions to address these challenges, we introduce vid-capbench, the first evaluation benchmark designed for assessing video captions in controllable t2v generation, as depicted in figure 1 . comparison between vidcapbench and several publicly available video caption benchmarks is presented in video captioning. the goal of video captioning is to describe a video across several key aspects, aiding understanding @xcite , retrieval @xcite , and motion control @xcite . in t2v generation, accurate and detailed video captions can enhance semantic alignment during model training @xcite . naive captioning models adopt free-form descriptions @xcite . to enhance controllability, miradata @xcite , vdc @xcite , and vript @xcite emphasize specific aspects like subjects, background, and shots, significantly benefiting t2v generation. other methods describe videos from an event perspective @xcite , capturing temporal information more effectively. despite advancements in caption controlling @xcite , evaluations with omissions may lead to a seesaw effect where gains in one dimension come at the cost of others, limiting the utility of the captioning model. ## vidcapbench in this section, we introduce the design and curation of vidcapbench. ## conclusion vidcapbench introduces a comprehensive evaluation framework for video captioning in t2v generation, assessing across four key dimensions: video aesthetics, video subject, video motion, and physical laws. to cater to different evaluation needs, vidcapbench comprises two subsets: one designed for automated evaluation prioritizing speed, and the other for human evaluation prioritizing accuracy. compared to existing benchmarks, vidcapbench exhibits greater stability and reliability. furthermore, the strong correlation between scores on vid-capbench and t2v quality metrics demonstrates its potential for guiding t2v training processes.