V ideo S core: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation (2024)

Abstract: The recent years have witnessed great advances in video generation. However, the development of automatic video metrics is lagging significantly behind. None of the existing metric is able to provide reliable scores over generated videos. The main barrier is the lack of large-scale human-annotated dataset. In this paper, we release VideoFeedback, the first large-scale dataset containing human-provided multi-aspect score over 37.6K synthesized videos from 11 existing video generative models. We train VideoScore (initialized from Mantis)based on VideoFeedback to enable automatic video quality assessment. Experiments show that the Spearman’s correlation betweenVideoScore and humans can reach 77.1 on VideoFeedback-test, beating the prior best metrics by about 50 points. Further result onother held-out EvalCrafter, GenAI-Bench, and VBench show that VideoScore has consistently much higher correlation with humanjudges than other metrics. Due to these results, we believe VideoScore can serve as a great proxy for human raters to (1) rate different video models to track progress (2) simulate fine-grained human feedback in Reinforcement Learning with Human Feedback (RLHF) to improve current video generation models.

## introduction powerful text-to-video (t2v) generative models have been exponentially emerging these days. in 2023 and 2024, we have witnessed an array of t2v models like sora @xcite , runway gen-2 @xcite , lumiere @xcite , pika foot_0 , luma-ai foot_1 , kling foot_2 , emu-video @xcite , stablevideodiffusion @xcite . these models have shown their potential to generate longer-duration, higher-quality, and more natural videos. despite the great progress video generative models have made, they still suffer from artifacts like unnaturalness, inconsistency and hallucination, which calls for reliable fine-grained metrics for evaluation and a robust reward model for reinforcement learning (rlhf). the recent literature has adopted a wide range of metrics to evaluate videos. however, these metrics suffer from the following issues: (1) some of them are computed over distributions and cannot be adopted to evaluate a single model output. examples include fvd @xcite and is @xcite . (2) most metrics can only be used to evaluate visual quality or text alignment, while failing on other aspects like motion smoothness, factual consistency, etc. examples of such metrics include clip @xcite , dino @xcite , brisque @xcite . (2) some metrics focus only on a single mean opinion score (mos), failing to provide finegrained subscores across different multiple aspects. examples include t2vqa @xcite , fastvqa @xcite , and dover @xcite . (3) several works @xcite propose to prompt multi-modal largelanguage-models (mllm) like gpt-4o @xcite or gemini-1.5 @xcite to produce multi-aspect quality assessment for given videos. however, our experiments show that they also have low correlation with humans. these feature-based metrics or mllm prompting methods either fail to provide reliable evaluation or cannot simulate the human feedback from real world well, which have lagged behind and restricted us from training better video generative models. since obtaining large-scale human feedback is highly costly, we can try to approximate humanprovided scores with model-based metrics. to this end, our work can be divided into two parts: (1) curating videofeedback, the first largescale dataset containing human-provided scores for 37.6k videos, (2) training videoscore on videofeedback, which is an automatic video metric to simulate human feedback. in preparation of videofeedback, we solicit prompts from vidprom @xcite and use 11 popular text-to-video models, including pika, lavie @xcite , svd @xcite , etc, to generate videos of various quality based on these prompts. we define five key aspects for evaluation as shown in to build the video evaluator, we select mantis-idefics2-8b @xcite as our main backbone model due to its superior ability to handle multi-image and video content, accommodating up to 128 video frames and supporting native resolution. after fine-tuning mantis on vide-ofeedback-train, we get our video evaluator, videoscore. experiments show that we achieve a spearman correlation of 77.1 on videofeed-back-test and 59.5 on evalcrafter @xcite for the text-to-video alignment aspect, sur-passing the best baseline by 54.1 and 4.4 respectively. the pairwise comparison accuracy gets 78.5 on genai-bench @xcite video preference part, and 72.1 in average on 5 aspects of vbench @xcite , surpassing the previous best baseline by 11.4 and 9.6 respectively. additional ablation studies with different backbone models confirmed that the mantis-based metric provides a gain of 12.1 compared to using the idefics2based metric. due to the significant improvement, we believe that videoscore can serve as the reliable metrics for future video generative models. ## videofeedback this section introduces the construction process of our dataset, videofeedback. we start by explaining how we gathered and filtered diverse text prompts for video generation, followed by the video-generation processes using 11 selected textto-video models. next, we outline the annotation pipeline that guides raters to score videos across multiple aspects defined in ## experiments in this section, we describe our experiment setup, including baseline methods for video evaluation, and evaluation benchmarks for video evaluation. we also discuss the training details of videoscore, and analyze our experiment results. ## conclusion in this paper, we introduce videoscore, which is trained on our meticulously curated dataset vide-ofeedback for video evaluation and can serve as good simulator for human feedback on generated videos. we hired 20 expert raters to annotate the 37.6k videos generated from 11 popular textto-video generative models across 5 key aspects, visual quality, temporal consistency, dynamic degree, text-to-video alignment and factual consistency. our iaa match ratio gets more than 60%. we test the performance of videoscore using spearman correlation on videofeedback-test and evalcrafter, and using pairwise comparison accuracy on genai-bench and vbench. the results show that videoscore consistently gets the best performance, surpassing the powerful baseline gpt-4o and gemini 1.5 flash/pro by a large margin. our work highlights the importance of using mllm for video evaluation and demonstrates the future prospects of simulating human scores or feedback in generative tasks, due to its rich world knowledge and the high-quality rating dataset with fine-grained and multiple dimensions.