A d D rift B ench: A Benchmark for Detecting Data Drift and Label Drift in Short Video Advertising (2025)

Abstract: With the commercialization of short video platforms (SVPs), the demand for compliance auditing of advertising content has grown rapidly. The rise of large vision-language models (VLMs) offers new opportunities for automating ad content moderation. However, short video advertising scenarios present unique challenges due to \textbf{data drift (DD)} and \textbf{label drift (LD)} . DD refers to rapid shifts in data distribution caused by advertisers to evade platform review mechanisms. LD arises from the evolving and increasingly standardized review guidelines of SVPs, which effectively alter the classification boundaries over time. Despite the significance of these phenomena, there is currently a lack of benchmark tools designed to evaluate model performance under such conditions. To address this gap, we propose \textbf{AdDriftBench (ADB)} . The ADB dataset consists of 3,480 short video ads, including 2,280 examples labeled under data drift scenarios, designed to evaluate the generalization capabilities of VLMs under rapidly shifting content distributions. An additional 1,200 examples represent label drift scenarios, aimed at assessing VLMs’ abilities in instruction following and fine-grained semantic understanding under varying auditing standards. Through extensive experiments on 16 open-source VLMs, we find that current models perform moderately in short video advertising contexts, particularly in handling fine-grained semantics and adapting to shifting instructions. Our dataset will be made publicly available.

## introduction the commercialization of short video platforms (svps) has led to a growing demand for the moderation of short video advertisements. traditional content moderation methods rely heavily on manual rules and small-scale models @xcite . recently, vision-language models (vlms) have demonstrated impressive capabilities in both visual and textual understanding @xcite @xcite , showing strong potential in tasks such as content comprehension and violation detection. however, the short video advertising domain is characterized by largescale data drift (dd) and label drift (ld), posing new challenges for vlms in terms of fine-grained semantic understanding and strict instruction following. dd refers to frequent shifts in data distribution caused by advertisers aggressively modifying their content to evade platform moderation policies. as illustrated in figure 1 (a), whereas previously advertisers might embed explicit violations (e.g., horse mating scenes) directly into videos, they now often overlay such content using picture-in-picture (pip) techniques. figure 1(b ) illustrates the performance degradation of a multimodal small model over time, highlighting how dd contributes to the model's decreasing accuracy. ld refers to changes in the classification boundarie, resulting from the increasingly standardized moderation rules on svps. as illustrated in figure 1(c), condom advertisements with suggestive content were previously allowed but are now considered violations under stricter policies. figure 1(d ) visualizes both dd and ld effects: quadrilaterals represent the distribution of video ads, and circles represent the classification boundaries. several benchmarks @xcite @xcite have been proposed to evaluate video content compliance (see to ensure data quality, we applied similaritybased deduplication, used models to pre-filter highrisk cases, and involved professional human reviewers for final validation. through a comparison of 16 widely-used open-source vlms, we find that their compliance identification performance in short video advertising scenarios is suboptimal. this highlights the need for improvement in both instruction following and fine-grained semantic reasoning. our contributions are as follows: benchmarks sv ad dd ld safewatch @xcite ✗ ✗ ✔ ✗ mmdt @xcite ✗ ✗ ✔ ✗ xd-violence @xcite ✗ ✗ ✗ ✗ ucf-crime @xcite ) ✗ ✗ ✗ ✗ fakesv @xcite ✔ ✗ ✔ ✗ fvc @xcite ✔ ✗ ✔ ✔ lspd @xcite ✗ ✗ ✗ ✗ kuaimod @xcite ✔ ✗ ✗ ✔ ## addriftbenchmark to ensure the results are convincing, we aimed to maintain a balanced distribution of data across each scenario, with the specific scene distribution shown in figure 3 . for the data drift scenario, we selected 6 primary scenes (such as gray market, pornography, gambling, and false advertising) and 12 secondary scenes (such as involves weight loss, unregulated industry, and personal privacy leakage). for the label drift scenario, we selected 10 scenes (such as guaranteed promises, vulgar condom ads, and gray market), with detailed scene definitions provided in appendix a ( data drift scenario design. in the short video advertising scenario, advertisers are continuously iterating materials in an attempt to bypass platform review rules, leading to the same violation scene appearing with different distributions of violating videos. we trained a 7-class small model and observed that over time, both the precision and recall of the model decreased (as illustrated in figure 1(b) ). since small models tend to overfit the data distribution of the training set, it can be assumed that the data distribution changes each month. therefore, we selected data from 8 months, ensuring that each secondary scene appeared with at least 30 samples per month. the distribution of primary and secondary scenes for each month is illustrated in the figure 4 . we evaluate vlms' ability to handle data drift by measuring their precision and recall in different scenes across different months. label drift scenario design. as illustrated in figure 1 (c), in short video advertising, label drift arises as platforms become more regulated and re-view policies grow increasingly strict. videos that previously passed review may now be rejected. to study this, we selected 10 scenarios where rule tightening has caused label drift (as illustrated in figure 3(b) ), and evaluated vlms' robustness to label drift by prompting them with different review criteria. specifically, we design two sets of prompts-lenient and strict-aimed at assessing the vlms' instruction-following ability and finegrained semantic understanding. ## conclusion this paper introduces adb, the first benchmark specifically designed for short video advertising scenarios. we evaluate 16 open-source vlms across two major types of distributional shifts-data drift and label drift. our findings reveal that current open-source vlms exhibit significant limitations in handling short video advertising content, particularly in their ability to cope with data and label drift. these shortcomings highlight two key challenges for existing vlms: limited fine-grained semantic understanding and insufficient adherence to strict instruction following.