Fighting FIR e with FIRE : Assessing the Validity of Text-to-Video Retrieval Benchmarks (2023)

Abstract: Searching troves of videos with textual descriptions is a core multimodal retrieval task. Owing to the lack of a purpose-built dataset for text-to-video retrieval, video captioning datasets have been re-purposed to evaluate models by (1) treating captions as positive matches to their respective videos and (2) assuming all other videos to be negatives. However, this methodology leads to a fundamental flaw during evaluation: since captions are marked as relevant only to their original video, many alternate videos also match the caption, which introduces false-negative caption-video pairs. We show that when these false negatives are corrected, a recent state-of-the-art model gains 25% recall points—a difference that threatens the validity of the benchmark itself. To diagnose and mitigate this issue, we annotate and release 683K additional caption-video pairs. Using these, we recompute effectiveness scores for three models on two standard benchmarks (MSR-VTT and MSVD). We find that (1) the recomputed metrics are up to 25% recall points higher for the best models, (2) these benchmarks are nearing saturation for Recall@10, (3) caption length (generality) is related to the number of positives, and (4) annotation costs can be mitigated through sampling. We recommend retiring these benchmarks in their current form, and we make recommendations for future text-to-video retrieval benchmarks.

## introduction text-to-video retrieval (tvr) is a challenging multimodal retrieval task @xcite with practical applications ranging from web search to organizing media collections @xcite . to measure tvr model improvement-despite a dearth of purpose-built tvr benchmarks-researchers created benchmarks by re-purposing video captioning datasets such as msr-vtt @xcite ., * correspondence to me@pedro.ai test caption: "cartoon girl is talking" predictions rater labels correct@1 : 0% : 100% : , : video 1 video 2 video 3 model test videos: e.g., msrvtt 1k : , : : , : metrics does "cartoon girl is talking" caption the video? legend : original label : fire label : relevant : irrelevant evaluation data figure 1: msr-vtt and msvd have one positive video per caption (each video's caption). captions often match multiple videos, leading to false negatives. when models rank false negatives highly, model quality is understated (full example in appendix figure 5 ). this leads to evaluations where reported metrics do not reflect their true value and are therefore not internally valid ( §2.2.1). early work established an evaluation paradigm that treated captions as search queries over the collection of captioned videos @xcite @xcite ; each caption and their corresponding video are positives (relevant) during retrieval, and all other caption-video pairs are negatives (irrelevant). however, even a cursory inspection of videos and captions reveals many additional positive caption-video pairs ( §2). in current benchmarks, true positives that are not the video's original caption are falsely assumed to be negatives. @xcite first identified this fundamental, falsenegative problem in tvr evaluation; our work builds on this by quantifying the absolute metric differences that false negatives induce (see discussion in §6). accurate absolute metrics are crucial in industrial settings where deployment criteria are often defined by minimum quality targets. these false implicit relevance labels introduce measurement error-e.g., clip4clip's @xcite recall@1 is underestimated by 25% points ( §2.2). we estimate measurement error by annotating 683k additional caption-video pairs, which we call the fire dataset ( §3). 1 a core measurement principle is that operationalized metrics should strongly correlate to the quantity they intend to measure @xcite . for example, recall@k operationalizes the intent to measure retrieval quality. label errors are a common way that measurements are invalidated @xcite . our work shows that since tvr metrics are computed with false negative label errors, recall@k does not accurately reflect retrieval quality, which negates the measurement's validity. in the remainder of this paper, we posit rationales of why models gain different score boosts ( §4.1) and estimate how useful the fire dataset is for evaluating future models ( §4.2 and §4.3). to conclude, we review the implications of our findings. looking to the past, retrieval effectiveness has been understated for some models, which gives an overly pessimistic view of recent advances @xcite . critically, our results also suggest that the msr-vtt benchmark is nearing saturation and should be retired soon in favor of a purpose-made benchmark. looking outward, we identify structurally similar benchmarks-such as photo retrieval-that likely also have the same false implicit relevance problem. a successful benchmark should avoid the pitfalls we identify in this paper, be faithful to the real-world user task it targets @xcite , improve reproducibility, and evolve ( §7). ## text-to-video retrieval evaluation this section reviews current tvr evaluation practices using two concepts: internal validity (campbell, 1957, §2.2.1) and construct validity (tague-sutcliffe, 1992, §2.2.2). internal validity refers to whether an evaluation reliably establishes a causeeffect relationship between the measured dependent variable and the independent variable to be estimated @xcite . in tvr evaluations, false negatives confound model quality and label errors (i.e., is the model wrong or is the label wrong?) which makes reliably establishing cause (model quality) and effect (retrieval score) difficult. construct validity "pertains to the degree to which the measure of a construct sufficiently measures the intended concept" (o'leary-kelly and j. vokurka, 1998)-in 1 data and code: pedro.ai/multimodal-retrieval-evaluation. tvr evaluations, an important intended concept is real-world search quality. construct validity asks: can we expect that measuring retrieval quality with the benchmarks at hand generalizes to real-world search quality? this section argues that tvr evaluations are not internally valid or construct valid. ## fire dataset collection and validation next, we describe and analyze the fire dataset. ## analysis experiments the difference fire makes on metrics ( ## recommendations for benchmarks towards improving tvr evaluations, we make recommendations for both current and future benchmarks. this paper only investigated the effects of false negatives in msr-vtt and msvd. however, it is likely that other similarly constructed benchmarks exhibit the same problem, and testing this is important. second, we show that for msr-vtt 13 msr-vtt is small. to avoid convergence to the sample mean, bootstrap sizes need to stay low. and msvd, certain metrics such as correct@10 are potentially saturated since improvements above clip4clip's 95.7% and 94% are plausibly noisy. consequently, since the remaining gains reside in re-ranking the top k, the community should consider retiring these evaluations. third, the introduction of multiple positives and use of various k values makes mean average precision attractive since: (1) it factors in preference for correctness at higher ranks and (2) it handles multiple positives. it is difficult to recommend that model developers exhaustively annotate model predictions. this suggests a future where query or video set size is a trade-off between annotation load and evaluation quality. for example, one might choose to trade-off annotation load with statistical power to differentiate between models @xcite . trec-style, annual shared tasks are one model for this @xcite ; instead of building a monolithic benchmark that becomes overfit over time @xcite , stakeholders develop evaluations that evolve with research objectives. looking forward, tvr evaluation would benefit from: (1) a purpose-built benchmark that is grounded in an actual use case so as to be ecologically valid (de vries et al., 2020) and (2) centralized evaluation by submitting runnable models to shared infrastructure such as dynabench @xcite . this would improve reproducibility, which was a limiting factor in selecting which model predictions to reproduce in this paper. this we show a log-scale density heatmap binned by the number of positive videos and caption length; on the margins, are histograms. from this figure, we can infer that: (1) if a caption is long, it is less likely to have many positive videos, and (2) if a caption is short, then the number of positive videos can vary widely. also makes calculating statistical tests easy (ethayarajh and jurafsky, 2020), which are often not reported @xcite . tvr modeling has advanced enough to demand better benchmarks for measuring future progress. ## related work the paper draws on ideas in multimodal retrieval, information retrieval, and evaluation methodology. improving benchmark quality: wray et al. ( 2021 ) is directly relevant to our work, and we share their motivation: to study the effects of false negatives in tvr evaluations. while we share motivation and our works are complementary, our work differs substantially in methods, contributions, and conclusions. the primary difference is this: our goal is to quantify the difference in absolute metrics that false negatives cause, even if there is no promise the data can be effectively reused in the future; @xcite develop automatically runnable proxy measures that improve the reliability of model rankings, but do not precisely quantify the impact of false negatives on existing metrics since automatic labeling is not equivalent to human annotation. both these works are valuable: our work conclusively quantifies that false negatives create differences of 25% absolute points and demonstrate that new measures like those by @xcite are necessary for current benchmarks. @xcite argue that video captioning datasets used in tvr evaluation are noisy due to low-quality captions but differ by identifying single query tasks as the root problem (as opposed to false negatives) and recommend multi-query evaluation where users make followup refinement queries. while the multi-query problem is important, we do not agree with the assessment that single-query problems should be abandoned for multi-query problems: for example, users often have a low tolerance for voice assistant errors and abandon their query entirely after an error. both problems are important. fortunately, the approaches are complementary and should be combined: the multiquery setting still has false negatives, whose effects on measurement can be mitigated with our methods ( §4.3). just as we use predictions to improve datasets, @xcite improve imagenet labels by using predictions to reduce the label space which makes the annotation task easier. figure 4 : the distribution of absolute differences between bootstrap re-sample estimates of clip4clip c@1 and the true sample mean, by re-sample size. this estimates the number of annotations to detect an effect size at 95% confidence. appendix e expands on this experiment by showing results for c@5, c@10, and teachtext. datasets and benchmarks @xcite . our work is in line with this broader initiative and critically examines text-tovideo retrieval evaluation methodology. we examine internal validity ( §2.2.1) and find a broken yardstick @xcite . by examining construct validity ( §2.2.2), we also argue that tvr evaluations should prize usefulness to ecologically valid use cases such as real-world text-to-video search @xcite . lastly, our experimental results suggest we may not be far off from retiring msr-vtt and msvd for tvr evaluation, something we should not be afraid to do in general @xcite ). an alternate approach is smaller, periodic evaluations as in trec @xcite @xcite . part of the solution is to create purpose-built datasets with clear goals @xcite as opposed to continually re-using datasets intended for different uses @xcite . structurally similar tasks: tvr is not the only evaluation with the implicit false negative problem. our critique is applicable to image retrieval bench-marks that use caption-media pairs from image captioning datasets @xcite as the only positives @xcite @xcite . ## conclusion in this work, we show that label errors (false negatives) in text-to-video retrieval benchmarks invalidate their internal validity-the measured metrics do not accurately reflect reality ( §2). following this, we critique the applicability of benchmarks to real-world use cases (construct validity). to estimate the impact of false negatives on benchmark metrics, we collect the fire dataset ( §3) which contains 683k relevance judgements. analysis experiments ( §4) suggest explanations for why clip4clip scores higher and estimate system pooling generalization. based on our findings, we highlight properties that future tvr benchmarks should have and outline approaches to addressing inherent challenges in retrieval evaluation ( §5). finally, we position our work in the broader effort to improve benchmarking by better aligning tasks with the intended use and improving measurement ( §6).