FIBER : Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Framework (2022)

Abstract: We propose fill-in-the-blanks as a video understanding evaluation framework and introduce FIBER – a novel dataset consisting of 28,000 videos and descriptions in support of this evaluation framework. The fill-in-the-blanks setting tests a model’s understanding of a video by requiring it to predict a masked noun phrase in the caption of the video, given the video and the surrounding text. The FIBER benchmark does not share the weaknesses of the current state-of-the-art language-informed video understanding tasks, namely: (1) video question answering using multiple-choice questions, where models perform relatively well because they exploit linguistic biases in the task formulation, thus making our framework challenging for the current state-of-the-art systems to solve; and (2) video captioning, which relies on an open-ended evaluation framework that is often inaccurate because system answers may be perceived as incorrect if they differ in form from the ground truth. The FIBER dataset and our code are available at https://lit.eecs.umich.edu/fiber/ .

## introduction despite current progress on multimodal (textual and visual) representations, language-informed video understanding is still a very challenging task for machine learning systems @xcite . this is due in large part to the task setup and the dataset construction. current video understanding datasets often have at least one of two major limitations. first, they have limited application value. e.g., multiple-choice questions @xcite @xcite do not reflect real-world tasks. second, they are based on subjective evaluation metrics, e.g., video captioning @xcite @xcite ), and are therefore hard to evaluate automatically, as the ground truth can be expressed in different ways. in this paper, we address these limitations by introducing a new dataset named fiber that collects multiple perspectives on the same video, focusing on noun phrases as a proxy for different entities and their interactions in the video. our data focuses on recall and tests the ability of models to capture a wide range of possible interpretations for a particular aspect of a video. we construct the fiber dataset by systematically blanking captions from an existing video captioning dataset named vatex @xcite and by providing additional correct answers for the blanks. vatex is a video captioning dataset that contains 40,000 10-second youtube videos with 10 english captions per video. 1 we build our video fill-in-the-blanks dataset by blanking random noun phrases from one of the english captions for each video, from a subset of vatex consisting of 28,000 videos. through extensive analyses, we show that the blanked noun phrases are essential for understanding important visual aspects from the video. to address the fill-in-the-blanks task, we propose a transformer-based @xcite multimodal model. our experiments show that our best multimodal model achieves a token-level f1 score of 71.4 while the f1 score of crowd workers is 82.5, indicating that this task is challenging for video and text understanding. the contribution of this work is threefold: (1) we propose a novel fill-in-the-blanks task as an evaluation framework that addresses the drawbacks associated with previous approaches to video understanding. in support of this framework, we introduce fiber, which is a novel dataset of 28,000 videos and fill-in-the-blanks captions with multiple correct answers. (2) we propose several unimodal baselines and two multimodal models for solving this task. (3) we provide a detailed analysis of the data to measure the diversity and complexity of the answers, and also conduct an error analysis of the models' performance, to gain insights into the blanked captions and videos that are hard for the models to solve. ## related work language-informed video understanding is a complex task that has been extensively addressed in the multimodal (natural language and computer vision) machine learning research through diverse tasks and benchmarks. ## video fill-in-the-blanks dataset we construct fiber -a large video understanding dataset that can evaluate the ability of a model to interpret and use a multimodal context by requiring the models to "fill in" (generate) a "blank" (a missing constituent) in this context. we build fiber by following two main steps: (1) data generation, where we compile a large set of video-caption pairs with selectively blanked words; and (2) data annotation, where crowd workers provide additional valid answers for these blanks. note that we could also develop a fill-in-theblanks dataset by completing only the first step: the data generation. however, this would result in only one valid answer (the original blanked word or phrase), which can lead to unfair evaluations that are too strict because of alternative correct answers being dismissed (e.g., "child" provided as an answer where the blanked word was "kid"). other than manual annotations, we found no highquality method to automatically obtain additional correct answers. for example, "building" and "tshirt" in ## multimodal method for video fill-in-the-blanks we propose an encoder-decoder multimodal method to perform the task of video fill-in-theblanks. we first encode the text and visual modalities together to obtain a semantic representation of the blanked caption and video. the decoder uses the semantic representation to generate text corresponding only to the answer to the blank. to correctly generate an answer, a model needs to learn which parts of videos relate to the missing parts of the caption. to accomplish this, we use the original transformer architecture @xcite , whose self-attention mechanism is particularly effective for encoding relations within an input sequence and have been shown to perform well in many language understanding tasks. we consider two types of encoders, namely the early-fusion encoder and the late-fusion (twostream) encoder. the structure of our multimodal 5 https://eric-xw.github.io/ vatex-website/download.html transformer encoder _____ performs a shot put at an outdoor course. transformer decoder video feature extractor v1 v2 vm linear embedding ... video frames blanked caption g1 g2 gk ... a young person t1 t2 tn ... (a) multimodal transformer encoder transformer decoder video feature extractor v1 v2 vm linear ... video frames blanked caption g1 g2 gk ... a young person video transformer encoder text transformer encoder _____ performs a shot put at an outdoor course. model with an early-fusion encoder is shown in fig. 4a . the input to the model consists of the tokenized blanked caption-text t 1 , . . . , t n , as well as a representation of the video consisting of multiple video sequence features v 1 , . . . , v m from a video feature extractor. the blanked captions are embedded by an embedding layer. the video features are projected into the encoder by a linear layer. we use a special token to represent the masked phrase and another one to separate the input text and video sequences. we add positional embeddings to each input token or video feature to represent the sequence order, and another embedding to indicate whether it belongs to the text or video sequence similarly to bert @xcite . the late-fusion model is shown in fig. 4b . the late-fusion model encodes the language and video first separately and then jointly. this is because the modalities may benefit from learning independently about their own context before using them together. ## experiments and results we perform experiments and evaluations using the dataset described in section 3. ## conclusions this paper introduced the fill-in-the-blanks evaluation framework for video understanding. the framework addresses drawbacks of alternative video understanding tasks, such as multiple-choice visual question answering or video captioning. our paper makes three important contributions. first, we introduced fiber, which is a large dataset consisting of 28,000 videos and tests based on filling blanks, building upon an existing video captioning dataset with a new set of manual annotations, and using a modified annotation framework to encourage diverse responses among annotators. this process can be easily replicated to create new fill-in-the-blanks data for other datasets and tasks. second, we conducted extensive analyses on the dataset to evaluate the quality of the annotations and to understand the patterns and limitations of the data. finally, we introduced a multimodal model that fuses language and visual information and found that the video-aware models significantly outperform the text-only models. notably, we found a consistent gap between model performance and human performance, which suggests room for improvement in future models addressing video understanding through the lens of the fill-in-the-blanks task. the fiber dataset and our code are available at https://lit.eecs.umich.edu/ fiber/ .