D e CEMBERT : Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization (2021)

Abstract: Leveraging large-scale unlabeled web videos such as instructional videos for pre-training followed by task-specific finetuning has become the de facto approach for many video-and-language tasks. However, these instructional videos are very noisy, the accompanying ASR narrations are often incomplete, and can be irrelevant to or temporally misaligned with the visual content, limiting the performance of the models trained on such data. To address these issues, we propose an improved video-and-language pre-training method that first adds automatically-extracted dense region captions from the video frames as auxiliary text input, to provide informative visual cues for learning better video and language associations. Second, to alleviate the temporal misalignment issue, our method incorporates an entropy minimization-based constrained attention loss, to encourage the model to automatically focus on the correct caption from a pool of candidate ASR captions. Our overall approach is named DeCEMBERT (Dense Captions and Entropy Minimization). Comprehensive experiments on three video-and-language tasks (text-to-video retrieval, video captioning, and video question answering) across five datasets demonstrate that our approach outperforms previous state-of-the-art methods. Ablation studies on pre-training and downstream tasks show that adding dense captions and constrained attention loss help improve the model performance. Lastly, we also provide attention visualization to show the effect of applying the proposed constrained attention loss.

## introduction video and language are ubiquitous in the world we live. the ability to understand the interplay of video and language is thus essential for intelligent agents to operate in real-world scenario. past success in video-and-language has mostly been driven by supervised learning, where models are learned on manually labeled data for a particular task (e.g., text-to-video retrieval). however, manually annotating video and language data is very expensive, hence limiting the scale of such datasets, and consequently also limiting the performance of models trained on the datasets. the self-supervised pretraining then finetuning paradigm offers an easy and generic solution to this dilemma, where models are first pre-trained on large-scale unlabeled data by performing various "proxy tasks", followed by finetuning the pre-trained model on downstream tasks where data is often limited. recent advances on language pre-training @xcite demonstrate the effectiveness of this approach, where transformerbased @xcite models pre-trained on large-scale unlabeled text corpus has shown to perform remarkably well across a wide range of natural language tasks @xcite @xcite . following this momentum, multimodal pre-training @xcite @xcite @xcite @xcite @xcite @xcite on large-scale image-text corpus @xcite @xcite and video-text corpus @xcite @xcite have also shown to outperform existing approaches @xcite @xcite on vision and language tasks @xcite @xcite @xcite . the most commonly used "proxy tasks" for multimodal pre-training are masked language modeling @xcite (mlm) and cross-modal matching @xcite @xcite ) (e.g., video-text matching), where mlm aims to learn a better language model in the presence of the extra asr captions dense captions [01:15] easier start [01:17] taking pieces paper go a blue paper, the table is made of wood, ... a green and white paper, the hand is holding a paper, ... a blue and white paper, the hand is on the table, ... @xcite . we show three clips and their corresponding asr captions and dense captions. we use green box to indicate correct matched asr caption for the middle clip. we highlight semantically misaligned asr caption in pink. as can be seen from this example, the asr captions are often incomplete and unpunctuated, and are semantically or temporally misaligned with their corresponding clips. in contrast, dense captions typically capture key objects, attributes and actions in the clips. to address the aforementioned issues, we propose to add dense captions @xcite as a complementary text input to the asr captions. beyond serving as an extra language input for better language modeling, dense captions also describes important object, attribute, and action details regarding several salient regions in the video frames, providing useful signals for video-text matching. in addition to its use in the pre-training stage, these dense captions also provide helpful clues for downstream tasks such as video question answering. in parallel, to alleviate the temporal misalignment issue, we propose a constrained attention loss that encourages the model to automatically focus on the relevant asr caption from a pool of continuous caption candidates. instead of using only a single paired asr caption for each clip, we also use the captions from its neighboring clips. we expect one of neighboring captions semantically aligns with the clip. to encourage the alignment between the clip and its relevant caption, we employ a "constrained attention loss" that encourages the attention mass from video features to the captions to be distributed mostly in one of the caption, by minimizing the entropy of attention scores. we evaluate our decembert (dense captions and entropy minimization) model on a wide range of video-and-language tasks, including video question answering @xcite ), text-to-video retrieval @xcite , and video captioning @xcite , where our approach outperforms previous state-of-the-art methods. to better understand the underlying factors that contribute to this success, we present comprehensive analyses concerning each of the added components. to summarize, our contribution is three-fold: (i) we propose incorporating automatically extracted dense captions as an extra text input for video-text pre-training. (ii) we propose an entropy minimization-based constrained attention loss to encourage the model to dynamically select the best matched captions from a pool of neighboring captions, to alleviate the inherent misalignment between the asr captions and the videos. (iii) extensive experiments on three video-and-language tasks (text-to-video retrieval, video captioning, and video question answering) across five datasets demonstrate the effectiveness of our approach. furthermore, we also provide comprehensive ablation study and visualization to quantitatively and qualitatively examine the effect of using dense captions and the proposed constrained attention loss. ## related work since the birth of bert @xcite , transformer @xcite language pre-training models @xcite @xcite @xcite @xcite which perform unsupervised pre-training followed by downstream task specific finetuning has became the de facto approach for various natural language understanding tasks @xcite @xcite . followed by this success, image-and-language pretraining models @xcite @xcite @xcite and video-and-language pre-training models @xcite @xcite @xcite @xcite have also shown promising results on many vision and language tasks @xcite @xcite . for video-and-language pre-training in particular, most existing work @xcite @xcite @xcite ) are trained on large-scale unlabeled instructional videos, such as howto100m @xcite videos. however, as the asr captions associated with these videos are noisy, i.e., they are often temporally or semantically misaligned with the video content. @xcite propose multiple instance learning noise contrastive learning (mil-nce) to address the temporal misalignment issue, but semantic misalignment still remains. moreover, mil-nce requires computing a separate similarity score from the target clip to each of the asr caption candidates, it does not suitable for the prevailing single-stream transformer pre-training ar-chitecture due to linearly increased computation cost. inspired by recent work @xcite ) that uses dense captions @xcite to improve image and video qa models, we propose to add dense captions an as auxiliary text input that provide aligned visual cues to ease the difficulties of learning a video-text matching objective from often temporally and semantically misaligned asr captions. in addition, we also propose a constrained attention loss, which employs an entropy minimizationbased regularization @xcite to the model to encourage higher attention scores from the video to the correct matched caption among a pool of asr caption candidates. ## method in this section, we describe the details of decem-bert, including its architecture, pre-training objectives, dense caption inputs, and the constrained attention loss. figure 2 shows an overview of de-cembert. input representations. input text (e.g., asr captions) are tokenized and represented as a sequence of wordpiece @xcite tokens. we use a trainable word embedding layer to encode the tokens into feature representations. we use appearance and motion features to represent videos. for appearance, we use a resnet152 @xcite model pre-trained on imagenet @xcite to extract 2d video features at 1fps. similarly, for motion, we use a 3d resnext @xcite @xcite to extract 3d video features at 1fps. the temporally aligned appearance and motion features are l2-normalized and concatenated together at feature dimension. we then apply a two-layer mlp to map the it to the same dimension as the word embeddings. next, we add learned positional embedding and token type embedding @xcite to the video and text representations to encode the position and token type information. the video and text representations are then concatenated as a single sequence as inputs to a 12-layer transformer encoder for pre-training and downstream task finetuning. dense captions. the original captions from asr systems might not well describe a video with rich content or can even be irrelevant to the video as discussed in section 1. moreover, as asr cap- transformer + position embedding + type embedding + type embedding video-text matching masked language modeling constrained attention loss video a blue paper, the table is made of wood, ... easier start taking pieces paper go cross dense captions asr captions tions are often incomplete and unpunctuated, they might also be sub-optimal for language modeling. therefore, we use dense captions @xcite automatically extracted from an off-the-shelf image dense captioning model @xcite as additional language input for the model. this dense captioning model is pre-trained on visual genome @xcite regional captions. to obtain video-level captions, we extract dense captions from frames sampled at every two seconds. there are on average 4.4 dense captions per frame, we sample two of them from each frame at each training step to avoid redundant information and reduce memory and computation cost. note that the other dense captions might still be sampled in another training step. the sampled dense captions are then concatenated together as video-level captions for training. these extracted dense captions provide rich and comprehensive information regarding the salient objects, attributes, and actions (see examples in figure 1 and figure 2 ), which helps to optimize a video-text matching objective during pre-training and provide essential visual clues for many downstream tasks such as video question answering. meanwhile, because the dense captions are text input with diverse semantics, it complements the typically short and incomplete asr captions as additional resources for better language modeling. we observe in our ablation study that adding dense captions improves both mlm accuracy and videotext matching accuracy, demonstrating the effectiveness of using them as extra inputs. pre-training objectives. during pre-training, we use masked language modeling @xcite (mlm) and cross-modality matching @xcite @xcite (also referred as videotext matching in our context) as our objectives to learn model parameters. for masked language modeling, the goal is to learn better language models conditioned on bidirectional text context and the video. we set a probability of 0.20 foot_1 to replace an input language token with [mask]. when dense captions are used as extra text input, we also perform masked language modeling on them with the same masking probability as the asr captions. for video-text matching, with a probability of 0.50, we replace the original asr captions with randomly sampled captions from other videos or clips as a negative. of the sampled negative asr captions, 50% of them are from different videos, while another 50% are from the same video but different clips. text from the same video clip is likely to have the same theme or similar context, and thus can serve as hard samples to improve the model's ability to do fine-grained matching. we do not designate a [cls] token before the start of input caption, instead we take the mean pooling of the output sequence hidden states to perform binary classification for video-text matching. empirically, we found this approach works better than using a ## experiments in this section, we compare our model with state-ofthe-art methods on three video-and-language downstream tasks (e.g., video captioning, text-to-video retrieval, and video question answering) across five datasets. we then present a comprehensive ablation study, where we show that each of our proposed components help improve the pre-training task performance and downstream task performance. lastly, we also provide an attention visualization example to demonstrate the effect of applying our proposed constrained attention loss. ## conclusion in this work, we propose decembert as an improved pre-training method for learning from noisy, unlabeled instructional videos. specifically, we propose adding automatically-extracted framelevel dense captions as an auxiliary text input for learning better video and language associations. we also propose a constrained attention loss that forces the model to automatically focus on the bestmatched caption from a pool of misalignment caption candidates via entropy minimization. comprehensive experiments on three popular video and language tasks (i.e., text-to-video retrieval, video captioning, and video question answering) across five datasets demonstrate the effectiveness of de-cembert compared to existing approaches. we also provide detailed ablation study and visualization to quantitatively and qualitatively examine the impact of our added components.