HERO : Hierarchical Encoder for V ideo+ L anguage Omni-representation Pre-training (2020)

Abstract: We present HERO, a novel framework for large-scale video+language omni-representation learning. HERO encodes multimodal inputs in a hierarchical structure, where local context of a video frame is captured by a Cross-modal Transformer via multimodal fusion, and global video context is captured by a Temporal Transformer. In addition to standard Masked Language Modeling (MLM) and Masked Frame Modeling (MFM) objectives, we design two new pre-training tasks: (i) Video-Subtitle Matching (VSM), where the model predicts both global and local temporal alignment; and (ii) Frame Order Modeling (FOM), where the model predicts the right order of shuffled video frames. HERO is jointly trained on HowTo100M and large-scale TV datasets to gain deep understanding of complex social dynamics with multi-character interactions. Comprehensive experiments demonstrate that HERO achieves new state of the art on multiple benchmarks over Text-based Video/Video-moment Retrieval, Video Question Answering (QA), Video-and-language Inference and Video Captioning tasks across different domains. We also introduce two new challenging benchmarks How2QA and How2R for Video QA and Retrieval, collected from diverse video content over multimodalities.

## introduction inspired by bert @xcite , largescale multimodal pre-training has prevailed in the realm of vision-and-language research @xcite @xcite . there are many early players in the area, including vilbert @xcite , lxmert @xcite , uniter @xcite , vl-bert @xcite and unicoder-vl @xcite . however, most large-scale pre-trained models are tailored for static images, not dynamic videos. videobert @xcite is the first to apply bert to learn joint embedding for videotext pairs. but since only discrete tokens are used to represent video frames, rich video frame features are not fully utilized. to remedy this, cbt @xcite proposes to use a contrastive loss, but mainly for video representation learning alone, with text input only considered as side information. univilm @xcite takes a step further and considers both understanding and generation tasks. several constraints inherently limit the success of existing models. (i) most model designs are direct adaptation of bert, taking simple concatenation of subtitle sentences and visual frames as input, while losing the temporal alignment between video and text modalities. (ii) pre-training tasks are directly borrowed from image+text pre-training methods, without exploiting the sequential nature of videos. (iii) compared to diverse image domains investigated in existing work, video datasets used in current models are restricted to cooking or narrated instructional videos @xcite , excluding video sources that contain dynamic scenes and complex social interactions. to tackle these challenges, we present a new video-and-language large-scale pre-training framework -hero (hierarchical encoder for omnirepresentation learning). as illustrated in figure 1 , hero takes as input a sequence of video clip frames and their accompanying subtitle sentences. 2 instead of adopting a flat bert-like encoder, hero encodes multimodal inputs in a hierarchical fashion, with (i) a cross-modal transformer to fuse a subtitle sentence and its accompanying local video frames, followed by (ii) a temporal transformer to obtain a sequentially contextualized embedding for each video frame, using all the surrounding frames as global context. the proposed hierarchical model first absorbs visual and textual local context on frame level, which is then transferred to a global video-level temporal context. experiments show that this novel model design achieves better performance than a flat bert-like architecture. four pre-training tasks are designed for hero: (i) masked language modeling (mlm); (ii) masked frame modeling (mfm); (iii) video-subtitle matching (vsm); and (iv) frame order modeling (fom). compared to prior work, the key novelty is vsm and fom, which encourage explicit temporal alignment between multimodalities as well as full-scale exploitation of the sequential nature of video input. in vsm, the model considers not only global alignment (predicting whether a subtitle matches the input video clip), but also local temporal alignment (retrieving the moment where the subtitle should be localized in the video clip). in fom, we randomly select and shuffle a subset of video frames, and the model is trained to restore their original order. extensive ablation studies demonstrate that both vsm and fom play a critical role in video+language pre-training. to empower the model with richer knowledge beyond instructional videos used in prior work, we jointly train hero with both howto100m (narrated instructional videos) @xcite and a large-scale tv dataset (containing tv episodes spanning across different genres) @xcite @xcite . compared to factual descriptions in howto100m, the tv dataset contains more complex plots that require comprehensive interpretation of human emotions, social dynamics and causal relations of events, making it a valuable supplement to howto100m and a closer approximation to real-life scenarios. existing pre-trained models are evaluated on youcook2 @xcite and msr-vtt @xcite datasets. youcook2 focuses on cooking videos only, and the captions in msr-vtt are very simple. to evaluate our model on more challenging benchmarks, we collect two new datasets on video-moment retrieval and question answering, how2r and how2qa. in addition, we evaluate hero on popular retrieval and qa tasks such as tvr @xcite and tvqa @xcite , where hero outperforms existing models by a large margin. we further demonstrate the generalizability of our model by adapting it to (i) diverse downstream tasks: video-and-language inference and video captioning tasks, achieving new state of the art on violin @xcite and tvc @xcite benchmarks; (ii) different video types: single-channel videos (video-only) and multi-channel videos (video + subtitle), reporting superior performance over existing state of the art on didemo (anne hendricks et al., 2017a) and msr-vtt. our main contributions are summarized as follows. (i) we present hero, a hierarchical transformer-based model for video+language representation learning. (ii) we propose new pretraining tasks vsm and fom, which complement mlm and mrm objectives by better capturing temporal alignment between multimodalities in both global and local contexts. (iii) different from previous work that mainly relies on howto100m, we include additional video datasets for pre-training, encouraging the model to learn from richer and more divserse visual content. (iv) we collect two new datasets based on howto100m for video-moment retrieval/qa, and will release the new benchmarks to foster future study. hero achieves new state of the art across all the evaluated tasks. ## related work since the birth of bert @xcite , there has been continuing advancement in language model pre-training, such as xlnet @xcite , roberta @xcite , albert @xcite , unilm @xcite , and t5 @xcite , which epitomizes the superb power of large-scale pre-training. satellited around bert, there is parallel growing interest in model compression @xcite and extension to generation tasks @xcite . branching out from language processing to multimodal, subsequent studies also emerge in vi-sion+language space. prominent work includes vilbert @xcite , lxmert @xcite , vl-bert @xcite , unicoder-vl @xcite , b2t2 @xcite , uniter @xcite and villa @xcite . a detailed review can be found in appendix a.7. contrast to the boom in image+text area, pretraining for video+language is still in its infancy. so far, videobert @xcite , cbt @xcite , mil-nce @xcite , act-bert @xcite and univilm @xcite are the only existing work exploring this space, covering downstream tasks from textbased video retrieval @xcite and video question answering @xcite to video captioning @xcite . in this paper, we aim to propel video+language omni-representation learning in four dimensions: (i) better model architecture design; (ii) better pretraining task design; (iii) diversification of training corpora; and (iv) new high-quality benchmarks for downstream evaluation. ## hierarchical video+language encoder in this section, we explain the proposed hero architecture and the four pre-training tasks in detail. ## experiments in this section, we describe comprehensive experiments on downstream tasks and provide ablation studies for in-depth analysis of different pretraining settings. to validate the effectiveness of hero, we evaluate on a wide variety of downstream tasks, including text-based video/ video-moment retrieval, video question answering, video-and-language inference, and video captioning. we consider 6 existing benchmarks: tvr @xcite , tvqa @xcite , violin @xcite , tvc @xcite , didemo (anne hendricks et al., 2017a), and msr-vtt @xcite . detailed descriptions and evaluation metrics on each task can be found in appendix a.6. ## conclusion in this paper, we present a hierarchical encoder for video+language omni-representation pre-training. our hero model presents a hierarchical architecture, consisting of cross-modal transformer and temporal transformer for multi-modal fusion. novel pre-training tasks are proposed to capture temporal alignment both locally and globally. pretrained on two large-scale video datasets, hero exceeds state of the art by a significant margin when transferred to multiple video-and-language tasks. two new datasets on text-based video-moment retrieval and video qa are introduced to serve as additional benchmarks for downstream evaluation. we consider extension of our model to other videoand-language tasks as future work, as well as developing more well-designed pre-training tasks. downstream task pre-training video ret. moment ret. 18 video moment ret. 18 r@1 r@10 r@100 r@1 r@10 r@100 r@1 r@10 r@100 pre-training greatly lifts hero performance on violin by approximately +2.9%. however, hero, without pre-training, presents worse performance than the sota baseline. unlike multistream, which leverages fine-grained region-level features, our results are reported on global framelevel features. therefore, it may be difficult for hero to capture the inconsistency between hypothesis and video content. for example, changes of hypotheses about region-level attributes (color, shape, and etc.) may result in different conclusions. extending hero for region-level video representations could be an interesting future direction. hero is also extensible to generation task: multi-modal video captioning. our results on tvc show that hero with pre-training surpasses mmt by a large margin. although pre-training is only applied to the encoder, it significantly improves hero performance on tvc across all metrics. when no pre-training is applied, hero is slightly inferior to the sota baseline. our hypothesis is that tvc has short video context (with video length of 9-second on average) but our model is designed for long video representation learning (tvr/tvqa with video length of 76-second on average). how to design pre-training tasks for mmt on tvc or including decoder pre-training for hero are left for future works.