Multimodal Pretraining for Dense Video Captioning (2020)

Abstract: Learning specific hands-on skills such as cooking, car maintenance, and home repairs increasingly happens via instructional videos. The user experience with such videos is known to be improved by meta-information such as time-stamped annotations for the main steps involved. Generating such annotations automatically is challenging, and we describe here two relevant contributions. First, we construct and release a new dense video captioning dataset, Video Timeline Tags (ViTT), featuring a variety of instructional videos together with time-stamped annotations. Second, we explore several multimodal sequence-to-sequence pretraining strategies that leverage large unsupervised datasets of videos and caption-like texts. We pretrain and subsequently finetune dense video captioning models using both YouCook2 and ViTT. We show that such models generalize well and are robust over a wide variety of instructional videos.

## introduction youtube recently reported that a billion hours of videos were being watched on the platform every day @xcite . in addition, the amount of time people spent watching online videos was estimated to grow at an average rate of 32% a year between 2013 and 2018, with an average person forecasted to watch 100 minutes of online videos per day in 2021 @xcite . an important reason for this fast-growing video consumption is information-seeking. for instance, people turn to youtube "hungry for how-to and learning content" @xcite . indeed, compared to traditional content format such as text, video carries richer information to satisfy such needs. but as a content media, videos are also inherently more difficult to skim through, making it harder to quickly target the relevant part(s) of a video. recognizing this difficulty, search engines started showing links to "key moments" within videos in search results, based on timestamps and short descriptions provided by the content creators themselves. 1 this enables users to get a quick sense of what the video covers, and also to jump to a particular time in the video if so desired. this effort echoes prior work in the literature showing how users of instructional videos can benefit from human-curated meta-data, such as a timeline pointing to the successive steps of a tutorial @xcite @xcite . producing such meta-data in an automatic way would greatly scale up the efforts of providing easier information access to videos. this task is closely related to the dense video captioning task considered in prior work @xcite , where an instructional video is first segmented into its main steps, followed by segment-level caption generation. to date, the youcook2 data set @xcite is the largest annotated data set for dense video captioning. it contains annotations for 2,000 cooking videos covering 89 recipes, with per-recipe training / validation split. restricting to a small number of recipes is helpful for early exploratory work, but such restrictions impose barriers to model generalization and adoption that are hard to overcome. we directly address this problem by constructing a larger and broader-coverage annotated dataset that covers a wide range of instructional topics (cooking, repairs, maintenance, etc.) we make the results of our annotation efforts publicly available as video timeline tags (vitt) foot_1 , consisting of around 8,000 videos annotated with timelines (on average 7.1 segments per video, each segment with a short free-text description). using youcook2 and the new vitt dataset as benchmarks for testing model performance and generalization, we further focus on the subproblem of video-segment-level caption generation, assuming segment boundaries are given @xcite @xcite . motivated by the high cost of collecting human annotations, we investigate pretraining a video segment captioning model using unsupervised signals -asr (automatic speech recognition) tokens and visual features from instructional videos, and unpaired instruction steps extracted from independent sources: recipe1m @xcite and wik-ihow @xcite . in contrast to prior work that focused on bert-style pretraining of encoder networks @xcite , our approach entails jointly pretraining both multimodal encoder and text-based decoder models via massstyle pretraining @xcite . our experiments show that pretraining with either text-only or multi-modal data provides significant gains over no pretraining, on both the established youcook2 benchmark and the new vitt benchmark. the results we obtain establish state-of-the-art performance on youcook2, and present strong performance numbers on the vitt benchmark. these findings help us conclude that the resulting models generalize well and are quite robust over a wide variety of instructional videos. ## data we present the datasets used for pretraining, finetuning, and evaluation in ## method to model segment-level caption generation, we adopt mass-style pretraining @xcite with transformer @xcite as the backbone architecture. for both pre-training and fine-tuning objectives, we have considered two variants: text-only and multi-modal. they are summarized in ## conclusions motivated to improve information-seeking capabilities for videos, we have collected and annotated a new dense video captioning dataset, vitt, which is larger with higher diversity compared to youcook2. we investigated several multimodal pretraining strategies for segment-level video captioning, and conducted extensive ablation studies. we concluded that mass-style pretraining is the most decisive factor in improving the performance on all the benchmarks used. even more to the point, our results indicate that most of the performance can be attributed to leveraging the asr signal. we achieve new state-of-the-art results on the youcook2 benchmark, and establish strong performance baselines for the new vitt benchmark, which can be used as starting points for driving more progress in this direction.