End-to-end Dense Video Captioning as Sequence Generation (2022)

Abstract: Dense video captioning aims to identify the events of interest in an input video, and generate descriptive captions for each event. Previous approaches usually follow a two-stage generative process, which first proposes a segment for each event, then renders a caption for each identified segment. Recent advances in large-scale sequence generation pretraining have seen great success in unifying task formulation for a great variety of tasks, but so far, more complex tasks such as dense video captioning are not able to fully utilize this powerful paradigm. In this work, we show how to model the two subtasks of dense video captioning jointly as one sequence generation task, and simultaneously predict the events and the corresponding descriptions. Experiments on YouCook2 and ViTT show encouraging results and indicate the feasibility of training complex tasks such as end-to-end dense video captioning integrated into large-scale pretrained models.

## introduction online videos have become an important source of knowledge and skills @xcite . in order to help users locate information of interest, search engines and video platforms often show anchors at "key moments", usually accompanied by descriptions of the segment's content @xcite . this is a direct application of the dense video captioning task @xcite , thus methods for improving performance on this task are relevant to any video platform. intuitively, dense video captioning can be decomposed into two subtasks: event localization and segment-level video captioning. following this approach, prior work @xcite @xcite @xcite @xcite has used a two-stage, "localize-thendescribe" pipeline. such methods usually involve two separate modules with different underlying model architectures for event localization and event captioning, with captions for dense events rendered based on the predicted event spans. recently, with the advance of large-scale datasets and model architectures, there has been an explosion of pretrained multimodal (for text, image, video) transformer models @xcite @xcite @xcite @xcite . such models have proved to be highly effective when fine-tuned for a wide range of downstream tasks, such as visual question answering @xcite , image captioning @xcite , visual common sense reasoning @xcite , visual entailment @xcite , etc. these end-tasks can be expressed as sequence generation tasks in a straightforward manner. in contrast, this is non-trivial for dense video captioning, as the segmentation subtask does not lend itself naturally to such a formulation. does this mean more complex tasks cannot benefit from the pretraining paradigm in an end-to-end fashion? in this work, we study dense video captioning as an example of a complex task that can be cast as sequence generation and, as a result, can benefit from large-scale pretraining. more specifically, we propose to solve the dense video captioning task as a single sequence-tosequence modeling task using a multimodal transformer. to this end, we design several task formulations to encode both segmentation and caption prediction in one target string. thus, our task formulations allow the model to simultaneously predict event locations and corresponding captions in one pass, using one decoder. this opens the door to leveraging large-scale pretrained models, as well as the option of participating in large-scale multi-task training more easily by reusing existing infrastructure. we evaluate our model on two dense video captioning benchmarks, youcook2 @xcite and vitt @xcite . our sequence generation formulations provide a feasible path forward -we obtain encouraging results compared to prior work that used a two-stage scheme with specialized architectures for each step. on the pretraining front: (a) we are able to benefit from models pretrained on very different data and tasks, such as t5 @xcite , (b) pretraining on more domain-specific data (wikihow) and pretraining tasks (predicting headings for how-to steps) lead to a similar amount of gain, but (c) having the domain-specific pretraining start from a t5 checkpoint (t5 + wikihow) provides a significantly larger gain. the noteworthy result is that, even in the presence of large-scale domain-and task-specific pretraining (wikihow), one can still observe measurable benefits from a task-agnostic general-purpose pretrained model (t5). while the primary motivation for modeling the two tasks jointly is to be able to utilize the pretraining paradigm, the segmentation subtask (finding event boundaries) and the captioning subtask (describing what happens in an event) are related tasks, and intuitively stand to benefit from being modeled jointly. our experimental results are aligned with this intuition: a model that does both segmentation and captioning simultaneously, outperforms (in terms of segmentation accuracy) a variant that focuses only on the segmentation task. overall, our results point to a viable alternative direction for modeling complex tasks such as endto-end dense video captioning, in which we can leverage the large-scale pretraining paradigm to achieve modeling improvements. ## task definition the dvc task consists of annotating each input video into multiple segments, where each segment corresponds to an event of interest accompanied by a short description (caption). figure 1 shows an example from the youcook2 dataset. modified dense video captioning in youcook2, each segment is marked by a start and an end time, often with gaps between segments. the burden of identifying not just the right start-time but also the right end-time increases the difficulty of the welcome to videoculinary. today we're making shrimp tempura. to make the shrimp straight, make a few shallow horizontal cuts on the stomach side of the shrimp. gently press and massage it. this will prevent the shrimp from curling up during ... ## method as noted earlier, prior work often decomposes dense video captioning into two subtasks, (a) a segmentation subtask, and (b) a segment-level captioning subtask. these two subtasks are often addressed with different model architectures. in contrast, our approach solves both subtasks simultaneously with one single model. we first describe how we jointly model segmentation and captioning subtasks as one single sequence generation task. to this end, we need to formulate target strings in ways that encode both segmentation and captioning predictions. the typical input to a dvc task includes both visual information and speech in textual form -automatic speech recognition (asr) tokens. we start by introducing our target string formulations assuming only textual input, with segmentation information expressed in terms of the positions of the corresponding asr tokens foot_2 . we then describe multi-modal models where the visual information is added to the input while retaining the aforementioned scheme to represent segmentation information. ## conclusion in this paper, we describe different task formulations for solving the dense video captioning task using an end-to-end sequence generation approach, which allows us to leverage pre-trained text-only encoder-decoder models. we conduct experiments on youcook2 and vitt in several pretraining settings. experimental results show that general (t5) and in-domain (wikihow) text-only pre-trained models both improve video partitioning and segmentation performance, and the gains are cumulative. also, the segmentation subtask benefits from joint modeling with the captioning subtask. we hope our work can inspire future studies on leveraging pre-trained models, large-scale text corpora and language generation formulations to solve multimodal tasks such as dense video captioning.