Video Paragraph Captioning as a Text Summarization Task (2021)

Abstract: Video paragraph captioning aims to generate a set of coherent sentences to describe a video that contains several events. Most previous methods simplify this task by using ground-truth event segments. In this work, we propose a novel framework by taking this task as a text summarization task. We first generate lots of sentence-level captions focusing on different video clips and then summarize these captions to obtain the final paragraph caption. Our method does not depend on ground-truth event segments. Experiments on two popular datasets ActivityNet Captions and YouCookII demonstrate the advantages of our new framework. On the ActivityNet dataset, our method even outperforms some previous methods using ground-truth event segment labels.

## introduction video captioning, the task of describing the content of a video in natural language, is a popular task both in computer vision and natural language processing. in the beginning, researchers try to generate sentence-level captions for short video clips @xcite . @xcite propose the task of dense video captioning. the system needs to detect event segments first and then generate captions. @xcite propose the task of video paragraph captioning: they use ground-truth event segments and focus on generating coherent paragraphs. @xcite follow the task setting and propose a recurrent transformer model that can generate more coherent and less repetitive paragraphs. considering the groundtruth event segments are often unavailable in practice, our goal is to generate paragraph captions without ground-truth segments. the conventional framework of video paragraph captioning is shown in figure 1a . given an untrimmed video, an event detection module outputs a set of non-redundant event segments. the event captioning module generates captions for these segments. the works of (park et al., 2019; zhou et al., 2019; lei et al., 2020) use ground-truth event segments and focus on the event captioning module. zhou et al. (2019) use extra humanannotated bounding boxes as supervision. (sah et al., 2017; zhou et al., 2018; mun et al., 2019) use predicted event segments and generate captions based on them. sah et al. (2017) also summarizes these captions to generate a paragraph. the above methods heavily depend on accurate event segments. according to previous works (zhou et al., 2018; mun et al., 2019), the performance of the event detection module is not so good, making it a performance bottleneck. to tackle this problem, we propose a novel framework vpcsum as shown in figure 1b . for a given video, we first extract dense event segment candidates (we call proposals), and a proposal captioning module is used to generate proposal captions. then we treat video paragraph captioning as a text summarization task to obtain the final summary (paragraph caption). in this work, we only consider extractive summarization, where the paragraph caption is composed by selecting from proposal captions. we conduct experiments on two popular datasets activitynet captions and youcookii. the results demonstrate the advantages of our framework. on the activ-itynet captions dataset, our method even outperforms some previous methods using ground-truth event segment labels. ## our vpcsum method as illustrated in figure 1b , our framework has three modules. proposal extraction: it extracts dense proposals for a video; proposal captioning: it generates captions for extracted proposals; caption summarization: it summarizes the generated proposal captions to obtain the video paragraph caption. we will introduce each module next. ## conclusion in this work, we view the task of video paragraph captioning as a text summarization task and propose a novel framework vpcsum. it allows us to use text summarization techniques to handle this challenging task. experimental results on two popular datasets show the advantages of our model. in the future, we will explore using abstractive summarization methods to generate better video paragraph captions.