A Challenging Multimodal Video Summary: Simultaneously Extracting and Generating Keyframe-Caption Pairs from Video (2023)

Abstract: This paper proposes a practical multimodal video summarization task setting and a dataset to train and evaluate the task. The target task involves summarizing a given video into a predefined number of keyframe-caption pairs and displaying them in a listable format to grasp the video content quickly. This task aims to extract crucial scenes from the video in the form of images (keyframes) and generate corresponding captions explaining each keyframe’s situation. This task is useful as a practical application and presents a highly challenging problem worthy of study. Specifically, achieving simultaneous optimization of the keyframe selection performance and caption quality necessitates careful consideration of the mutual dependence on both preceding and subsequent keyframes and captions. To facilitate subsequent research in this field, we also construct a dataset by expanding upon existing datasets and propose an evaluation framework. Furthermore, we develop two baseline systems and report their respective performance.

## introduction the popularity of video sharing platforms has increased, which has resulted in a substantial increase in daily video-watching by individuals. as a result, there is increasing interest in practical video summarization systems that can comprehend video content efficiently, and many previous studies have proposed different automatic video summarization methods to address this demand. most early attempts only considered video and image data, and these methods were developed in the vision and image processing community @xcite , e.g., keyframe detection @xcite @xcite and video storyboarding @xcite . however, the recent trend 1 this dataset and our systems (code) are publicly available at https://github.com/keitokudo/multi-vidsum . n pairs of keyframe & caption the number of keyframe 4 = multi-vidsum system figure 1: overview of the multi-vidsum task. we assume that the number of the output keyframe-caption pairs is given (e.g., 4) depending on summary display space and user preference. we first formulate the task definition and develop evaluation metrics to assess the task appropriately. then, we construct training and evaluation datasets to process this task. in addition, we develop two baseline systems using the constructed data, thereby providing an effective benchmark for future research and offering insights into handling the target task based on performance comparisons. the challenge of this task lies in the simultaneous execution of keyframe selection and caption generation while maintaining sufficient consider-ation of their relative importance. specifically, to generate appropriate captions, we attempt to select keyframes that are well-aligned with the desired content @xcite . thus, the system must consider these interdependencies and make optimal choices for keyframes and captions, which is challenging. therefore, the proposed task provides a practical setting that bridges the gap with realworld applications. our primary contributions are summarized as follows: (1) we propose the challenging and practical multi-vidsum task. (2) we generate a dataset for this task to facilitate effective training and evaluation of relevant systems. (3) we develop a new evaluation metric for this task. (4) we develop and analyze two baseline systems, thereby providing a benchmark and insights for future research. ## related work here, we introduce several related studies. note that other related works (e.g., multinodal generation and keyframe detection) are listed in appendix a. multimodal summarization multimodal summarization studies have explored incorporating images in document summarization @xcite . similarly, in terms of video summarization tasks, previous studies have proposed methods to generate summary results using both images and text @xcite . however, in these methods, the text output by the model is extracted from the input automatic speech recognition results or documents; thus, such methods do not generate summaries by considering visual information explicitly. in our research, our goal is to generate summaries by considering visual information and selecting appropriate frames simultaneously. video storytelling video storytelling @xcite , which is a promising video summarization task, has limitations in terms of its dataset, model, and evaluation criteria. for example, the relevant dataset is small and inadequate for training and evaluation. in addition, this method relies on gold data to derive the number of keyframecaption pairs, which results in unrealistic task settings. also, keyframe detection was not assessed due to an absence of keyframe annotations. therefore, in the current study, we constructed a large dataset with corrected keyframe annotations from the open domain to address these limitations. dense video captioning in the dense video captioning task, events in video segments are detected and captions are provided. for example, a previous study @xcite introduced the first dataset for this task, i.e., the activitynet captions, and proposed a corresponding baseline system. we extend this task to the proposed multi-vidsum task, where keyframes serve as compressed segments. in addition, we create a distinct multi-vidsum dataset by adding keyframe annotations to the existing ac-tivitynet captions. the multi-vidsum task poses a more significant challenge as it requires a precise selection of n keyframe-caption pairs as representative summaries. visual storytelling similar to the multi-vidsum task, visual storytelling @xcite has been proposed previously. the visstory task involves generating story text that is relevant to the input images. although the concept of the visstory task is similar to that of the multi-vidsum task, the multi-vidsum takes the video data as the input; thus, it is a more challenging setting than the visual storytelling task. in addition, we use this dataset to pretrain our baseline model (see section 5.3.1 for details). ## proposed multi-vidsum task in this section, we define the proposed multi-vidsum task. the multimodal multi-vidsum task attempts to present users with a set of keyframes from the input video with relevant captions. ## multi-vidsum dataset we constructed a dataset for training and evaluating the multi-vidsum task in machine learning approaches. to impact the community more, we constructed a dataset for the multi-vidsum task by expanding activitynet captions @xcite , which is commonly used, including the shared task of dense-captioning events in videos. 5 we tasked crowd workers to add keyframe annotations that align with the captions in the ac-tivitynet captions. 6 note that the original activi-tynet captions dataset has no keyframe annotation; thus, including the additional annotations to handle the proposed multi-vidsum task is considered a unique contribution to the field. ## baseline systems here, we propose two baseline systems to tackle the proposed task. to address the challenges discussed in section 3.3.1, we present two different baseline systems: iterative refinement model, which iteratively selects keyframes and generates captions alternately and simul-determination model that selects and generates all keyframes and captions collectively. in simul-determination model, given the anticipated difficulty of simultaneous selection and generation of all keyframes and captions, we implement a method that incorporates a pretrained image captioning model. by comparing the outcomes of these two baseline systems, we seek to provide insights to researchers interested in undertaking future investigations of related tasks. ## experiments and results settings as described in section 3.4, we employed multiple evaluation metrics to assess the keyframes and captions. specifically, in this experimental evaluation, we used akmex and akmcos for the keyframes and the meteor @xcite and bleurt @xcite for the captions. for the test set, we constructed a dataset consisting of 439 videos. in addition, to ensure accuracy and consistency, we performed a re-captioning process on the videos (section 4.3). regarding the training data, the specifics differ among the baseline models discussed in section 5. results ## conclusion in this paper, we have proposed a practical and challenging task setting of a multimodal video summarization, multi-vidsum. we also extended the ac-tivitynet captions with the keyframe annotations obtained by the human annotators (crowd workers). to the best of our knowledge, our dataset is the first large video dataset with keyframe annotation, and its quality was assured by human evaluations. we also proposed evaluation criteria of multi-vidsum task to appropriately evaluate this task. extensive experiments were conducted on our dataset using two baseline methods, and we provided a benchmark and relevant insights. we hope our dataset will make new movements in vision-and-language research and accelerate them.