MM - AVS : A Full-Scale Dataset for Multi-modal Summarization (2021)

Abstract: Multimodal summarization becomes increasingly significant as it is the basis for question answering, Web search, and many other downstream tasks. However, its learning materials have been lacking a holistic organization by integrating resources from various modalities, thereby lagging behind the research progress of this field. In this study, we release a full-scale multimodal dataset comprehensively gathering documents, summaries, images, captions, videos, audios, transcripts, and titles in English from CNN and Daily Mail. To our best knowledge, this is the first collection that spans all modalities and nearly comprises all types of materials available in this community. In addition, we devise a baseline model based on the novel dataset, which employs a newly proposed Jump-Attention mechanism based on transcripts. The experimental results validate the important assistance role of the external information for multimodal summarization.

## introduction multimodal summarization refines salient information from one or more modalities, including text, image, audio, and video ones @xcite . given the rapid dissemination of multimedia data over the internet, multimodal summarization has been widely explored in recent years. meanwhile, some multimodal datasets @xcite @xcite have been introduced to advance the development of this research field. however, a majority of them are restricted in scale and too oriented, such as being less than one hundred examples or merely containing chinese texts. moreover, the materials from different modalities are rarely collected across the board, especially videos and their accompanying materials that possess abundant external information for multimodal comprehension and fusion. in this work, we introduce a full-scale multimodal article and video summarization (mm-avs) dataset foot_0 with documents, summaries, images, captions, videos, audios, transcripts, and titles in english. the significance of mm-avs for the multimodal summarization community includes but not limited to: 1) mm-avs is a large-scale multimodal collection compared with existing video containing dataset and its generation codes 1 has been released, which can be readily extended for existing and future multimodal summarization approaches; 2) mm-avs is collected from cnn foot_1 and daily mail foot_2 , which makes it available to more researchers due to english-based and comparable with the popular text-based cnn/daily mail corpus; and 3) mm-avs firstly collects nearly all types of materials from all modalities, inclusively with videos, audios, transcripts, images, captions, and titles that are rarely assembled. in addition, we implement a general multimodal summarization baseline based on transcripts for multimodal summarization on mm-avs. this method employs a jump-attention mechanism to align features between text and video. further, we use the multi-task learning to simultaneously optimize document and video summarizations. evaluations on mm-avs illustrate the benefits of external information such as videos and transcripts for multimodal summarization without alignment. ## related work multi-modal summarization generates a condensed multimedia summary from multi-modal materials, such as texts, images, and videos. for instance, @xcite introduced an idea of illustrating complex sentences as multimodal summaries by combining pictures, structures, and sim-dataset doc summary image video title abs.sum ext.label image caption video audio* transcript* msmo @xcite mmss @xcite e-dailymail (chen and zhuge, 2018) ec-product @xcite mms @xcite how2 ( sanabria et al., 2018 compressed texts. libovickỳ et al. (2018) and palaskar et al. (2019) studied abstractive text summarization for open-domain videos. li et al. (2017) constructed mms dataset and developed an extractive multi-modal summarization method that automatically generated a textual summary based on a topic-related set of documents, images, audios, and videos. @xcite combined image selection and output to alleviate the modality bias based on the msmo dataset. chen and zhuge (2018) extended daily mail with images and captions to e-daily mail dataset and employed a hierarchical encoder-decoder model to align sentences and images. recently, an aspectaware model and a large-scale chinese e-commerce product summarization dataset ec-product were introduced to incorporate visual information for e-commerce product summaries @xcite . the above mentioned datasets are rarely constructed comprehensively, which ignore the abundant visual information underlying in videos. the only video-containing work is restricted in scale, which hampers its use for deep-learning based methods. in this study, we will build a full-scale multimodal dataset to address these issues. ## mm-avs dataset to facilitate a straightforward comparison for the multimodal summarization approaches with the text-based ones, mm-avs extends cnn/dm collections to multimodalities. each example of mm-avs contains a document accompanying with multi-sentence summary, title, images, captions, videos, and their corresponding audios and transcripts. ## experiments we conduct experiments on the mm-avs dataset and evaluate the performance by rouge @xcite . r-1, r-2, and r-l respectively represent rouge-1, rouge-2, and rouge-l f1-scores, which are widely used to calculate the n-grams overlapping between decoded summaries and references.