MLASK : Multimodal Summarization of Video-based News Articles (2023)

Abstract: In recent years, the pattern of news consumption has been changing. The most popular multimedia news formats are now multimodal - the reader is often presented not only with a textual article but also with a short, vivid video. To draw the attention of the reader, such video-based articles are usually presented as a short textual summary paired with an image thumbnail. In this paper, we introduce MLASK (MultimodaL Article Summarization Kit) - a new dataset of video-based news articles paired with a textual summary and a cover picture, all obtained by automatically crawling several news websites. We demonstrate how the proposed dataset can be used to model the task of multimodal summarization by training a Transformer-based neural model. We also examine the effects of pre-training when the usage of generative pre-trained language models helps to improve the model performance, but (additional) pre-training on the simpler task of text summarization yields even better results. Our experiments suggest that the benefits of pre-training and using additional modalities in the input are not orthogonal.

## introduction automatic summarization is one of the basic tasks both in natural language processing -text summarization -and in computer vision -video summarization. multimodal summarization (mms) builds a bridge between those two fields. early works on multimodal summarization explored the usage of the secondary modality as an auxiliary source of information to guide the refinement process of the main modality. @xcite collected videos and news articles covering a handcrafted list of recent significant world events by querying a web search engine and trained a model to mimic the reference summaries written by human annotators. @xcite were the first to introduce the task of multimodal summarization with multimodal output (msmo). they collected a large-scale dataset of news articles paired with corresponding images and trained a system to generate a textual summary and choose a single image as a pictorial summary. by introducing the multimodal output, a uni-modal solution was no longer sufficient as a baseline. building upon this, @xcite extended the task to video-based msmo. based on a textual document and a short video clip, besides generating the textual summary, the system was also challenged to select a single frame from the video as a cover picture. we believe there is still a lot of questions that remain unanswered, e.g.: how to evaluate multimodal outputs? or how to approach pre-training? in this paper, we contribute to the area of videobased msmo (vmsmo) by: 1) introducing a fullscale vmsmo dataset in czech, extending the very limited available resources for this task to a new language; 2) exploring the pre-training strategies by transferring knowledge from the simpler task of text-to-text summarization; 3) re-defining the training labels to consider intra-video similarities; 4) proposing a human evaluation framework for assessing the quality of vmsmo. ## related work in our work, we build upon recent advances in three fields: text summarization, video summarization, and multimodal summarization. ## mlask dataset ## multimodal summarization in our experiments, a video-based news article is represented by a pair (v, x). v corresponds to the video input -a sequence of frames: @xmath0, v 2 , . . . , v n ). x is the news article presented as a sequence of tokens: @xmath1, x 2 , . . . , x m ). we assume that for each article, there is a groundtruth textual summary @xmath2, y 2 , . . . , y l ) and a ground-truth cover picture p . the task is to generate a textual summary y that includes the main points of the article and to choose a frame v to act as a cover picture (pictorial summary). ## experiments we analyze several aspects of the proposed model: first, we study the effect of the visual features. second, we analyze the contribution of pre-training the model on text-only summarization data. third, we exploit the smooth frame labels to further improve the model. the results are presented in ## human evaluation previous works on vmsmo evaluated the system performance by employing human judges to assess the quality of generated textual summary: @xcite measured to what extent the system summaries were sufficient to answer questions generated from the reference summary and ranked them based on informativeness, coherence, and succinctness; @xcite scored the system summaries based on informativeness and satisfaction. we believe no prior work employed human annotators to judge the quality of a chosen cover frame (pictorial summary) in the context of textual summary. for the similar task of multimodal summarization with unimodal output, @xcite collected annotations for the subset of the wikihow dataset @xcite that measured whether the textual output was faithful to the source pair of document and image. ## conclusions in this paper, we explored the recently proposed task of video-based multimodal summarization with multimodal output. we extended the available resources to a new language by introducing a multimodal summarization dataset in czech. we explored the pre-training strategies, showing that transferring knowledge from the simpler, unimodal task of text-to-text summarization helps with the final performance in multimodal settings. we were also able to show that the usage of innervideo similarities, via the introduction of smooth labels during training, helps to stabilize the training. we conducted a human evaluation of the frameselection process to confirm the quality of the proposed multimodal mms model. our findings indicate that the mms model pre-trained on the textto-text summarization is not effective in utilizing video features and that future works should carefully examine to what extent the model is able to make use of multimodal input and whether the improvement is orthogonal to e.g., using more data. limitations mlask dataset collection. while curating the mlask dataset, we applied a series of rule-based filters (section 3.1) and collected only those documents that followed a strict html structure. no large-scale human evaluation was applied to check the data validity. we sampled a random subset of 100 articles and checked the data preparation and collection manually. language and domain bias. we acknowledge that our findings are based on a single dataset, in a particular language (czech) and from a particular domain (news articles). due to the novelty of the task, previous datasets proposed for vmsmo (