MCLS : A Large-Scale Multimodal Cross-Lingual Summarization Dataset (2023)

Abstract: “Multimodal summarization which aims to generate summaries with multimodal inputs, e.g., textand visual features, has attracted much attention in the research community. However, previousstudies only focus on monolingual multimodal summarization and neglect the non-native readerto understand the cross-lingual news in practical applications. It inspires us to present a newtask, named Multimodal Cross-Lingual Summarization for news (MCLS), which generates cross-lingual summaries from multi-source information. To this end, we present a large-scale multimodalcross-lingual summarization dataset, which consists of 1.1 million article-summary pairs with 3.4million images in 44 * 43 language pairs. To generate a summary in any language, we propose aunified framework that jointly trains the multimodal monolingual and cross-lingual summarizationtasks, where a bi-directional knowledge distillation approach is designed to transfer knowledgebetween both tasks. Extensive experiments on many-to-many settings show the effectiveness ofthe proposed model.”

## introduction the goal of multimodal summarization is to produce a summary with the help of multi-source inputs, e.g., text and visual features. with the rapid growth of multimedia content on the internet, this task has received increasing attention from the research communities and has shown its potential in recent years. it benefits users from better understanding and accessing verbose and obscure news, and thus can help people quickly master the core ideas of a multimodal article. in the literature, many efforts have been devoted to the multimodal summarization fields, e.g., sportssum @xcite , moviesum @xcite , msmr @xcite , mmss @xcite , mss @xcite , how2 @xcite , msmo @xcite , e-dailymail @xcite , ec-product @xcite , mm-avs @xcite , and mm-sum @xcite . all these datasets cover video summarization, movie summarization, meeting records summarization, sentence summarization, product summarization, and news summarization. with the predefined task, former state-of-the-art multimodal summarization models have achieved great outcomes. for instance, @xcite and @xcite explore the hierarchy between the textual article and visual features, and integrate them into the mas model. @xcite design a multistage fusion network to model the fine-grained interactions between the two modalities. and @xcite study multiple multimodal fusion methods to infuse the visual features into generative pre-trained language models, e.g., bart @xcite . despite their efforts and effectiveness, existing methods are all conducted in monolingual scenarios. in practical applications, for non-native news viewers, they desire some native language summaries to better understand the contents of the news in other languages. to our knowledge, little research work has been devoted to multimodal cross-lingual summarization. one important season is the lack of a large-scale multimodal cross-lingual benchmark. to assist those non-native readers, we propose a new task: multimodal cross-lingual summarization for news (mcls). as shown in figure 1 , the inputs consist of two parts: the image sequence and textual article in the source language (e.g., english), and the summary outputs can be in any target language (e.g.,