Large Scale Multi-Lingual Multi-Modal Summarization Dataset (2023)

Abstract: Significant developments in techniques such as encoder-decoder models have enabled us to represent information comprising multiple modalities. This information can further enhance many downstream tasks in the field of information retrieval and natural language processing; however, improvements in multi-modal techniques and their performance evaluation require large-scale multi-modal data which offers sufficient diversity. Multi-lingual modeling for a variety of tasks like multi-modal summarization, text generation, and translation leverages information derived from high-quality multi-lingual annotated data. In this work, we present the current largest multi-lingual multi-modal summarization dataset (M3LS), and it consists of over a million instances of document-image pairs along with a professionally annotated multi-modal summary for each pair. It is derived from news articles published by British Broadcasting Corporation(BBC) over a decade and spans 20 languages, targeting diversity across five language roots, it is also the largest summarization dataset for 13 languages and consists of cross-lingual summarization data for 2 languages. We formally define the multi-lingual multi-modal summarization task utilizing our dataset and report baseline scores from various state-of-the-art summarization techniques in a multi-lingual setting. We also compare it with many similar datasets to analyze the uniqueness and difficulty of M3LS. The dataset and code used in this work are made available at “ https://github.com/anubhav-jangra/M3LS ”.

## introduction the world we live in today is very diverse, with over 7,000+ languages spoken across the globe 2 . these languages have varying traits and are spoken by communities of various sizes depending upon ## related work the field of text summarization is more than 5 decades old @xcite , and has evolved to a great extent in recent years. prior to the advances in sequence-to-sequence frameworks @xcite , people mainly focused on extractive summarization techniques that aim to generate summary via extracting words, phrases, or sentences @xcite @xcite . @xcite proposed the pointer-generator networks, an attentive recurrent neural network based framework @xcite . recent years have seen great progress in research in automatic summarization leveraging transformer based models @xcite and attention mechanism @xcite . in this section we discuss the related works showcasing multi-modal datasets and multi-lingual datasets. a detailed size comparison of these datasets with m3ls is shown in ## m3ls task given for each language l @xmath0, we have data m l @xmath1, i l k >, where when k 1 ̸ = k 2 , we have multi-modal crosslingual summarization, otherwise the task is multimodal mono-lingual summarization, a graphic representation of the task is shown in figure 1 . developments in summarization techniques such as secondary enhancements using images with multimodal output @xcite , video-based multimodal summarization @xcite and using multiobjective optimization @xcite . on the other hand, development of multi-lingual transformer based models like xue et al. ( 2020 ) has publicly available checkpoints fine-tuned for multiple language modelling tasks, including multilingual summarization. development of such models requires highquality heterogeneous data and improvements in various models utilizing multi-modal shared attention layers for annotated data with image-text pairs for a specific language task. to address these issues, we present m3ls and in this section we discuss various steps involved in its construction. ## experiments 6.1 setup depending upon the number of instances in each language within m3ls, we perform a train:test:validation split with a ratio of 80:10:10 if the number of instances is below 50k and 90:5:5 otherwise. to conduct our experiments in a multilingual setting, we survey publicly available tokenizers and sentence segmenters for multiple languages, and we combine them within one dedicated package for our experiments. we further define a set of rules for sentence segmentation for languages lacking such support from external packages within our package 19 . we compile our package using segtok foot_14 for the indo-european language, indicnlp foot_15 for indian languages, fugashi (mccann, 2020) for japanese (ja) and chinese foot_16 for chinese. for data pre-processing steps such as stopword removal, we collect stopwords from the nltk foot_17 package, and publicly available stopwords present in the spacy foot_18 repository for all languages in a centralized pipeline for our experiments. we evaluated the performance of various summarization techniques utilizing our dataset, including simpler techniques such as lead-3 and random which have proven to be quite useful in past @xcite @xcite . we have also included statistics based centroid @xcite and graph based textrank @xcite techniques. to have a fair comparison across multiple languages using a shared dedicated model, we have evaluated the performance of an abstractive technique in a multi-lingual setting utilizing a pre-trained checkpoint 25 for summarization of the transformer-based mt5 @xcite model. finally to explore the multi-modal aspect of our dataset, we evaluate the performance of a multimodal encoder-decoder based technique @xcite that utilizes images and text to generate a multi-modal text summary. however the publicly available implementation 26 for msmo restricts us to evaluate it only for the english language. however, to compare this score, we evaluate the performance of three state-of-the-art transformer-based models -pegasus @xcite , bart @xcite , and t5 @xcite for summarization which are compatible with the english language. since, two of the pre-trained models we described above are fine-tuned on xsum and xl-sum datasets which are extracted from the same source -bbc news -we avoid fine-tuning on models to have a fair comparison of the models and we explain the scores in discussions. in all techniques, we set the generated summary length threshold as the average length of gold summary for the corresponding language in our corpus. ## results and discussion we evaluate the generated summaries against the gold summaries using the rouge @xcite evaluation metric. we report the rouge-1, rouge-2, rouge-l f-scores across every baseline discussed above @xcite (refer to tables 1 and 3 ). we additionally report bertscore for english baselines @xcite (refer to ## conclusion in this work, we release a large-scale multi-modal multi-lingual summarization dataset comprising of over 1.1m+ news articles and spanning 20 languages, and motivate the problem statement of multi-modal multi-lingual summarization using m3ls. to the best of our knowledge, this is the first ever multi-modal summarization data set spanning several languages. the proposed dataset is the largest summarization dataset for 13 out of 20 languages. we have evaluated the performance of various baselines to establish the quality of the proposed dataset in both multi-modal and multilingual settings. through this work, we hope to instigate research in various less-explored languages in the community for various research problems including but not limited to summarization, headline generation, keyword extraction, image caption generation, multi-modal embedding generation, etc. in future works, we plan to work on shared models which address the m3ls task utilizing our dataset.