C ross S um: Beyond E nglish-Centric Cross-Lingual Summarization for 1,500+ Language Pairs (2023)

Abstract: We present CrossSum, a large-scale cross-lingual summarization dataset comprising 1.68 million article-summary samples in 1,500+ language pairs. We create CrossSum by aligning parallel articles written in different languages via cross-lingual retrieval from a multilingual abstractive summarization dataset and perform a controlled human evaluation to validate its quality. We propose a multistage data sampling algorithm to effectively train a cross-lingual summarization model capable of summarizing an article in any target language. We also introduce LaSE, an embedding-based metric for automatically evaluating model-generated summaries. LaSE is strongly correlated with ROUGE and, unlike ROUGE, can be reliably measured even in the absence of references in the target language. Performance on ROUGE and LaSE indicate that our proposed model consistently outperforms baseline models. To the best of our knowledge, CrossSum is the largest cross-lingual summarization dataset and the first ever that is not centered around English. We are releasing the dataset, training and evaluation scripts, and models to spur future research on cross-lingual summarization. The resources can be found at https://github.com/csebuetnlp/CrossSum

## introduction cross-lingual summarization (hereinafter xls) is the task of generating a summary in a target language given a source text in another language. the task is challenging as it combines summarization and translation in one task, both challenging tasks in their own right. earlier approaches to xls thus employed pipeline methods such as translate-thensummarize @xcite and summarizethen-translate @xcite . not only are they computationally expensive, having to use multiple input article: [...] 新型コロナウイルスに対し、 様々な既存の 治療法の効果を試す世界的規模の臨床試験の一貫として、 デキ サメタゾンが試された。(dexamethasone was tested as part of a global clinical trial to test the effectiveness of various existing therapies against the new coronavirus.) [...] その結果、 人 工呼吸器を必要とする重症患者の致死率が3割下がり。(as a result, the case fatality rate of critically ill patients who require a ventilator is reduced by 30%.) [...] ボリス･ジョンソン 英首相は 「イギリス科学界の素晴らしい成果」 を歓迎し。(british prime minister boris johnson welcomed "the great achievements of the british scientific community".) [...]「しかもこれ は、 世界中で手に入る薬だ」("and this is a medicine available all over the world".) [...] きわめて安いステロイド剤だった (but a very cheap steroid that has been used for a long time.) summary: িবজ্ঞানীরা বলেছন েড�ােমথােসান নােম স�া ও সহজলভয্ একিট ওষু ধ কেরানাভাইরােস গুরুতর অসু � েরাগীেদর জীবন রক্ষা করেত সাহাযয্ করেব। (scientists say a cheap and readily available drug called dexamethasone will help save the lives of critically ill patients with coronavirus.) figure 1 : a sample article-summary pair from cross-sum, the article is written in japanese, and the summary is in bengali. we translate the texts to english inside parentheses for better understanding. words and phrases of the article relevant to the summary are color-coded. the success of sequence-to-sequence (seq2seq) models @xcite and the advances in transformer-based models @xcite have aided in the emergence of end-to-end methods that can perform xls with one single model @xcite . the availability of xls datasets @xcite has also helped this task gain popularity in recent times. however, they cover only a few languages, contain a small number of samples for training and evaluation, or use english as the pivot language (i.e., the target language always remains english), thereby limiting their applicability to a great extent. to democratize xls beyond high-resource languages, in this work, we introduce crosssum, a large-scale xls dataset containing 1.68 million article-summary samples in 1,500+ language pairs. we align parallel articles 1 written in different languages via cross-lingual retrieval from the multilingual xl-sum @xcite dataset. we introduce and rigorously study the notions 'induced pairs' and 'implicit leakage' to increase the coverage of the dataset while at the same time ensuring maximum quality. we also perform a controlled human evaluation of crosssum spanning nine languages from high-to low-resource and show that the alignments are highly accurate. we design mls, a multistage language sampling algorithm, for successfully training models that can generate a summary in any target language for an input article in any source language, both from a set of languages present in the training dataset. for the first time, we perform xls with crosssum on a broad and diverse set of languages without relying on english as the standalone pivot, consistently outperforming many-to-one and one-to-many models, as well as summarize-then-translate baselines. we propose lase, an embedding-based metric for evaluating summaries when reference summaries may not be available in the target language but may be available in another language, potentially opening new doors for evaluating lowresource languages. furthermore, we demonstrate the reliability of lase by its high correlation with rouge @xcite , the de-facto metric for evaluating text summarization systems. to the best of our knowledge, crosssum is the largest publicly available abdtractive xls dataset, both in terms of the number of samples and the number of language pairs. we are releasing the dataset, training and evaluation scripts, and models hoping that these resources will encourage the community to push the boundaries of xls beyond english and other high-resource languages. ## the crosssum dataset the most straightforward way of curating a highquality xls dataset is via crowd-sourcing @xcite . however, it may be difficult to find crowd workers having professional command over low-resource languages or distant language pairs. moreover, scalability issues might arise due to the time and budget constraints for 1 we re-purpose the terminology of parallel corpus here. crowd-sourcing. therefore, synthetic @xcite and automatic methods @xcite have gained traction over crowd-sourcing. automatic curation of an xls dataset is simply to pair an article a in a source language with the summary of a parallel article b written in a different target language (figure 1 ), assuming the availability of a multilingual dataset having identical contents in different languages. two contemporary works have compiled large-scale multilingual summarization datasets, namely xl-sum @xcite (1.35m samples in 45 languages) and massivesumm @xcite (28.8m samples in 92 languages). though substantially larger than the other, massivesumm is not publicly available. since public availability is crucial for promoting open research, we opted for xl-sum, distributed under a non-commercial license. additionally, all articles of xl-sum are crawled from a single source, bbc news. we observed that bbc publishes similar news content in different languages and follow similar summarization strategies. hence adopting xl-sum would increase the quality and quantity of the article-summary pairs. unlike previous automatic methods, there are no explicit links between parallel articles in xl-sum. fortunately, language-agnostic sentence representations @xcite have achieved state-of-the-art results in crosslingual text mining @xcite , and hence, we use them to search identical contents across languages. for simplicity foot_0 , we perform the search over summaries only. to ensure maximum quality, we set two conditions for a summary s a in language a to be aligned with another summary s b in language b: figure 2 : training on the dataset respecting the original xl-sum splits causes unusually high rouge scores (marked red) in many-to-one models due to implicit data leakage. therefore, we redid the splits taking the issue into account, and consequently, models trained on the new set (marked blue) do not exhibit any unusual spike. ## human evaluation of crosssum to establish the validity of our automatic alignment pipeline, we conducted a human evaluation to study the quality of the cross-lingual alignments. we selected all possible combinations of language pairs from a list of nine languages ranging from high-resource to low-resource to assess the alignment quality in different pair configurations (e.g., high-high, low-high, low-low) as per the language diversity categorization by @xcite . we chose three high-resource languages, english, arabic, and (simplified) chinese (categories 4 and 5); three mid-resource languages, indonesian, bengali, and urdu (category 3); and three low-resource languages, punjabi, swahili, and pashto (categories 1 and 2), as representative languages and randomly sampled fifty cross-lingual summary alignments from each language pair for annotation. as a direct evaluation of these pairs would require bilinguallyproficient annotators for both languages, which are practically intractable for distantly related languages (e.g., bengali-swahili), we resorted to a pivoting approach during annotation for language pairs that do not contain english. for a language pair (l 1l 2 ), where l 1 ̸ = en and l 2 ̸ = en, we sampled alignments (x, y) such that ∃(x, e) ∈ (l 1 -en) and ∃(y, e) ∈ (l 2en), for an english article e. in other words, we ensure that both the articles of the sampled cross-lingual pair have a corresponding cross-lingual pair with an english article. an alignment (x, y) would be deemed correct if both (x, e) and (y, e) are correct. this formulation thus reduced the original problem to annotating samples from language pairs (l 1en) and (l 2en), where l 1 and l 2 are from the previously selected languages that are not english. we hired bilingually proficient expert annotators adept in the language of interest and english. two annotators labeled each language pair where one language is english. we presented them with corresponding summaries of the cross-lingual pairs (and optionally the articles themselves) and elicited yes/no answers to the question: "can the provided sequences be considered summaries for the same article?" 5 we deem a sequence pair accurate if both annotators judge it as valid. we show the alignment accuracies of the language pairs in figure 3 . as evident from the figure, the annotators judge the aligned summaries to be highly accurate, with an average accuracy of 95.67%. we used cohen's kappa @xcite to establish the interannotator agreement and show the corresponding statistics in ## training & evaluation methodologies in this section, we discuss the multistage sampling strategy for training cross-lingual text generation models and our proposed metric for evaluating model-generated summaries. ## experiments & discussions one model capable of generating summaries in any target language for an input article from any source language is highly desirable. however, it may not be the case that such a 'many-to-many' model (m2m in brief) would outperform many-toone (m2o) or one-to-many (o2m) models 7 , which are widely-used practices for xls @xcite . in this section, we establish that the m2m model, trained in the presence of samples from all possible language pairs using the mls algorithm from section 4, consistently outperforms m2o, o2m, and summarize-then-translate (s.+t.) baselines given equal training steps. in addition to the proposed m2m model, we train five different m2o and o2m models using five highly spoken 8 and typologically diverse pivot 7 discussed in detail in appendix c. 8 https://w.wiki/pss (i.e., the 'one' in m2o and o2m) languages: english, chinese (simplified), hindi, arabic, and russian. as another baseline, we use a summarizethen-translate pipeline. as fine-tuning pretrained language models @xcite have shown state-of-the-art results on monolingual and multilingual text summarization @xcite , we fine-tune each model using a pretrained mt5 @xcite ) by providing explicit cross-lingual supervision. we show the results on rouge-2 f1 and lase in figures 4 and 5 9 . we limit our evaluation only to the languages supported by mt5, fasttext, and m2m-100 (the translation model used in s.+t.). results indicate that the m2m model consistently outperforms m2o, o2m, and s.+t., with an average rouge-2 (lase) score of 8.15 (57.15) over all languages tested, 3.12 (9.02) above s.+t. moreover, compared to the o2m models on language pairs where the pivots are the targets, the m2m model scores 1.80 (5.84) over m2os, and on those where the pivots are the sources, 6.52 (51.80) over o2ms. upon inspection of the model outputs, we found the m2o models to be able to generate non-trivial summaries. in contrast, the o2m models completely failed to produce cross-lingual summaries, performing in-language summarization (the language of the summary is the same as that of its input article) for all targets. we hypothesize that varying the target language in a batch hampers the decoder's ability to generate from a specific language, possibly because of the vast diversity of target languages in the batch (discussed further in appendix e). s.+t. performed well on high-resource languages but poorly on lowresource ones. this was revealed to be a limitation of the translation model used in the pipeline. ## analysis of results statistical significance while the scores obtained from the experiments in section 5 indicate that the proposed m2m model performs better than the others, the differences are very close in many language pairs. therefore, a statistical significance test is still warranted to support our claim further. as such, for each language pair experimented on, we performed the bootstrap resampling test @xcite with the m2m model against the best-performing model among the others in a one vs. all manner: if m2m has the best (rouge-2/lase) score, we compare it with the model with e n g li s h s p a n is h j a p a n e s e f r e n c h a r a b ic c h in e s e t u r k is h k o r e a n h in d i p e r s ia n p o r t u g u e s e r u s s ia n v ie t n a m e s e s e r b ia n u k r a in ia n i n d o n e s ia n t h a i t a m il u z b e k b e n g a li u r d u p u n ja b i m a r a t h i s w a h il i a m h a r ic y o r u b a s o m a li g a e li c w e ls h a z e r b a ij a n i b u r m e s e p a s h t o g u ja r a t i n e p a li s in h a la s. + t. (en-x) o2m (en-x) m2m (en-x) results (@xmath0.05) in how reliable is lase? at first, we validated the reliability of lase by showing its correlation with rouge-2. we took different checkpoints of the in-language summarization model used in s.+t. and computed rouge-2 and lase for the nine languages in section 3 for each checkpoint. the correlation coefficients of the calculated scores are shown in the second column of however, the purpose of lase is to show that it is language-agnostic and can even be computed in the absence of references in the target language. therefore, we evaluate the summaries with references in a different language from the target using the m2m model. for each target language, we first compute the standard lase for different source languages (denoted as lase-in-lang). we again compute lase after swapping the reference texts with the references in the language of the input text 11 (denoted as lase-out-lang). we then show the correlation between the two variants of lase in the third column of from these two experiments, we can conclude that lase is an ideal metric for the evaluation of summarization systems and can be computed in a language-independent manner. ## related works pipeline-based methods were popular at the beginning stages of xls research @xcite @xcite , breaking the task into a sequence of summarization and translation tasks. end-to-end methods that performed xls with a single model gained popularity with the emergence of neural models. @xcite used knowledge distillation (hinton et al., 11 our curation method ensures that such summaries always exist in the corresponding test sets. 12 since many test sets of the language pairs from section 3 have too few samples for reliable evaluation (e.g., punjabi-pashto), for each target language, we use only the top-5 source languages by the number of their test set samples. 2015) to train a student xls model from two summarization and translation teacher models. using a synthetic dataset, @xcite al. (2019 2021 ) introduced multiple pretraining objectives specifically tailored to cross-lingual tasks that showed improved results on xls. we refer our readers to @xcite for a more comprehensive literature review. until recently, xls was limited primarily to english-chinese due to the lack of benchmark datasets. to promote the task beyond this language pair, @xcite introduced wikilingua, a large-scale many-to-one dataset with english as the pivot language, while perez-beltrachini and lapata (2021) introduced xwikis, containing 4 languages in 12 directions. more recently, @xcite explored zeroshot cross-lingual summarization by prompting @xcite large language models like chat-gpt foot_7 , gpt-4 (openai, 2023), and bloomz @xcite . ## conclusion & future works in this work, we presented crosssum, a largescale, non-english-centric xls dataset containing 1.68 million samples in 1,500+ language pairs. crosssum provides the first publicly available xls dataset for many of these pairs. performing a limited-scale human evaluation of crosssum, we introduced mls, a multistage sampling algorithm for general-purpose cross-lingual generation, and lase, a language-agnostic metric for evaluating summaries when reference summaries in the target languages may not be available. we demonstrated that training one multilingual model can help towards better xls than baselines. we also shed light on the potential to perform zero-shot and few-shot xls with crosssum. we share our findings and resources in the hopes of making the xls research community more inclusive and diverse. in the future, we will investigate the use of cross-sum for other summarization tasks, e.g., multidocument @xcite and multi-modal summarization @xcite . we would also like to explore better techniques for m2m, zeroshot, and few-shot cross-lingual summarization.