MLSUM : The Multilingual Summarization Corpus (2020)

Abstract: We present MLSUM, the first large-scale MultiLingual SUMmarization dataset. Obtained from online newspapers, it contains 1.5M+ article/summary pairs in five different languages – namely, French, German, Spanish, Russian, Turkish. Together with English news articles from the popular CNN/Daily mail dataset, the collected data form a large scale multilingual dataset which can enable new research directions for the text summarization community. We report cross-lingual comparative analyses based on state-of-the-art systems. These highlight existing biases which motivate the use of a multi-lingual dataset.

## introduction the document summarization task requires several complex language abilities: understanding a long document, discriminating what is relevant, and writing a short synthesis. over the last few years, advances in deep learning applied to nlp have contributed to the rising popularity of this task among the research community @xcite @xcite . as with other nlp tasks, the great majority of available datasets for summarization are in english, and thus most research efforts focus on the english language. the lack of multilingual data is partially countered by the application of transfer learning techniques enabled by the availability of pre-trained multilingual language models. this approach has recently established itself as the de-facto paradigm in nlp @xcite . under this paradigm, for encoder/decoder tasks, a language model can first be pre-trained on a large corpus of texts in multiple languages. then, the model is fine-tuned in one or more pivot languages for which the task-specific data are available. at inference, it can still be applied to the different languages seen during the pre-training. because of the dominance of english for large scale corpora, english naturally established itself as a pivot for other languages. the availability of multilingual pre-trained models, such as bert multilingual (m-bert), allows to build models for target languages different from training data. however, previous works reported a significant performance gap between english and the target language, e.g. for classification @xcite and question answering @xcite tasks. a similar approach has been recently proposed for summarization @xcite obtaining, again, a lower performance than for english. for specific nlp tasks, recent research efforts have produced evaluation datasets in several target languages, allowing to evaluate the progress of the field in zero-shot scenarios. nonetheless, those approaches are still bound to using training data in a pivot language for which a large amount of annotated data is available, usually english. this prevents investigating, for instance, whether a given model is as fitted for a specific language as for any other. answers to such research questions represent valuable information to improve model performance for low-resource languages. in this work, we aim to fill this gap for the automatic summarization task by proposing a largescale multilingual summarization (mlsum) dataset. the dataset is built from online news outlets, and contains over 1.5m article-summary pairs in 5 languages: french, german, spanish, russian, and turkish, which complement an already established summarization dataset in english. the contributions of this paper can be summarized as follows: ## mlsum as described above, the vast majority of summarization datasets are in english. for arabic, there exist the essex arabic summaries corpus (easc) @xcite and kalimat (el-haj and koulali, 2013); those comprise circa 1k and 20k samples, respectively. pontes et al. (2018) proposed a corpus of few hundred samples for spanish, portuguese and french summaries. to our knowledge, the only large-scale non-english summarization dataset is the chinese lcsts @xcite . with the increasing interest for cross-lingual models, the nlp community have recently released multilingual evaluation datasets, targeting classification (xnli) and qa @xcite tasks, as described in 2.2, though still no large-scale dataset is avaulable for document summarization. to fill this gap we introduce mlsum, the first large scale multilingual summarization corpus. our corpus provides more than 1.5 millions articles in french (fr), german (de), spanish (es), turkish (tr), and russian (ru). being similarly built from news articles, and providing a similar amount of training samples per language (except for russian), as the previously mentioned cnn/daily mail, it can effectively serve as a multilingual extension of the cnn/daily mail dataset. ## models we experimented on mlsum with the established models and baselines described below. those include supervised and unsupervised methods, extractive and abstractive models. for all the experiments, we train models on a per-language basis. we used the recommended hyperparameters for all languages, in order to facilitate assessing the robustness of the models. we also tried to train one model with all the languages mixed together, but we did not see any significant difference of performance. ## results and discussion the results we now present allow to compare the models across languages, and to investigate or hypothesize where their performance variations may come from. to explain those, we note the following factors: (1) differences in the data, independently from the language, such as the structure of the article, the abstractiveness of the summaries, or the quantity; and turning to the observed results, we report in russian, the low-resource language in mlsum for all experimental setups, the performance on russian is comparatively low. this can be explained by at least two factors. first, the corpus is the most abstractive (see how abstractive are the models? we report novelty in figure 1 . as previous works noted @xcite , pointer-generator networks are poorly abstractive, as they rely heavily on the copy mechanism. this is particularly true for russian: the lack of data probably makes it easier to learn to copy than to cope with natural language generation. as expected, pretrained language models such as m-bert are consistently and largely more abstractive, since they are exposed to other texts during pretraining. ## conclusion we presented mlsum, the first large-scale multi-lingual summarization dataset, comprising over 1.5m article/summary pairs in french, german, russian, spanish, and turkish. we detailed its construction, and its complementary nature to the cnn/dm english summarization dataset. we reported extensive preliminary experiments, highlighting biases observed in existing models, and analyzed the relative performances across languages of state-of-the-art models. in future works we plan to add other languages including arabic and hindi, and to investigate the adaptation of neural metrics to multilingual summarization.