Mixed-Lingual Pre-training for Cross-lingual Summarization (2020)

Abstract: Cross-lingual Summarization (CLS) aims at producing a summary in the target language for an article in the source language. Traditional solutions employ a two-step approach, i.e. translate -> summarize or summarize -> translate. Recently, end-to-end models have achieved better results, but these approaches are mostly limited by their dependence on large-scale labeled data. We propose a solution based on mixed-lingual pre-training that leverages both cross-lingual tasks such as translation and monolingual tasks like masked language models. Thus, our model can leverage the massive monolingual data to enhance its modeling of language. Moreover, the architecture has no task-specific components, which saves memory and increases optimization efficiency. We show in experiments that this pre-training scheme can effectively boost the performance of cross-lingual summarization. In NCLS dataset, our model achieves an improvement of 2.82 (English to Chinese) and 1.15 (Chinese to English) ROUGE-1 scores over state-of-the-art results.

## introduction text summarization can facilitate the propagation of information by providing an abridged version for long articles and documents. meanwhile, the globalization progress has prompted a high demand of information dissemination across language barriers. thus, the cross-lingual summarization (cls) task emerges to provide accurate gist of articles in a foreign language. traditionally, most cls methods follow the twostep pipeline approach: either translate the article into the target language and then summarize it @xcite , or summarize the article in the source language and then translate it (wan * equal contribution @xcite . although this method can leverage off-the-shelf summarization and mt models, it suffers from error accumulation from two independent subtasks. therefore, several end-to-end approaches have been proposed recently @xcite @xcite , which conduct both translation and summarization simultaneously. easy to optimize as these methods are, they typically require a large amount of cross-lingual summarization data, which may not be available especially for low-resource languages. for instance, ncls @xcite proposes to co-train on monolingual summarization (ms) and machine translation (mt) tasks, both of which require tremendous labeling efforts. on the other hand, the pre-training strategy has proved to be very effective for language understanding @xcite and cross-lingual learning @xcite . one of the advantages of pre-training is that many associated tasks are selflearning by nature, which means no labeled data is required. this greatly increases the amount of training data exposed to the model, thereby enhancing its performance on downstream tasks. therefore, we leverage large-scale pre-training to improve the quality of cross-lingual summarization. built upon a transformer-based encoderdecoder architecture @xcite , our model is pre-trained on both monolingual tasks including masked language model (mlm), denoising autoencoder (dae) and monolingual summarization (ms), and cross-lingual tasks such as crosslingual masked language model (cmlm) and machine translation (mt). this mixed-lingual pretraining scheme can take advantage of massive unlabeled monolingual data to improve the model's language modeling capability, and leverage crosslingual tasks to improve the model's cross-lingual representation. we then finetune the model on the downstream cross-lingual summarization task. furthermore, based on a shared multi-lingual vocabulary, our model has a shared encoder-decoder architecture for all pre-training and finetuning tasks, whereas ncls @xcite sets aside taskspecific decoders for machine translation, monolingual summarization, and cross-lingual summarization. in the experiments, our model outperforms various baseline systems on the benchmark dataset ncls @xcite . for example, our model achieves 3.27 higher rouge-1 score in chinese to english summarization than the state-of-the-art result and 1.28 higher rouge-1 score in english to chinese summarization. we further conduct an ablation study to show that each pretraining task contributes to the performance, especially our proposed unsupervised pretraining tasks. ## conclusion we present a mix-lingual pre-training model for cross-lingual summarization. we optimize a shared encoder-decoder architecture for multi-lingual and multi-task objectives. experiments on a benchmark dataset show that our model outperforms pipelinebased and other end-to-end baselines. through an ablation study, we show that all pretraining objectives contribute to the model's performance.