Towards Unifying Multi-Lingual and Cross-Lingual Summarization (2023)

Abstract: To adapt text summarization to the multilingual world, previous work proposes multi-lingual summarization (MLS) and cross-lingual summarization (CLS). However, these two tasks have been studied separately due to the different definitions, which limits the compatible and systematic research on both of them. In this paper, we aim to unify MLS and CLS into a more general setting, i.e., many-to-many summarization (M2MS), where a single model could process documents in any language and generate their summaries also in any language. As the first step towards M2MS, we conduct preliminary studies to show that M2MS can better transfer task knowledge across different languages than MLS and CLS. Furthermore, we propose Pisces, a pre-trained M2MS model that learns language modeling, cross-lingual ability and summarization ability via three-stage pre-training. Experimental results indicate that our Pisces significantly outperforms the state-of-the-art baselines, especially in the zero-shot directions, where there is no training data from the source-language documents to the target-language summaries.

## introduction the world we live in is multi-lingual. with globalization, text resources in various languages flood the internet, where global users can easily access their desired information. under this background, the text summarization community presents multilingual summarization (mls) and cross-lingual summarization (cls), respectively. as shown in figure 1 , mls aims at building a unified model to process documents in multiple languages and generate summaries in the corresponding language @xcite @xcite @xcite ## related work multi-lingual summarization. multi-lingual summarization (mls) aims to process documents in multiple languages and generate their summaries in the corresponding language. @xcite present multiling-2015 dataset. later, this task receives increasing attention @xcite . recently, largescale mls datasets @xcite @xcite @xcite together with sophisticated methods @xcite @xcite early work typically focuses on pipeline methods @xcite @xcite @xcite , i.e., translation and then summarization or summarization and then translation. recently, with the availability of large-scale cls datasets @xcite systematically investigate the capabilities of models trained with mls, cls and m2ms settings. pre-trained models for summarization. pretrained models have shown their superiority in summarization task, e.g., bart @xcite and t5 @xcite . to enhance the summarization ability during the pre-training stage, pe-gasus @xcite introduces the gap sentence generation (gsg) objective to enable the model to generate key sentences in an article from the remaining ones. further, primera @xcite extends gsg from single-document to multidocument summarization. in dialogue scenarios, @xcite present mdialbart for crosslingual dialogue summarization. among these pre-trained summarization models, pegasus and primera only focus on monolingual summarization. though mdialbart aims at cls, the model is merely built for a single crosslingual direction (i.e., english ⇒ german/chinese) and a specific scenario (i.e., dialogue). our pisces is the first multi-lingual pre-trained model for general summarization. ## does unifying all directions in a single model help each other? as discussed previously, m2ms unifies all summarization directions in a single model. therefore, we wonder can such a setting help the model better transfer task knowledge across different languages compared with the settings of mls and cls? to answer the question, we conduct preliminary studies to investigate the influence of different settings. ## pisces in this section, we propose pisces, a pre-trained multi-lingual model for m2ms with the backbone of transformer @xcite . figure 2 shows the overview of pisces, which contains three pre-training stages. specifically, the meta pre-training ( § 4.1) lets the pre-trained model learn general language modeling via monolingual denoising objective in multiple languages. then, to improve the transferability across different languages, the cross-lingual pre-training ( § 4.2) adds noises to the source-language sentences, and encourages the model to translate them into parallel sentences in the target language. note that the parallel sentences used in this stage might involve the languages which are not seen in downstream tasks, and it is the key to improving the transferability to these languages. finally, to narrow the gap between the pre-training and fine-tuning stages, the task-specific pre-training ( § 4.3) trains the model with pseudo m2ms samples, which are constructed from the multi-lingual unlabeled corpora via gap sentences selection and machine translation. during the three-stage pre-training process, the model gradually learns the ability of language modeling, then the cross-lingual ability, and finally the adaptation to the specific task. ## conclusion in this paper, we unify mls and cls to m2ms. through carefully-designed preliminary studies, we discuss that unifying mls and cls to m2ms is valuable. in addition, we propose pisces, the first pre-trained m2ms model, which contains three pretraining stages to enable the model learn the multilingual language modeling, cross-lingual ability and summarization ability. extensive experiments show its superiority compared with the state-ofthe-art baselines (mbart-50 and mt5). the case study further demonstrates that our model could even generate summaries for the documents whose language does not occur in the fine-tuning stage.