Jointly Learning to Align and Summarize for Neural Cross-Lingual Summarization (2020)

Abstract: Cross-lingual summarization is the task of generating a summary in one language given a text in a different language. Previous works on cross-lingual summarization mainly focus on using pipeline methods or training an end-to-end model using the translated parallel data. However, it is a big challenge for the model to directly learn cross-lingual summarization as it requires learning to understand different languages and learning how to summarize at the same time. In this paper, we propose to ease the cross-lingual summarization training by jointly learning to align and summarize. We design relevant loss functions to train this framework and propose several methods to enhance the isomorphism and cross-lingual transfer between languages. Experimental results show that our model can outperform competitive models in most cases. In addition, we show that our model even has the ability to generate cross-lingual summaries without access to any cross-lingual corpus.

## introduction neural abstractive summarization has witnessed rapid growth in recent years. variants of sequenceto-sequence models have shown to obtain promising results on english @xcite or chinese summarization datasets. however, cross-lingual summarization, which aims at generating a summary in one language from input text in a different language, has been rarely studied because of the lack of parallel corpora. early researches on cross-lingual abstractive summarization are mainly based on the summarization-translation or translationsummarization pipeline paradigm and adopt different strategies to incorporate bilingual features @xcite @xcite into the pipeline model. recently, @xcite first propose a neural cross-lingual summarization system based on a large-scale corpus. they first translate the texts automatically from the source language into the target language and then use the teacher-student framework to train a cross-lingual summarization model. @xcite further improve this teacher-student framework by using genuine summaries paired with the translated pseudo source sentences to train the cross-lingual summarization model. @xcite propose a multi-task learning framework to train a neural cross-lingual summarization model. cross-lingual summarization is a challenging task as it requires learning to understand different languages and learning how to summarize at the same time. it would be difficult for the model to directly learn cross-lingual summarization. in this paper, we explore this question: can we ease the training and enhance the cross-lingual summarization by establishing alignment of context representations between two languages? learning cross-lingual representations has been proven a beneficial method for cross-lingual transfer for some downstream tasks @xcite @xcite . the underlying idea is to learn a shared embedding space for two languages to improve the model's ability for cross-lingual transfer. recently, it has been shown that this method can also be applied to context representations @xcite . in this paper, we show that the learning of cross-lingual representations is also beneficial for neural crosslingual summarization models. we propose a multi-task framework that jointly learns to summarize and align context-level representations. concretely, we first integrate monolingual summarization models and cross-lingual summarization models into one unified model and then build two linear mappings to project the context representation from one language to the other. we then design several relevant loss functions to learn the mappers and facilitate the cross-lingual summarization. in addition, we propose some methods to enhance the isomorphism and cross-lingual transfer between different languages. we also show that the learning of aligned representation enables our model to generate cross-lingual summaries even in a fully unsupervised way where no parallel crosslingual data is required. we conduct experiments on several public crosslingual summarization datasets. experiment results show that our proposed model outperforms competitive models in most cases, and our model also works on the unsupervised setting. to the best of our knowledge, we are the first to propose an unsupervised framework for learning neural crosslingual summarization. in summary, our primary contributions are as follow: ## overview we show the overall framework of our proposed model in figure 1 . our model consists of two encoders, two decoders, two linear mappers, and two discriminators. suppose we have an english source text @xmath0, . . . , x m } and a chinese source text @xmath1, . . . , y n }, which consist of m and n words, respectively. the english encoder φ e x (res. chinese encoder φ e y ) transforms x (res. y) into its context representation z x (res. z y ), and the decoder φ d x (res. φ d y ) reads the memory z x (res. z y ) and generates the corresponding english summary x (res. chinese summary ỹ). the mappers m x : z @xmath2: z @xmath3, and the discriminators d x and d y are used for discriminating between the encoded representations and the mapped representations. taking english-to-chinese summarization for example, our model generates cross-lingual summaries as follows: first we use the english encoder to get the english context representations, then we use the mapper to map english representations into chinese space. lastly the chinese decoder is used to generate chinese summaries. in section 3, we describe the techniques we adopt to enhance the cross-lingual transferability of the model. in section 4 and section 5, we describe the unsupervised training objective and supervised training objective for cross-lingual summarization, respectively. ## unsupervised training objective we describe the objective of unsupervised crosslingual summarization in this section. the whole training procedure can be found in alg. 1. summarization loss given an english textsummary pair x and x , we use the encoder φ e x and the decoder φ d x to generate the hypothetical english summary x that maximizes the output summary probability given the source text: @xmath4. we adopt maximum loglikelihood training with cross-entropy loss between hypothetical summary x and gold summary x : ## conclusions in this paper, we propose a framework that jointly learns to align and summarize for neural crosslingual summarization. we design training objectives for supervised and unsupervised cross-lingual summarizations, respectively. we also propose methods to enhance the isomorphism and crosslingual transfer between languages. experimental results show that our model outperforms supervised baselines in most cases and outperforms unsupervised baselines in all cases.