A Variational Hierarchical Model for Neural Cross-Lingual Summarization (2022)

Abstract: The goal of the cross-lingual summarization (CLS) is to convert a document in one language (e.g., English) to a summary in another one (e.g., Chinese). The CLS task is essentially the combination of machine translation (MT) and monolingual summarization (MS), and thus there exists the hierarchical relationship between MT&MS and CLS. Existing studies on CLS mainly focus on utilizing pipeline methods or jointly training an end-to-end model through an auxiliary MT or MS objective. However, it is very challenging for the model to directly conduct CLS as it requires both the abilities to translate and summarize. To address this issue, we propose a hierarchical model for the CLS task, based on the conditional variational auto-encoder. The hierarchical model contains two kinds of latent variables at the local and global levels, respectively. At the local level, there are two latent variables, one for translation and the other for summarization. As for the global level, there is another latent variable for cross-lingual summarization conditioned on the two local-level variables. Experiments on two language directions (English-Chinese) verify the effectiveness and superiority of the proposed approach. In addition, we show that our model is able to generate better cross-lingual summaries than comparison models in the few-shot setting.

## introduction the cross-lingual summarization (cls) aims to summarize a document in source language (e.g., english) into a different language (e.g., chinese), which can be seen as a combination of machine translation (mt) and monolingual summarization (ms) to some extent @xcite . the cls can help people effectively master the core points of an article in a foreign language. under the background of globalization, it becomes more important and is now coming into widespread use in real life. many researches have been devoted to dealing with this task. to our knowledge, they mainly fall into two categories, i.e., pipeline and end-to-end learning methods. (i) the first category is pipeline-based, adopting either translationsummarization @xcite or summarization-translation @xcite paradigm. although being intuitive and straightforward, they generally suffer from error propagation problem. (ii) the second category aims to train an endto-end model for cls @xcite . for instance, @xcite focus on using a pre-constructed probabilistic bilingual lexicon to improve the cls model. furthermore, some researches resort to multi-task learning @xcite @xcite . @xcite separately introduce mt and ms to improve cls. @xcite design several additional training objectives (e.g., ms, back-translation, and reconstruction) to enhance the cls model. @xcite utilize a mixed-lingual pre-training method with several auxiliary tasks for cls. as pointed out by @xcite , it is challenging for the model to directly conduct cls as it requires both the abilities to translate and summarize. although some methods have used the related tasks (e.g., mt and ms) to help the cls, the hierarchical relationship between mt&ms and cls are not well modeled, which can explicitly enhance the cls task. apparently, how to effectively model the hierarchical relationship to exploit mt and ms is one of the core issues, especially when the cls data are limited. 1 in many other related nlp tasks @xcite @xcite , the conditional variational auto-encoder (cvae) @xcite has shown its superiority in learning hierarchical structure with hierarchical latent variables, which is often leveraged to capture the semantic connection between the utterance and the corresponding context of conversations. inspired by these work, we attempt to adapt cvae to model the hierarchical relationship between mt&ms and cls. therefore, we propose a variational hierarchical model to exploit translation and summarization simultaneously, named vhm, for cls task in an end-to-end framework. vhm employs hierarchical latent variables based on cvae to learn the hierarchical relationship between mt&ms and cls. specifically, the vhm contains two kinds of latent variables at the local and global levels, respectively. firstly, we introduce two local variables for translation and summarization, respectively. the two local variables are constrained to reconstruct the translation and source-language summary. then, we use the global variable to explicitly exploit the two local variables for better cls, which is constrained to reconstruct the target-language summary. this makes sure the global variable captures its relationship with the two local variables without any loss, preventing error propagation. for inference, we use the local and global variables to assist the cross-lingual summarization process. we validate our proposed training framework on the datasets of different language pairs @xcite : zh2ensum (chinese⇒english) and en2zhsum (english⇒chinese). experiments show that our model achieves consistent improvements on two language directions in terms of both automatic metrics and human evaluation, demonstrating its effectiveness and generalizability. fewshot evaluation further suggests that the local and global variables enable our model to generate a satisfactory cross-lingual summaries compared to existing related methods. our main contributions are as follows foot_2 : ## background machine translation (mt). given an input sequence in the source language x @xmath0, the goal of the neural mt model is to produce its translation in the target language y @xmath1. the conditional distribution of the model is: ## methodology fig. 1 demonstrates an overview of our model, consisting of four components: encoder, variational hierarchical modules, decoder, training and inference. specifically, we aim to explicitly exploit the mt and ms for cls simultaneously. therefore, we firstly use the encoder ( § 3.1) to prepare the representation for the variational hierarchical module ( § 3.2), which aims to learn the two local variables for the global variable in cls. then, we introduce the global variable into the decoder ( § 3.3). finally, we elaborate the process of our training and inference ( § 3.4). ## conclusion in this paper, we propose to enhance the cls model by simultaneously exploiting mt and ms. given the hierarchical relationship between mt&ms and cls, we propose a variational hierarchical model to explicitly exploit and combine them in cls process. experiments on zh2ensum and en2zhsum show that our model significantly improves the quality of cross-lingual summaries in terms of automatic metrics and human evaluations. particularly, our model in the few-shot setting still works better, suggesting its superiority and generalizability.