Cross-Lingual Abstractive Summarization with Limited Parallel Resources (2021)

Abstract: Parallel cross-lingual summarization data is scarce, requiring models to better use the limited available cross-lingual resources. Existing methods to do so often adopt sequence-to-sequence networks with multi-task frameworks. Such approaches apply multiple decoders, each of which is utilized for a specific task. However, these independent decoders share no parameters, hence fail to capture the relationships between the discrete phrases of summaries in different languages, breaking the connections in order to transfer the knowledge of the high-resource languages to low-resource languages. To bridge these connections, we propose a novel Multi-Task framework for Cross-Lingual Abstractive Summarization (MCLAS) in a low-resource setting. Employing one unified decoder to generate the sequential concatenation of monolingual and cross-lingual summaries, MCLAS makes the monolingual summarization task a prerequisite of the CLS task. In this way, the shared decoder learns interactions involving alignments and summary patterns across languages, which encourages attaining knowledge transfer. Experiments on two CLS datasets demonstrate that our model significantly outperforms three baseline models in both low-resource and full-dataset scenarios. Moreover, in-depth analysis on the generated summaries and attention heads verifies that interactions are learned well using MCLAS, which benefits the CLS task under limited parallel resources.

## introduction cross-lingual summarization (cls) helps people efficiently grasp salient information from articles in a foreign language. neural approaches to cls require large scale datasets containing millions of cross-lingual document-summary pairs @xcite , 2019; @xcite . however, two challenges arise with these approaches: 1) most languages are low-resource, thereby lacking document-summary paired data; 2) large parallel datasets across different languages for neuralbased cls are rare and expensive, especially under the current trend of neural networks. therefore, a low-resource setting is more realistic, and challenging, one for cross-lingual summarization. to our best knowledge, cross-lingual summarization under low-resource settings has not been well investigated and explored. therefore, in this paper, we will develop a new model for cross-lingual abstractive summarization under limited supervision. for low-resource settings, multi-task learning has been shown to be an effective method since it can borrow useful knowledge from other relevant tasks to use in the target task @xcite @xcite . crosslingual summarization can be viewed as the combination of two tasks, i.e., monolingual summarization (ms) and cross-lingual translation @xcite . a wealth of relationships exist across the target summaries of ms and cls tasks, such as translation alignments and summarization patterns. illustrated in figure 1 , "叙利亚" is mapped to "syria", and similar maping is done with the other aligned phrases. obviously, leveraging these relationships is crucial for the task of transferring summarization knowledge from high-resource languages to low-resource languages. unfortunately, existing multi-task frameworks simply utilize independent decoders to conduct ms and cls task separately @xcite , which leads to failure in capturing these relationships. to solve this problem, we establish reliant connections between ms and cls tasks, making the monolingual task a prerequisite for the crosslingual task. specifically, one decoder is shared by both ms and cls tasks; this is done by setting the generation target as a sequential concatenation of a monolingual summary and the corresponding cross-lingual summary. sequentially generating monolingual and cross-lingual summaries, the decoder also conducts the translation task between them, which enhances the interactions between different languages. these interactions implicitly involve translation alignments, similarity in semantic units, and summary patterns across different lingual summaries. to demonstrate these decoder interactions, we further visualize them by probing transformer attention heads in the model. based on this process, the new structure with these advanced interactions enhances low-resource scenarios which require the model to be capable of transferring summary knowledge from high-resource languages to low-resource language. we name our model multi-task cross-lingual abstractive summarization (mclas) under limited resources. in terms of a training strategy under limited resources, we first pre-train mclas on large-scale monolingual document-summary parallel datasets to well-equip the decoder with general summary capability. given a small amount of parallel crosslingual summary samples, the model is then finetuned and is able to transfer the learned summary capability to the low-resource language, leveraging the interactions uncovered by the shared decoder. experiments on zh2ensum @xcite and a newly developed en2desum dataset demonstrate that mclas offers significant improvements when compared with state-of-the-art cross-lingual summarization models in both low-resource scenarios and full-dataset scenario. at the same time, we also achieved competitive performances in the en2zhsum dataset @xcite . human evaluation results show that mclas produces more fluent, concise and informative summaries than baselines models under limited parallel resources. in addition, we analyzed the length of generated summaries and the success of monolingual generation to verify advantages offered by identifying interactions between languages. we further investigate the explainability of the proposed multi-task structure by probing the attention heads in the unified decoder, proving that mclas learns the alignments and interactions between two languages, and this facilitates translation and summarization in the decoder stage. our analysis provides a clear explanation of why mclas is capable of supporting cls under limited resources. our implementation and data are available at https://github.com/woodenwhite/mclas . ## mclas with limited parallel resources to strengthen the connections mentioned, we propose making the monolingual task a prerequisite for the cross-lingual task through modeling interactions. according to previous work @xcite @xcite , interactions between cross-lingual summaries (important phrase alignments, sentence lengths, and summary patterns, etc) are crucial for the final summary's quality. we leverage these interactions to further transfer the rich-resource language knowledge. detailed descriptions of this step are presented in following sections. ## probing into attention heads we have observed a successful alignment between s a and s b produced by our model in section 5.9. in this section, we dig into this and analyze how the model learns the relationships. for a cls task from document d a to s b , our hypotheses are: (1) the unified decoder is implicitly undertaking translation from s a to s b ; (2) the unified decoder also conducts both monolingual and cross-lingual summarization. to verify these hypotheses, we visualize attention distributions of the transformer decoders trained on en2zhsum. neural models can be explicitly explained using probing into the attention heads @xcite . we follow the previous work and visualize the function of all attention heads in the decoder to verify the relationships of the concatenated cross-lingual summaries (i.e., translation) and cross-lingual document-summary pairs (i.e., summarization). ## discussions an ideal low-resource experiment should be conducted with real low-resource languages. although possible, it takes much effort to acquire such datasets. hence, it is the second-best choice that we simulate our low-resource scenarios by artificially limiting the amount of the available data. some may question it about the feasibility of our method in real low-resource languages since machine translation systems, which is used to generate document-summary pairs, would be of lower quality for truly low-resource languages. for this concern, we consider it still possible to acquire thousands of high-quality human translated parallel summaries, as duan et al. (2019b) adopt on their test set, to apply our method. ## conclusion in this paper, we propose a novel multi-task learning framework mclas to achieve cross-lingual abstractive summarization with limited parallel resources. our model shares a unified decoder that sequentially generates both monolingual and crosslingual summaries. experiments on two crosslingual summarization datasets demonstrate that our framework outperforms all the baseline models in low-resource and full-dataset scenarios. figure 9 : visualization of all the 48 self attention heads. the x-axis and y-axis are both concatenated sourcelanguage summary s a and target-language summary s b tokens. each row contains all of the attention heads of corresponding layer from bottom to the top. the darker color shows the more highly related associations between tokens. figure 10 : visualization of all the 48 encoder-decoder attention heads. the x-axis is concatenated source-language summary s a and target-language summary s b tokens while the y-axis is document d a tokens. each row contains all of the attention heads of corresponding layer from bottom to the top. the darker color shows the more highly related associations between tokens.