Attend, Translate and Summarize: An Efficient Method for Neural Cross-Lingual Summarization (2020)

Abstract: Cross-lingual summarization aims at summarizing a document in one language (e.g., Chinese) into another language (e.g., English). In this paper, we propose a novel method inspired by the translation pattern in the process of obtaining a cross-lingual summary. We first attend to some words in the source text, then translate them into the target language, and summarize to get the final summary. Specifically, we first employ the encoder-decoder attention distribution to attend to the source words. Second, we present three strategies to acquire the translation probability, which helps obtain the translation candidates for each source word. Finally, each summary word is generated either from the neural distribution or from the translation candidates of source words. Experimental results on Chinese-to-English and English-to-Chinese summarization tasks have shown that our proposed method can significantly outperform the baselines, achieving comparable performance with the state-of-the-art.

## introduction cross-lingual summarization is to produce a summary in a target language (e.g., english) from a document in a different source language (e.g., chinese). cross-lingual summarization can help people efficiently understand the gist of an article written in an unfamiliar foreign language. traditional cross-lingual summarization methods are pipeline-based. these methods either adopt summarize-then-translate @xcite or employ translate-thensummarize @xcite . the pipeline-based approach is intuitive and straightforward, but it suffers from error propagation. due to the difficulty of acquiring cross-lingual summarization dataset, some previous researches focus on zero-shot methods @xcite 10 foshan young couple was detained for charging 10 yuan for buying train tickets online for migrant workers a young couple in foshan who help migrant workers book train tickets online have been detained after receiving a 10-yuan handling fee for each ticket. migrant workers called injustice, and they did not overcharge, much better than scalpers. lawyers said that under the law, disguised mark-up scalping and other acts constitute scalping tickets serious, will be punished. ## background in this paper, we implement our method based on transformer @xcite encoderdecoder framework, where the encoder first maps the input sequence and the decoder generates an output sequence @xmath0, y 2 , • • • , y m ) from the continuous representations. the encoder and decoder are trained jointly to maximize the conditional probability of target sequence given a source sequence: transformer is composed of stacked encoder and decoder layers. the encoder layer is a self-attention block followed by a position-wise feed-forward block. compared with the encoder layer, the decoder layer has an extra encoder-decoder attention block. for self-attention and encoder-decoder attention, a multi-head attention block is used to obtain information from different representation subspaces at different positions. each head corresponds to a scaled dot-product attention, which operates on query q, key k, and value v : ## our model inspired by the phenomenon that some words contained in a cross-lingual summary can be obtained by translating some source words (figure 1 ), we introduce a novel cross-lingual summarization method. it first attends to some source words, then obtains the translation candidates of them, and finally generates words from the translation candidates or the neural distribution. our proposed method is a hybrid between transformer and an additional translation layer, which is depicted in figure 2 and described as follows. attend. inspired by the pointer-generator network @xcite , we employ the encoderdecoder attention distribution α h t (the last layer) to help focus on some salient words in the source text. since α h t is a multi-head attention, we take the mean value over the heads as follow: translate. with the attention distribution on the source words, we also need to know what should each source word be translated into. to achieve that, we obtain a probabilistic bilingual lexicon p l (w 1 ⇒ w 2 ) from existing machine translation corpora and then acquire the translation probability p t based on p l (w 1 ⇒ w 2 ). ## related work cross-lingual summarization. the traditional cross-lingual summarization approaches are based on the pipelined paradigm and can be categorized into translate-then-summarize @xcite and summarize-thentranslate @xcite . @xcite translate the hindi document into english and then generate the english headline. @xcite train a robust abstractive summarization system on noisy english documents and clean english reference summaries. then the system can learn to produce fluent summaries from disfluent inputs, which enables the system to summarize translated documents. @xcite summarize the romanian news and then translate the summary into english. @xcite apply the summarizethen-translate scheme to english-to-chinese crosslingual summarization, which extracts english sentences considering both the informativeness and translation quality of sentences and automatically translates the english summary into chinese. they also argue that summarize-then-translate is better, since it can alleviate both the computational expense of translating sentences and sentence extraction errors caused by incorrect translations. there have been some researches focusing on improving cross-lingual summarization with bilingual information. @xcite translates the english document into chinese and extracts sentences based on the original english sentences and chinese translation. @xcite propose a compressive method which calculates the sentence scores based on the aligned bilingual phrases obtained by machine translation service and performs compression via deleting redundant or poorly translated phrases. @xcite introduce an abstractive method that constructs a pool of bilingual concepts represented by the bilingual elements of the source-side predicate-argument structures and the target-side counterparts. recently, end-to-end methods have been applied to cross-lingual summarization. due to the lack of supervised training data, @xcite and @xcite focus on zero-shot training methods that use machine translation or monolingual summarization or both to teach the cross-lingual system. @xcite propose to acquire largescale datasets via a round-trip translation strategy. they incorporate monolingual summarization or machine translation into cross-lingual summarization training using multi-task learning. neural abstractive summarization. @xcite present the first neural abstractive summarization model, an attentive convolutional encoder and a neural network language model decoder, which learns to generate news headlines from the lead sentences of news articles. their approach has been further improved with recurrent decoders @xcite , abstractive meaning representations @xcite , hierarchical networks @xcite , variational autoencoders @xcite , hybrid strategy @xcite , selective mechanism @xcite , and entailment knowledge. @xcite propose a pointer-generator network, which allows copying words from the source text with the copying mechanism @xcite . @xcite incorporate entailment knowledge into summarization model to improve the correctness of the generated summaries. @xcite apply guidance signals of keywords to both the encoder and decoder in the abstractive summarization model. inspired by the pointer-generator network and the translation pattern in obtaining cross-lingual summaries, we introduce a novel model in this paper, which integrates the operation of attending, translating, and summarizing. ## conclusion and future work in this paper, we present a novel method consistent with the translation pattern in the process of obtaining a cross-lingual summary. this method first attends to the source words, then obtains the translation candidates, and incorporates them into the generation of the final summary. experimental results have shown that our method can significantly outperform the baseline and achieve comparable performance with the state-of-the-art. furthermore, our method has two advantages over the state-ofthe-art: (1) our model requires only an additional probabilistic bilingual lexicon rather than largescale parallel datasets of other tasks, thus reducing the model's dependence on data and making it easier for the model to migrate to other domains or other language pairs. (2) our model has a much smaller size and a much faster training efficiency. in our future work, we consider incorporating our method into the multi-task method. besides, we will also explore the influence of probabilistic bilingual lexicon obtained by learning only from monolingual data on our method.