Context-Aware Cross-Attention for Non-Autoregressive Translation (2020)

Abstract: Non-autoregressive translation (NAT) significantly accelerates the inference process by predicting the entire target sequence. However, due to the lack of target dependency modelling in the decoder, the conditional generation process heavily depends on the cross-attention. In this paper, we reveal a localness perception problem in NAT cross-attention, for which it is difficult to adequately capture source context. To alleviate this problem, we propose to enhance signals of neighbour source tokens into conventional cross-attention. Experimental results on several representative datasets show that our approach can consistently improve translation quality over strong NAT baselines. Extensive analyses demonstrate that the enhanced cross-attention achieves better exploitation of source contexts by leveraging both local and global information.

## introduction different from autoregressive translation @xcite models that generate each target word conditioned on previously generated ones, non-autoregressive translation @xcite models break the autoregressive factorization and produce the target words in parallel. given a source sentence x, the probability of generating its target sentence y with length t is defined by nat as: p(y|x) = p l (t |x; θ) t @xmath0; θ), where p l (•) is a separate conditional distribution to predict the length of target sequence. as nat models can predict all tokens independently and simultaneously, recent works have fully investigated their superiority on decoding efficiency @xcite @xcite @xcite @xcite . however, there still exists a gap between at and nat models in terms of effectiveness. in encoder-decoder frameworks, the cross-attention module dynamically selects relevant source-side information (key) given a target-side token (query) @xcite . through qualitative and quantitative analyses, we found that it is difficult for the nat decoder to adequately capture the source context due to the lack of autoregressive factorization. as shown in to alleviate this localness perception problem in nat, we propose a context-aware cross-attention to model both local and global contexts simultaneously. for local attention, we limit the scope of cross-attention to adjacent tokens surrounding the source word with the maximum alignment probability. we then combine the local attention weights with the original global ones by a gating mechanism (in section 3). reference frank found an apartment and was dating a girl at the same time. nat output frank found an apartment and was socializing with a girl. attention 交往0.68 弗兰克0.18 女孩0.04 at output frank found an apartment and was dating a girl. attention 交往0.81 女孩0.15 弗兰克0.03 ours output frank found an apartment and was dating a girl. attention 交往0.69 女孩0.11 弗兰克0.09 experiments are conducted on four commonly-cited datasets on translation task (i.e. wmt16 romanian⇒english, wat17 japanese⇒english, wmt14 english⇒german and wmt17 chinese⇒english) and show that our approach can consistently improve translation quality by around 0.5 bleu point over advanced nat models (in section 4). further analyses reveal that our method can enhance abilities of nat to learn syntactic and semantic information as well as phrase patterns (in section 5). ## localness perception problem to validate our motivation, we conduct a statistical analysis. following @xcite , we employ the locality entropy to measure how the cross-attention concentrate around a source word that corresponds with y t . as shown in ## context-aware cross-attention for nat in this section, we introduce the detail of our proposed context-aware cross-attention networks (ccan), which perceives the original and local cross-attention simultaneously. original cross-attention for the target-side query q, source-side key k and value v . the i-th original cross-attention ψ i can be calculated with dot-product: ψ @xmath4. the original attention of the i-th element is the weighted sum of values att(ψ i , v ) = sof tmax(ψ i )v (in figure 1(a) ). our approach for the i-th position in target side, we propose a locally-sensitive cross-attention component for nat to capture the neighbor signals. for simplicity, we adopt a straightforward but has been proven effective way @xcite @xcite : constricting the attention 0.03 0.04 0.68 0.05 0.02 0.18 socializing 0.14 0.15 0.71 decoder encoder dating decoder encoder (a) vanilla non-autoregressive model 0.03 0.04 0.68 0.05 0.02 0.18 socializing 0.14 0.15 0.71 decoder encoder dating decoder encoder (b) localness modeling in fixed window scope to a nearby window around the aligned j-th element. in practice, we choose the source element with the highest attention weight as the aligned element, and the local range can be modeled as follows: ## analysis in this section, we conduct extensive analyses on wmt14 en-de to better understand how our method contribute to performance gains. ## conclusion and future work we reveal a localness perception problem in nat. to alleviate it, we propose the context-aware approach to make the cross-attention pay more attention to source-side local words, which in turn improves the translation performance over several benchmarks. in future work, we will investigate selectively choosing the context @xcite rather than the fixed window size. besides, it is interesting to enhance nat model with extra signals, such as cross-lingual position embedding @xcite , larger context @xcite and pre-trained initialization @xcite .