Modeling Concentrated Cross-Attention for Neural Machine Translation with G aussian Mixture Model (2021)

Abstract: Cross-attention is an important component of neural machine translation (NMT), which is always realized by dot-product attention in previous methods. However, dot-product attention only considers the pair-wise correlation between words, resulting in dispersion when dealing with long sentences and neglect of source neighboring relationships. Inspired by linguistics, the above issues are caused by ignoring a type of cross-attention, called concentrated attention, which focuses on several central words and then spreads around them. In this work, we apply Gaussian Mixture Model (GMM) to model the concentrated attention in cross-attention. Experiments and analyses we conducted on three datasets show that the proposed method outperforms the baseline and has significant improvement on alignment quality, N-gram accuracy, and long sentence translation.

## introduction recently, neural machine translation (nmt) has been greatly improved with transformer @xcite , which mainly relies on the attention mechanism. the attention mechanism in transformer consists of self-attention and crossattention, where cross-attention is proved more important to translation quality than self-attention @xcite . even if the self-attention is modified to a fixed template, the translation quality would not significantly reduce @xcite , while cross-attention plays an irreplaceable role in nmt. cross-attention in transformer is realized by dot-product attention, which calculates the attention distribution base on the pairwise similarity. however, modeling cross-attention with the dotproduct attention still has some weaknesses due to its calculation way. first, when dealing with long sentences, the attention distribution with dotproduct attention tends to be dispersed @xcite , which proved unfavorable for translation @xcite @xcite . second, dot-product attention is difficult to explicitly consider the source neighboring relationship @xcite , resulting in ignoring the words with lower similarity but nearby the important word which determine phrase structure or morphology. research in linguistics and cognitive science suggests that human attention to language can be divided into two categories: distributed attention and concentrated attention @xcite @xcite . specifically, distributed attention is scattered on all source words, and the degree of attention is determined by correlation. on the contrary, concentrated attention only focuses on a few central words and then spreads on the words around them. accordingly, we consider that cross-attention can be divided into these two types of attention as well, where dis-tributed attention is well modeled by dot-product attention, but concentrated attention is ignored. figure 1 shows an attention example of @xmath0: pencil case). only with distributed attention, some irrelevant words (such as 'desk') robbed some attention weight, resulting in attention dispersion. besides, the correlation to the function words (both 'a' and 'some') are low and similar, but they are important to determine the singular/plural forms of the target word. in concentrated attention, attention distribution is more concentrated and can capture neighboring relationships, which make up for the lack of distributed attention. in this paper, to explicitly model the concentrated cross-attention, we apply the gaussian mixture model (gmm) @xcite to construct gaussian mixture attention. specifically, gaussian mixture attention first focuses on some central words, and then pays attention to the words around the central words, where the attention decreases as the word away from the central word. since crosslingual alignments are often one-to-many, gaussian mixture attention is more flexible to model multiple central words, which is not possible with a single gaussian distribution. then, gaussian mixture attention and dot-product attention are fused to jointly determine the total attention. experiments we conducted on three datasets show that our method outperforms the baseline on translation quality. further analyses show that our method enhances cross-attention, thereby improving the performance of alignment quality, n-gram accuracy and long sentence translation. our contributions can be summarized as follows: ## background our method is applied on cross-attention in transformer @xcite , so we first briefly introduce the architecture of transformer with a focus on its dot-product attention. then, we give the concept of the gaussian mixture model. ## the proposed method to improve cross-attention, in addition to dotproduct attention, we introduce the gaussian mixture attention into each head of cross-attention. as shown in figure 2 , we first calculate the dotproduct attention and gaussian mixture attention, and then fuses them through a gating mechanism to determine the total attention distribution. the proposed gaussian mixture attention is constructed by mean, variance, and weight, all of which are predicted based on the target word, as shown in figure 3 . the specific details will be introduced following. ## experiments we conducted experiments on three datasets and compare with the baseline and previous methods to evaluate the performance of the proposed method. ## analysis we conducted extensive analyses to understand the specific improvements of our method in attention entropy, alignment quality, phrase fluency, and long sentence translation. unless otherwise specified, all the results are reported on wmt14 @xmath1. ## conclusion inspired by linguistics, we decompose the crossattention into distributed attention and concentrated attention. to model the concentrated attention, we apply gmm to construct the gaussian mixture attention, which effectively resolves the weakness of dot-product attention. experiments show that the proposed method outperforms the strong baseline on three datasets. further analyses show the specific advantages of the proposed method in attention distribution, alignment quality, n-gram accuracy, and long sentence translation.