Self-Attention with Cross-Lingual Position Representation (2020)

Abstract: Position encoding (PE), an essential part of self-attention networks (SANs), is used to preserve the word order information for natural language processing tasks, generating fixed position indices for input sequences. However, in cross-lingual scenarios, machine translation, the PEs of source and target sentences are modeled independently. Due to word order divergences in different languages, modeling the cross-lingual positional relationships might help SANs tackle this problem. In this paper, we augment SANs with cross-lingual position representations to model the bilingually aware latent structure for the input sentence. Specifically, we utilize bracketing transduction grammar (BTG)-based reordering information to encourage SANs to learn bilingual diagonal alignments. Experimental results on WMT’14 English \Rightarrow German, WAT’17 Japanese \Rightarrow English, and WMT’17 Chinese \Leftrightarrow English translation tasks demonstrate that our approach significantly and consistently improves translation quality over strong baselines. Extensive analyses confirm that the performance gains come from the cross-lingual information.

## introduction although self-attention networks (sans) @xcite have achieved the state-of-the-art performance on several natural language processing (nlp) tasks @xcite @xcite , they possess the innate disadvantage of sequential modeling due to the lack of positional information. therefore, absolute position encoding (ape) @xcite and relative position encoding (rpe) @xcite were introduced to better capture the sequential dependencies. however, either absolute or relative pe is language-independent and its embedding bush with sharon held a talk bush held a talk with sharon [source] [re-ordered] 0 1 2 3 4 5 [abs pos] 0 3 4 5 1 2 [xl pos] 布什 与 沙龙 举行 了 会谈 remains fixed. this inhibits the capacity of sans when modelling multiple languages, which have diverse word orders and structures @xcite . recent work have shown that modeling cross-lingual information (e.g., alignment or reordering) at encoder or attention level improves translation performance for different language pairs @xcite @xcite . inspired by their work, we propose to augment sans with cross-lingual representations, by encoding reordering indices at embedding level. taking english⇒chinese translation task for example, we first reorder the english sentence by deriving a latent bracketing transduction grammar (btg) tree @xcite (fig. 1a ). similar to absolute position, the reordering information can be represented as cross-lingual position (fig. 1b ). in addition, we propose two strategies to incorporate cross-lingual position encoding into sans. we conducted experiments on three commonlycited datasets of machine translation. results show that exploiting cross-lingual pe consistently improves translation quality . further analysis reveals that our method improves the alignment quality ( §sec. 4.3) and context-free transformer @xcite ( §sec. 4.4). furthermore, contrastive evaluation demonstrates that nmt models benefits from the cross-lingual information rather than denoising ability ( §sec. 4.5). ## background position encoding to tackle the position unaware problem, absolute position information is injected into the sans: self-attention the sans compute the attention of each pair of elements in parallel. it first converts the input into three matrices q, k, v, representing queries, keys, and values, respectively: the output is then computed as a weighted sum of values by att(q, k, v). sans can be implemented with multi-head attention mechanism, which requires extra splitting and concatenation operations. specifically, w q , w k , w v and q, k, v in eq. ( 3 ) is split into h sub-matrices, yielding h heads. for the h-th head, the output is computed by: finally, these subspaces are combined with concatenation operation: ## experiments we conduct experiments on word order-diverse language pairs: wmt'14 english⇒german (en-de), wat'17 japanese⇒english (ja-en), and wmt'17 chinese⇔english (zh-en & en-zh). for english⇒german, the training set consists of 4.5 million sentence pairs and newstest2013 & 2014 are used as the dev. and test sets, respectively. bpe with 32k merge operations is used to handle low-frequency words. for japanese⇒english, we follow @xcite to use the first two sections as training data, which consists of 2.0 million sentence pairs. the dev. and test sets contain 1790 and 1812 sentences. for chinese⇔english, we follow @xcite to get 20 million sentence pairs. we develop on devtest2017 and test on newstest2017. we use sacrebleu @xcite as the evaluation metric with statistical significance test @xcite . we evaluate the proposed xl pe strategies on transformer. the baseline systems include relative pe @xcite and directional san @xcite . we implement them on top of opennmt @xcite . in addition, we report the results of previous studies @xcite @xcite @xcite . the reordered source sentences are generated by btg-based preordering model @xcite trained with above sub-word level foot_0 parallel corpus. at training phase, we first obtain word alignments from parallel data using giza++ or fastalign, and then the training process is to find the optimal btg tree for source sentence consistent with the order of the target sentence based on the word alignments and parallel data. at decoding phase, we only provide source sentences as input and the model can output reordering indices, which will be fed into nmt model. thus, bilingual alignment information is only used to preprocess training data, but not necessary at decoding time. for fair comparison, we keep the transformer decoder unchanged and validate different position representation strategies on the encoder. we conduct all experiments on the transformer-big with four v100 gpus. ## conclusions and future work in this paper, we presented a novel cross-lingual position encoding to augment sans by considering cross-lingual information (i.e., reordering indices) for the input sentence. we designed two strategies to integrate it into sans. experiments indicated that the proposed strategies consistently improve the translation performance. in the future, we plan to extend the cross-lingual position encoding to non-autoregressive mt @xcite and unsupervised nmt @xcite .