Reducing Position Bias in Simultaneous Machine Translation with Length-Aware Framework (2022)

Abstract: Simultaneous machine translation (SiMT) starts translating while receiving the streaming source inputs, and hence the source sentence is always incomplete during translating. Different from the full-sentence MT using the conventional seq-to-seq architecture, SiMT often applies prefix-to-prefix architecture, which forces each target word to only align with a partial source prefix to adapt to the incomplete source in streaming inputs. However, the source words in the front positions are always illusoryly considered more important since they appear in more prefixes, resulting in position bias, which makes the model pay more attention on the front source positions in testing. In this paper, we first analyze the phenomenon of position bias in SiMT, and develop a Length-Aware Framework to reduce the position bias by bridging the structural gap between SiMT and full-sentence MT. Specifically, given the streaming inputs, we first predict the full-sentence length and then fill the future source position with positional encoding, thereby turning the streaming inputs into a pseudo full-sentence. The proposed framework can be integrated into most existing SiMT methods to further improve performance. Experiments on two representative SiMT methods, including the state-of-the-art adaptive policy, show that our method successfully reduces the position bias and thereby achieves better SiMT performance.

## introduction simultaneous machine translation (simt) @xcite @xcite starts translating while receiving the streaming source inputs, which is crucial to many live scenarios, such as simultaneous interpretation, live broadcast and synchronized subtitles. compared with full-sentence machine translation (mt) waiting for the complete source sentence, simt is more challenging since the source sentence is always incomplete during translating. to process the incomplete source, simt has a different architecture from full-sentence mt, as shown in figure 1 . full-sentence mt applies the seq-to-seq architecture @xcite , where each target word can be translated based on a complete source sentence. simt always applies prefix-to-prefix architecture @xcite to force each target word to only align with a source prefix rather than the complete source sentence, where the source prefix consists of partial source words in the front position and is monotonically non-decreasing at each step. although the prefix-to-prefix architecture effectively adapts to the streaming inputs by removing the subsequent source words, it intensifies the structural gap between simt and full-sentence mt, resulting in the following issues. first, since each target word is forced to align with a monotonically non-decreasing source prefix, the source words in different positions become no longer fair. specifically, the source words in the front position participate in more target words' translation due to earlier appearance, and hence are always illusoryly considered more important, resulting in position bias @xcite . due to the position bias, simt model prefers to pay more attention to the source words in front position during testing, which not only robs the attention of the words that are supposed to be aligned (increase mis-translation error) @xcite , but also results in great overlap on attention distribution (aggravate the duplication translation error) @xcite . we will analyze the detailed causes and disadvantages of position bias in sec.3. second, prefix-to-prefix architecture directly removes the subsequent source words, resulting in the lost of some potential full-sentence information @xcite . most importantly, the prefixto-prefix training makes the model insensitive to the full-sentence length, which can provide a global planning for translation @xcite . under these grounds, we propose a length-aware framework (laf) for simt to turn the incomplete source into a pseudo full-sentence, thereby reducing the position bias. we aim to extend the incomplete source sentence in simt to the full-sentence length and meanwhile guarantee that future source words would not be leaked to fulfill the streaming inputs during testing. to this end, laf first predicts the full-sentence length based on the current incomplete source sentence. then, laf fills the future source positions (between the current source length and predicted full-sentence length) with the positional encoding @xcite to construct the pseudo full-sentence. accordingly, each target word is translated based on the pseudo full-sentence and no longer forced to align with the source prefix. laf can be integrated into most of the existing simt methods to further improve performance by bridging the structural gap between simt and full-sentence mt. we apply laf on two representative and strong simt methods, and experiments on iwslt15 @xmath0. ## background we first introduce full-sentence mt and simt with the focus on the prefix-to-prefix architecture. ## preliminary analysis on position bias in this section, we analyze the phenomenon and cause of position bias in simt. in full-sentence mt, the source sentence is complete, so that each source word participates in the translation of all target words. while in prefix-to-prefix architecture for simt, each target word is forced to align with an increasing source prefix, which directly causes that the source words in the front position participate in the translation of more target words during training and hence are always illusoryly considered more important, resulting in position bias. a theoretical analysis of position bias refers to appendix a. during testing, position bias is reflected in the preference of paying more attention to the source words in front positions. to explore the specific impact of position bias, we select the samples with the same source length (77 sentences) in wmt15 @xmath1, and then calculated the average attention weight obtained by each source position in the bucket. since the times of each source position being paid attention to may be different in simt, the average attention weight is averaged on the times of being attended, so the evaluation is fair for each source position. specifically, give the attention weight α ij between target word y i and source word x j , the average attention weight a j at source position j is calculated as: what is position bias? figure 3 (a) shows the average attention obtained by different source positions foot_0 in two representative simt methods, compared with full-sentence mt. simt has a significant difference from the full-sentence mt on the average attention to the source position. in fullsentence mt, the average attention on each position is similar and the back position gets slightly more attention @xcite . however, in both the fix and adaptive policy in simt, the front source positions obviously get more attention due 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 source target (a) full-sentence mt 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 to position bias, especially the first source word. compared with wait-k, mma alleviates the position bias by dynamically adjusting 'waiting' or 'translating', but the first source position still abnormally gets more attention. note that the average attention on the back positions in simt is higher since the times they are attended are less (the denominator in eq.( 6 ) is smaller). specific attention characteristics furthermore, we compare the characteristics of attention distribution in full-sentence mt and simt, shown in figure 3 . in simt, more attention weights are concentrated on the front source positions @xcite , which is not conducive to translation. first, the biased attention on front positions robs the attention of the aligned source word, resulting in mis-translation error. second, much overlapping on attention distribution aggravates the duplication translation error, where a human evaluation proposed by @xcite shows that duplication error in simt is 500% of full-sentence mt. besides, in some cases, even if the aligned source words have not been received, the prefix-to-prefix architecture still forces the target word to align with the irrelevant source prefix, resulting in the confusion on attention @xcite . ## the proposed method based on the preliminary analyses on position bias, we hope that in simt, target words can also align with the reasonable source positions as them in full-sentence mt, including the future positions even though the words on these positions have not yet been received. along this line, we develop a length-aware framework (laf) to turn the streaming inputs into pseudo full-sentence and thereby allow the target words to align with the full-sentence positions rather than a prefix, as shown in figure 5 . the details are introduced following. ## experiments 6.1 datasets we evaluate laf on the following datasets. iwslt15 foot_1 @xmath2. following the previous setting @xcite , we replace words that the frequency less than 5 by ⟨unk⟩, and the vocabulary sizes are 17k and 7.7k for english and vietnamese respectively. wmt15 foot_2 @xmath3.5m pairs) following ma et al. (2019), arivazhagan et al. (2019) and ma et al. ( 2020 ), we use new-stest2013 as validation set (3000 pairs) and new-stest2015 as test set (2169 pairs). bpe @xcite was applied with 32k merge operations and the vocabulary is shared across languages. ## analysis we conduct extensive analyses to understand the specific improvements of our method. unless otherwise specified, all the results are reported on @xmath4.10) and mma (@xmath5.57) under similar latency. ## conclusion in this paper, we develop a length-aware framework for simt to reduce the position bias brought by incomplete source. experiments show that our method achieves promising results by bridging the structural gap between simt and full-sentence mt.