G aussian Multi-head Attention for Simultaneous Machine Translation (2022)

Abstract: Simultaneous machine translation (SiMT) outputs translation while receiving the streaming source inputs, and hence needs a policy to determine where to start translating. The alignment between target and source words often implies the most informative source word for each target word, and hence provides the unified control over translation quality and latency, but unfortunately the existing SiMT methods do not explicitly model the alignment to perform the control. In this paper, we propose Gaussian Multi-head Attention (GMA) to develop a new SiMT policy by modeling alignment and translation in a unified manner. For SiMT policy, GMA models the aligned source position of each target word, and accordingly waits until its aligned position to start translating. To integrate the learning of alignment into the translation model, a Gaussian distribution centered on predicted aligned position is introduced as an alignment-related prior, which cooperates with translation-related soft attention to determine the final attention. Experiments on En-Vi and De-En tasks show that our method outperforms strong baselines on the trade-off between translation and latency.

## introduction simultaneous machine translation (simt) @xcite @xcite , which outputs translation before receiving the complete source sentence, is mainly used for streaming translation tasks, such as simultaneous interpretation, live broadcast and online translation. different from full-sentence machine translation which waits for the complete source sentence, simt requires a policy to determine where to start translating when given the streaming inputs. the simt policy has to trade off between translation quality and latency and an ideal one should wait for the right number of source words, which are sufficient but not excess, until deciding to output target words @xcite . for full-sentence translation, each target word is generated based on the attended source information, where each source word provides different amount of information for the target word. among them, the most informative source word can be considered as an aligned word to the target word @xcite . then for simt, the alignment can be a good guider for the policy to determine where to start translating. for high translation quality, the simt policy is supposed to start translating after receiving the aligned source word to ensure enough source information for the translation. to consider low latency, the simt policy is expected not to wait for too many words after receiving the aligned source word. therefore, if the alignment can be modeled in the simt model explicitly, translation quality and latency can be controlled in a unified manner for an ideal simt policy. however, the existing simt methods, mostly employing fixed or adaptive policy, do not reflect the alignment information in the modeling, which makes them trade off between translation quality and latency in a separate way. fixed policy, such as wait-k policy @xcite , waits for a fixed number of source words and then performs read and write operations one after another, so it is a rule-based policy and precludes alignment modeling. adaptive policy, such as milk @xcite and mma @xcite , determines read/write operations by sampling from a bernoulli distribution, as shown in figure 1(a) , where the decisions are made independently and no relationship between the decision and translation is introduced, so it has to employ an additional loss to control the latency. besides, some methods @xcite apply external ground-truth alignment as an ideal position to start translating, but the performance is inferior to mma since separating translation and alignment. in this paper, to explicitly involve alignments in the simt modeling, we propose gaussian multihead attention (gma) to develop a simt policy with the guidance of alignments. to determine where to start translating with alignment, gma first models the aligned source position of the current target word via predicting the relative distance from the previous aligned source position, called incremental steps, shown in figure 1 @xcite . meanwhile, a relaxation offset after the aligned position is set to allow the model to wait for some additional source inputs, thereby providing a controllable trade-off between translation quality and latency in practice. accordingly, gma starts translating after receiving the aligned source position and waiting for the extra relaxation offset. to jointly learn alignments (i.e., simt policy) and translation, a gaussian distribution centered on predicted aligned position is introduced as a prior attention over the received source words. as a result, gma finally uses the posterior attention for translation derived from the alignment-related gaussian prior and translationrelated soft attention. experiments on @xmath0. ## background gma is applied on the multi-head attention in transformer @xcite , so we briefly introduce simt and the multi-head attention. ## the proposed method the architecture of gma is shown in figure 2 . for simt policy, gma predicts the aligned source position of the current target word, and accordingly determines the output position. to integrate the learning of simt policy within the translation, we introduce a gaussian prior centered on the predicted aligned position, which is multiplied with soft attention (eq.( 2 )) to get final attention distribution. due to the unimodality of gaussian prior, it enables the model to learn the position that gets the highest soft attention (i.e., alignment), thereby developing a reasonable simt policy. besides, since the gaussian prior is continuous and differentiable, it can be integrated into the translation model directly and adjusted with the learning of translation. ## related work a reasonable policy is the key to the simt performance. early policies used segmented translation @xcite @xcite . @xcite al. (2017) arivazhagan et al. (2019) proposed milk, using a bernoulli variable to determine read/write. ma et al. (2020) proposed mma to implement milk on transformer. bahar et al. (2020) and wilken et al. (2020) used the external ground-truth alignments to train the policy. zhang et al. (2020) proposed an adaptive segmentation policy mu for simt. liu et al. (2021) proposed cross-attention augmented transducer for simt. alinejad et al. (2021) introduced an fullsentence model to generate a ground-truth action sequence and accordingly train a simt policy. miao et al. (2021) proposed a generative simt policy. previous methods often neglected to jointly model alignments with translation, and meanwhile introduce additional loss functions to control the latency. however, gma jointly learns alignment and translation, and thereby controls the latency through a simple gaussian prior probability. ## analysis we conduct extensive analyses to study the specific improvements of gma. unless otherwise specified, all the results are reported on @xmath1. ## conclusion in this paper, we propose the gaussian multi-head attention to develop a simt policy which starts translating a target word after receiving its aligned source word. experiments and analyses show that our method achieves promising results on performance, alignments quality and streaming degree. shaolei zhang and yang feng. 2021b. modeling concentrated cross-attention for neural machine translation with gaussian mixture model. in findings of the association for computational linguistics: emnlp 2021, pages 1401-1411, punta cana, dominican republic. association for computational linguistics. shaolei zhang and yang feng. 2021c. universal simultaneous machine translation with mixture-of-experts wait-k policy. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 7306-7317, online and punta cana, dominican republic. association for computational linguistics. shaolei zhang and yang feng. 2022a. modeling dual read/write paths for simultaneous machine translation. in proceedings of the 60th annual meeting of the association for computational linguistics, dublin, ireland. association for computational linguistics. shaolei zhang and yang feng. 2022b. reducing position bias in simultaneous machine translation with length-aware framework. in proceedings of the 60th annual meeting of the association for computational linguistics, dublin, ireland. association for computational linguistics. shaolei zhang, yang feng, and liangyou li. 2021. future-guided incremental transformer for simultaneous translation. in proceedings of the aaai conference on artificial intelligence, volume 35, pages 14428-14436, online. baigong zheng, renjie zheng, mingbo ma, and liang huang. 2019a. simpler and faster learning of adaptive policies for simultaneous translation. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 1349-1354, hong kong, china. association for computational linguistics. baigong zheng, renjie zheng, mingbo ma, and liang huang. 2019b. simultaneous translation with flexible policy via restricted imitation learning. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 5816-5822, florence, italy. association for computational linguistics.