Language Model Augmented Monotonic Attention for Simultaneous Translation (2022)

Abstract: The state-of-the-art adaptive policies for Simultaneous Neural Machine Translation (SNMT) use monotonic attention to perform read/write decisions based on the partial source and target sequences. The lack of sufficient information might cause the monotonic attention to take poor read/write decisions, which in turn negatively affects the performance of the SNMT model. On the other hand, human translators make better read/write decisions since they can anticipate the immediate future words using linguistic information and domain knowledge. In this work, we propose a framework to aid monotonic attention with an external language model to improve its decisions. Experiments on MuST-C English-German and English-French speech-to-text translation tasks show the future information from the language model improves the state-of-the-art monotonic multi-head attention model further.

## introduction a typical application of simultaneous neural machine translation (snmt) is conversational speech or live video caption translation. in order to achieve live translation, an snmt model alternates between performing read from source sequence and write to target sequence. for a model to decide whether to read or write at certain moment, either a fixed or an adaptive read/write policy can be used. earlier approaches in simultaneous translation such as @xcite and @xcite employ a fixed policy that alternate between read and write after the waiting period of k tokens. to alleviate possible long delay of fixed polices, recent works such as monotonic infinite lookback attention (milk) @xcite , and monotonic multihead attention (mma) @xcite developed flexible policies using monotonic attention @xcite . while these monotonic attention anticipates target words using only available prefix source and target sequence, human translators anticipate the target words using their language expertise (linguistic anticipation) as well as contextual information (extra-linguistic anticipation) @xcite . inspired by human translation experts, we aim to augment monotonic attention with future information using language models (lm) @xcite . integrating the external information effectively into text-to-text machine translation (mt) systems has been explored by several works @xcite @xcite . also, integrating future information implicitly into snmt system during training is explored in wu et al. (2020) by simultaneously training different wait-k snmt systems. however, no previous works make use of explicit future information both during training and inference. to utilize explicit future information, we explored to integrate future information from lm directly into the output layer of the mma model. however, it did not provide any improvements (refer to appendix a), thus motivating us to explore a tighter integration of the lm information into snmt model. in this work, we explicitly use plausible future information from lm during training by transforming the monotonic attention mechanism. as shown in figure 1, at each step, the lm takes the prefix target (and source, for cross-lingual lm) sequence and predicts the probable future information. we hypothesize that aiding the monotonic attention with this future information can improve mma model's read/write policy, eventually leading to better translation with less delay. several experiments on must-c (di gangi et al., 2019) english-german and english-french speech-to-text translation tasks with our proposed approach show clear improvements of latency-quality trade-offs over the state-of-the-art mma models. ## conclusion in this work, we provide a generic framework to integrate the linguistic and extra-linguistic information into simultaneous models. we rely on lan-guage models to extract this plausible future information and propose a new monotonic attention mechanism to infuse this information. several experiments on speech-to-text translation tasks show the effectiveness of proposed approach on obtaining superior quality-latency trade-offs, compared to the state-of-the-art monotonic multihead attention. sonia vandepitte. 2001. anticipation in conference interpreting: a cognitive process. alicante journal of english studies / revista alicantina de estudios ingleses, 0(14):323-335. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, ł ukasz kaiser, and illia polosukhin. 2017. attention is all you need. in advances in neural information processing systems 30, pages 5998-6008. curran associates, inc. xueqing wu, yingce xia, lijun wu, shufang xie, weiqing liu, jiang bian, tao qin, and tie-yan liu. 2020. learn to use future information in simultaneous translation.