Cross Attention Augmented Transducer Networks for Simultaneous Translation (2021)

Abstract: This paper proposes a novel architecture, Cross Attention Augmented Transducer (CAAT), for simultaneous translation. The framework aims to jointly optimize the policy and translation models. To effectively consider all possible READ-WRITE simultaneous translation action paths, we adapt the online automatic speech recognition (ASR) model, RNN-T, but remove the strong monotonic constraint, which is critical for the translation task to consider reordering. To make CAAT work, we introduce a novel latency loss whose expectation can be optimized by a forward-backward algorithm. We implement CAAT with Transformer while the general CAAT architecture can also be implemented with other attention-based encoder-decoder frameworks. Experiments on both speech-to-text (S2T) and text-to-text (T2T) simultaneous translation tasks show that CAAT achieves significantly better latency-quality trade-offs compared to the state-of-the-art simultaneous translation approaches.

## introduction simultaneous translation, which starts to translate input sentences before they are finished, is of importance to many real-life applications such as teleconference systems and time-sensitive spoken document analysis and conversion. while a substantial progress has been made on offline machine translation @xcite @xcite , more research on simultaneous translation is yet highly desirable. central to the task is performing high-quality low-latency translation, which involves the key challenges of developing optimal policies for the read-write action paths as well as generating high-quality target sequences based only on partial source sequences. this paper aims to optimize the policy and translation model jointly, by expanding target sequences with blank symbols for read actions. the loss function can be defined as negative loglikelihood (nll) of marginal distribution through all expanded paths. a similar problem in automatic speech recognition (asr) has been tackled with rnn-t (recurrent neural network transducer) @xcite by an efficient forward-backward algorithm. however, rnn-t is trained based on the monotonic alignment between source and target sequences, which is not suitable for simultaneous translation, as it cannot properly consider reordering. on the other hand, the forward-backward algorithm is not available for attention-based encoderdecoder @xcite architectures, including transformer @xcite , due to the deep coupling between source contexts and target history contexts. to solve this problem, we separate the cross attention mechanism from target history representation in attention-based encoder-decoder, which can also be viewed as rnn-t with the joiner being augmented by cross attention mechanism, resulting in cross attention augmented transducer (caat). however, cross attention mechanism removes the alignment constraint in rnn-t which originally encourages an appropriate latency. to ensure latency under control, jointly minimizing a latency loss is required. both the nll loss and latency loss can be efficiently optimized by a forward-backward algorithm. the main contributions of this paper are threefold: (1) we propose a novel architecture, cross attention augmented transducer, which jointly optimizes the policy and translation model by considering all possible read-write simultaneous translation action paths. (2) we introduce a novel latency loss whose expectation can be optimized by a forward-backward algorithm. training with this latency loss ensures the latency of caat simultaneous translation model to be under control. (3) the proposed model achieves significantly better latency-quality trade-offs compared to the state-ofthe-art simultaneous translation approaches. ## related work recent work on simultaneous translation falls into two categories. the first category uses a fixed policy for the read/write actions. @xcite propose simultaneous translation with the wait-if-* policy for an offline model. @xcite propose a wait-k policy for both the training and inference period. the second category includes models with a flexible policy learned and/or adaptive to current context. @xcite introduce an agent trained by reinforcement learning from the interaction with a pre-trained offline neural machine translation model. zheng et al. (2019a) train the agent by supervise learning with label sequences generated via the rank of golden target words given partial input. a special subcategory of flexible policy jointly optimize policy and translation by monotonic attention customized to translation model, e.g., monotonic infinite lookback (milk) attention @xcite on rnn encoder-decoder @xcite and monotonic multihead attention (mma) @xcite on transformer @xcite . end-to-end speech-to-text (s2t) simultaneous translation has been investigated in @xcite , among which @xcite adapt latency metrics from t2t simultaneous translation to s2t simultaneous translation, and experiment with both the fixed and flexible policy. @xcite study the effect of speech block processing on s2t simultaneous translation. @xcite experiment with the wait-k policy based on a source language ctc segmenter. in our work, we optimize the marginal distribution of all expanded paths motivated by rnn-t @xcite . unlike rnn-t, the caat model removes the monotonic constraint, which is critical for considering reordering in machine translation tasks. the optimization of our latency loss is motivated by sequence discriminative training in asr @xcite . ## conclusions this paper proposes cross attention augmented transducer (caat), a novel simultaneous translation model that jointly optimizes policy and translation model by considering all possible read-write action paths. crucial to the model is a now, chimpanzees are well-known for their aggression. nun, schimpansen sind bekannt für ihre aggressivität. nun, schimpansen sind für ihre aggression bekannt. yonghui wu, mike schuster, zhifeng chen, quoc v le, mohammad norouzi, wolfgang macherey, maxim krikun, yuan cao, qin gao, klaus macherey, et al. 2016. google's neural machine translation system: bridging the gap between human and machine translation. arxiv preprint arxiv:1609.08144. ching-feng yeh, jay mahadeokar, kaustubh kalgaonkar, yongqiang wang, duc le, mahaveer jain, kjell schubert, christian fuegen, and michael l seltzer. 2019. transformer-transducer: end-toend speech recognition with self-attention. arxiv preprint arxiv:1910.12977. qian zhang, han lu, hasim sak, anshuman tripathi, erik mcdermott, stephen koo, and shankar kumar. 2020. transformer transducer: a streamable speech recognition model with transformer encoders and rnn-t loss. in 2020 ieee international conference on acoustics, speech and signal processing, icassp 2020, barcelona, spain, may 4-8, 2020, pages 7829-7833. ieee. baigong zheng, renjie zheng, mingbo ma, and liang huang. 2019a. simultaneous translation with flexible policy via restricted imitation learning. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 5816-5822, florence, italy. association for computational linguistics. renjie zheng, mingbo ma, baigong zheng, and liang huang. 2019b. speculative beam search for simultaneous translation. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 1395-1402, hong kong, china. association for computational linguistics.