Neural Simultaneous Speech Translation Using Alignment-Based Chunking (2020)

Abstract: In simultaneous machine translation, the objective is to determine when to produce a partial translation given a continuous stream of source words, with a trade-off between latency and quality. We propose a neural machine translation (NMT) model that makes dynamic decisions when to continue feeding on input or generate output words. The model is composed of two main components: one to dynamically decide on ending a source chunk, and another that translates the consumed chunk. We train the components jointly and in a manner consistent with the inference conditions. To generate chunked training data, we propose a method that utilizes word alignment while also preserving enough context. We compare models with bidirectional and unidirectional encoders of different depths, both on real speech and text input. Our results on the IWSLT 2020 English-to-German task outperform a wait-k baseline by 2.6 to 3.7% BLEU absolute.

## introduction simultaneous machine translation is the task of generating partial translations before observing the entire source sentence. the task fits scenarios such as live captioning and speech-to-speech translation, where the user expects a translation before the speaker finishes the sentence. simultaneous mt has to balance between latency and translation quality. if more input is consumed before translation, quality is likely to improve due to increased context, but latency also increases. on the other hand, consuming limited input decreases latency, but degrades quality. there have been several approaches to solve simultaneous machine translation. in @xcite , a fixed policy is introduced to delay translation by a fixed number of words. alternatively, satija and pineau (2016); @xcite ; @xcite use reinforcement learning to learn a dynamic policy to determine whether to read or output words. @xcite adapt the decoding algorithm without relying on additional components. however, these methods do not modify the training of the underlying nmt model. instead, it is trained on full sentences. @xcite introduce a holistic framework that relaxes the hard notion of read/write decisions at training time, allowing it to be trained jointly with the rest of the nmt model. in this paper, we integrate a source chunk boundary detection component into a bidirectional recurrent nmt model. this component corresponds to segmentation or read/write decisions in the literature. it is however trained jointly with the rest of the nmt model. we propose an algorithm to chunk the training data based on automatically learned word alignment. the chunk boundaries are used as a training signal along with the parallel corpus. the main contributions of this work are as follows: in @xcite ; @xcite , a wait-k policy is proposed to delay the first target word until k source words are read. the model alternates between generating s target words and reading s source words, until the source words are exhausted. afterwards, the rest of the target words are generated. in addition, @xcite convert the training data into chunks of predetermined fixed size. in contrast, we train models that learn to produce dynamic context-dependent chunk lengths. the idea of exploiting word alignments to decide for necessary translation context can be found in several recent papers. @xcite train an agent to imitate read/write decisions derived from word alignments. in our architecture such a separate agent model is replaced by a simple additional output of the encoder. @xcite use word alignments to tune a pretrained language representation model to perform word sequence chunking. in contrast, our approach integrates alignmentbased chunking into the translation model itself, avoiding the overhead of having a separate component and the need for a pretrained model. moreover, in this work we improve on pure alignmentbased chunks using language models (section 6.3) to avoid leaving relevant future source words out of the chunk. press and smith (2018) insert -tokens into the target using word alignments to develop an nmt model without an attention mechanism. those tokens fulfill a similar purpose to wait decisions in simultaneous mt policies. @xcite propose an attentionbased model that integrates an additional monotonic attention component. while the motivation is to use hard attention to select the encoder state at the end of the source chunk, they avoid using discrete attention to keep the model differentiable, and use soft probabilities instead. the hard mode is only used during decoding. we do not have to work around discrete decisions in this work, since the chunk boundaries are computed offline before training, resulting in a simpler model architecture. ## simultaneous machine translation the problem of offline machine translation is focused on finding the target sequence e i 1 = e 1 ...e i of length i given the source sequence f j 1 of length j. in contrast, simultaneous mt does not necessarily require the full source input to generate the target output. in this work, we formulate the problem by assuming latent monotonic chunking underlying the source and target sequences. formally, let s k 1 = s 1 ...s k ...s k denote the chunking sequence of k chunks, such that s @xmath0, j k ), where i k denotes the target position of last target word in the k-th chunk, and j k denotes the source position of the last source word in the chunk. since the source and target chunks are monotonic, the beginnings of the source and target chunks do not have to be defined explicitly. the chunk positions are subject to the following constraints: we use ẽk = e i k-1 +1 ...e i k to denote the k-th target chunk, and @xmath1...f j k to denote its corresponding source chunk. the target sequence e i 1 can be rewritten as ẽk 1 , similarly, the source sequence can be rewritten as f j 1 = f k 1 . we introduce the chunk sequence s k 1 as a latent variable as follows: in this work, we drop the marginalization sum over chunk sequences and use fixed chunks during training. the chunk sequences are generated as described in section 6. ## search decoding in simultaneous mt can be seen as an asynchronous process that takes a stream of source words as input and produces a stream of target words as output. in our approach, we segment the incoming source stream into chunks and output a translation for each chunk individually, however always keeping the full source and target context. algorithm 1: simultaneous decoding lists in bold, [] is the empty list, += appends to a list input : source word stream f j 1 output : target word stream e algorithm 1 explains the simultaneous decoding process. one source word f j (i.e. its embedding fj ) is read at a time. we calculate the next step of the shared forward encoder (equation 9 ), including source boundary detection (equation 10 ). if the boundary probability p(b j ) is below a certain threshold t b , we continue reading the next source word f j+1 . if, however, a chunk boundary is detected, we first feed all word embeddings fk of the current chunk into the backward encoder (equation 11 ), resulting in representations ←h k for each of the words in the current chunk. after that, the decoder is run according to equations 12-18. note, that it attends to representations -→ h and ←h of all source words read so far, not only the current chunk. here, we perform beam search such that in each decoding step those combinations of target words and target chunk boundary decisions are kept that have the highest joint probability. a hypothesis is considered final as soon as it reaches a position i where a chunk boundary b @xmath2. note that the length of a chunk translation is not restricted and hypotheses of different lengths compete. when all hypotheses in the beam are final, the first-best hypothesis is declared as the translation ẽk of the current chunk and all its words are flushed into the output stream at once. during search, the internal states of the forward encoder and the decoder are saved between consecutive different calls while the backward decoder is initialized with a zero state for each chunk. ## streaming speech recognition to translate directly from speech signal, we use a cascaded approach. the proposed simultaneous nmt system consumes words from a streaming automatic speech recognition (asr) system. this system is based on a hybrid lstm/hmm acoustic model @xcite , trained on a total of approx. 2300 hours of transcribed english speech from the corpora allowed by iwslt 2020 evaluation, including must-c, ted-lium, and lib-rispeech. the acoustic model takes 80-dim. mfcc features as input and estimates state posterior probabilities for 5000 tied triphone states. it consists of 4 bidirectional layers with 512 lstm units for each direction. frame-level alignment and state tying were bootstrapped with a gaussian mixtures acoustic model. the lm of the streaming recognizer is a 4-gram count model trained with kneser-ney smoothing on english text data (approx. 2.8b running words) allowed by the iwslt 2020 evaluation. the vocabulary consists of 152k words and the out-of-vocabulary rate is below 1%. acoustic training and the hmm decoding were performed with the rwth asr toolkit @xcite . the streaming recognizer implements a version of chunked processing @xcite which allows for the same blstmbased acoustic model to be used in both offline and online applications. by default, the recognizer updates the current first-best hypothesis by viterbi decoding starting from the most recent frame and returns the resulting word sequence to the client. this makes the first-best hypothesis "unstable", i.e. past words can change depending on the newly received evidence due to the global optimization of the viterbi decoding. to make the output more stable, we made the decoder delay the recognition results until all active word sequences share a common prefix. this prefix is then guaranteed to remain unchanged independent of the rest of the utterance and thus can be sent out to the mt model. ## experiments we conduct experiments on the iwslt simultaneous translation task for speech translation of ted talks from english to german. ## conclusion we proposed a novel neural model architecture for simultaneous mt that incorporates a component for splitting the incoming source stream into translatable chunks. we presented how we generate training examples for such chunks from statistical word alignment and how those can be improved via language models. experiments on the iwslt 2020 english-to-german task proved that the proposed learned source chunking outperforms a fixed wait-k strategy by a large margin. we also investigated the value of backwards source encoding in the context of simultaneous mt by comparing uniand bidirectional versions of our architecture.