From Simultaneous to Streaming Machine Translation by Leveraging Streaming History (2022)

Abstract: Simultaneous Machine Translation is the task of incrementally translating an input sentence before it is fully available. Currently, simultaneous translation is carried out by translating each sentence independently of the previously translated text. More generally, Streaming MT can be understood as an extension of Simultaneous MT to the incremental translation of a continuous input text stream. In this work, a state-of-the-art simultaneous sentence-level MT system is extended to the streaming setup by leveraging the streaming history. Extensive empirical results are reported on IWSLT Translation Tasks, showing that leveraging the streaming history leads to significant quality gains. In particular, the proposed system proves to compare favorably to the best performing systems.

## introduction simultaneous machine translation (mt) is the task of incrementally translating an input sentence before it is fully available. indeed, simultaneous mt can be naturally understood in the scenario of translating a text stream as a result of an upstream automatic speech recognition (asr) process. this setup defines a simultaneous speech translation (st) scenario that is gaining momentum due to the vast number of industry applications that could be exploited based on this technology, from person-toperson communication to subtitling of audiovisual content, just to mention two main applications. these real-world streaming applications motivate us to move from simultaneous to streaming mt, understanding streaming mt as the task of simultaneously translating a potentially unbounded and unsegmented text stream. streaming mt poses two main additional challenges over simultaneous mt. first, the mt system must be able to leverage the streaming history beyond the sentence level both at training and inference time. second, the system must work under latency constraints over the entire stream. with regard to exploiting streaming history, or more generally sentence context, it is worth mentioning the significant amount of previous work in offline mt at sentence level @xcite , document level @xcite @xcite @xcite , and in related areas such as language modelling @xcite that has proved to lead to quality gains. also, as reported in @xcite , more robust st systems can be trained by taking advantage of the context across sentence boundaries using a data augmentation strategy similar to the prefix training methods proposed in @xcite . this data augmentation strategy was suspected to boost re-translation performance when compared to conventional simultaneous mt systems @xcite . nonetheless, with the notable exception of @xcite , sentences in simultaneous mt are still translated independently from each other ignoring the streaming history. @xcite proposed an end-to-end streaming mt model with a transformer architecture based on an adaptive computation time method with a monotonic encoderdecoder attention. this model successfully uses the streaming history and a relative attention mechanism inspired by transformer-xl @xcite . indeed, this is an mt model that sequentially translates the input stream without the need for a segmentation model. however, it is hard to interpret the latency of their streaming mt model because the authors observe that the current sentence-level latency measures, average proportion (ap) @xcite , average lagging (al) @xcite and differentiable average lagging (dal) @xcite do not perform well on a streaming setup. this fact is closely related to the second challenge mentioned above, which is that the system must work under latency constraints over the entire stream. indeed, current sentence-level latency measures do not allow us to appropriately gauge the latency of streaming mt systems. to this purpose, @xcite recently proposed a stream-level adaptation of the sentence-level latency measures based on the conventional re-segmentation approach applied to the st output in order to evaluate translation quality @xcite . in this work, the simultaneous mt model based on a unidirectional encoder-decoder and training along multiple wait-k paths proposed by @xcite is evolved into a streamingready simultaneous mt model. to achieve this, model training is performed following a sentenceboundary sliding-window strategy over the parallel stream that exploits the idea of prefix training, while inference is carried out in a single forward pass on the source stream that is segmented by a direct segmentation (ds) model @xcite . in addition, a refinement of the unidirectional encoder-decoder that takes advantage of longer context for encoding the initial positions of the streaming mt process is proposed. this streaming mt system is thoroughly assessed on iwslt translation tasks to show how leveraging the streaming history provides systematic and significant bleu improvements over the baseline, while reported stream-adapted latency measures are fully consistent and interpretable. finally, our system favourably compares in terms of translation quality and latency to the latest state-of-the-art simultaneous mt systems @xcite . this paper is organized as follows. next section provides a formal framework for streaming mt to accommodate streaming history in simultaneous mt. section 3 presents the streaming experimental setup whose results are reported and discussed in section 4. finally, conclusions and future work are drawn in section 5. ## streaming mt in streaming mt, the source stream x to be translated into y comes as an unsegmented and unbounded sequence of tokens. in this setup, the decoding process usually takes the greedy decision of which token appears next at the i-th position of the translation being generated nevertheless, for practical purposes, the concept of sentence segmentation is usually introduced to explicitly indicate a monotonic alignment between source and target sentences in streaming mt. let us consider for this purpose the random variables a and b for the source and target segmentation of the stream, respectively. variables a and b can be understood as two vectors of equal length denoting that the n-th source sentence starts at position a n , while the n-th target sentence does so at position b n . in the next sections, we reformulate simultaneous mt in terms of the more general framework of streaming mt. this reformulation allows us to consider opportunities for improvement of previous simultaneous mt models. ## evaluation figure 2 reports the evolution of bleu scores on the german-english iwslt 2010 dev set as a function of the k value in the wait-k policy for a range of streaming history lengths (@xmath0, 20, 40, 60, 80}). we show results for the 3 encoders introduced previously. history lengths were selected taking into account that the average sentence length is 20 tokens. a history length of zero (@xmath1. the bleu scores for the offline mt systems with a bidirectional encoder are also reported using horizontal lines, in order to serve as reference values. we report offline results for @xmath2, @xmath3. all systems used the reference segmentation during decoding. as observed, bleu scores of the simultaneous mt systems leveraging on the streaming history (@xmath4. indeed, as the streaming history increases, bleu scores also do reaching what it seems the optimal history length at @xmath5. as expected, when replacing the unidirectional encoder by the pbe, bleu scores improve as the wait-k value increases, since pbe has additional access to those tokens from j + 1 up to k. for instance, for @xmath6, pbe is 0.7 bleu points above the unidirectional encoder. on the other hand, it can be observed how using an encoder which is not fully bidirectional during training, creates a performance gap with respect to the offline bidirectional model when carrying out inference in an offline manner (@xmath7. it can be also observed how the pbe model is better prepared for this scenario and shows a smaller gap. it is important to keep in mind that although both offline and pbe models behave the same way during inference for a large enough k, during training time the pbe model, trained using the multi-k with k randomly sampled for each batch, has been optimized jointly for low, medium and high latencies. in general, the bidirectional encoder shows poor performance for simultaneous mt. this can be explained by the fact that there exists a mismatch between the training condition (whole source available) and the inference condition (only a prefix of the source is available for @xmath8. these results are consistent with @xcite . keep in mind that this bidirectional model is different from the offline one because it has been subject to the constraints of eq. 7 during training. as a result of the bleu scores reported in figure 2 , the streaming mt system with @xmath9. following (schneider and waibel, 2020)'s setup, the test set is lowercased and concatenated into a single stream. in order to measure the latency of the pipeline defined by the segmenter followed by mt system, it is necessary to take into account not only the latency of the mt system but also that of the segmenter. thankfully this is straightforward to do in our pipeline, as a segmenter with a future window of length w modifies the pipeline policy so that, at the start of the stream, w read actions are carried out to fill up the future window. then, every time the mt system carries out a read action, it receives one token from the segmenter. thus, the integration of the segmenter into the pipeline is transparent from a latency viewpoint. figure 3 shows bleu scores versus streamadapted al and dal (s @xmath10.85) figures reported with segmenters of future window length @xmath11, 1, 2, 3, 4} for a streaming evaluation on the iwslt 2010 test set. points over each curve correspond to @xmath12, 2, 4, 8, 16} values of the wait-k policy used at inference time. results for a @xmath13. as shown, stream-adapted al and dal figures achieved by our streaming mt system are reasonable, lagging 2-10 tokens behind the speaker for nearly maximum bleu scores with a best bleu score of 29.5 points. the same happens with ap figures ranging from 0.6 for @xmath14.3 for @xmath15. these figures highlight the advantages of tying together our translation policy with the sentence segmentation provided by the ds model. every time the ds model emits an end-of-sentence event, the mt model is forced to catch-up and translate the entire input. in this way, the mt model never strays too far from the speaker, even if the source-target length ratio differs from the γ defined at inference time. see appendix a for streaming translation results in the reverse direction (@xmath16. next, we compare our proposed streaming mt (str-mt) model with the λ = 0.3 act system (schneider and waibel, 2020) in terms of bleu score and stream-adapted latency measures on on-trac and our streaming mt system exhibit a similar progression, which is to be expected given that they are both based on the multi-k approach. however, our system consistently outperforms the on-trac system by 1-2 bleu. this confirms the importance of utilizing streaming history in order to significantly improve results, and how the proposed pbe model can take better advantage of the history. rwth and kit systems are closer in translation quality to our proposal than on-trac, for al between 5 and 7. however, these systems do not show a flexible latency policy and are not comparable to our system at other regimes. indeed, for that to be possible, these systems need to be re-trained, in contrast to our system in which latency is adjusted at inference time. ## conclusions in this work, a formalization of streaming mt as a generalization of simultaneous mt has been proposed in order to define a theoretical framework in which our two contributions have been made. on the one hand, we successfully leverage streaming history across sentence boundaries for a simultaneous mt system based on multiple wait-k paths that allows our system to greatly improve the results of the sentence-level baseline. on the other hand, our pbe is able to take into account longer context information than its unidirectional counterpart, while keeping the same training efficiency. our proposed mt system has been evaluated under a realistic streaming setting being able to reach similar translation quality than a state-of-theart segmentation-free streaming mt system at a fraction of its latency. additionally, our system has been shown to be competitive when compared with state-of-the-art simultaneous mt systems optimized for sentence-level translation, obtaining excellent results using a single model across a wide range of latency levels, thanks to its flexible inference policy. in terms of future work, additional training and inference procedures that take advantage of the streaming history in streaming mt are still open for research. one important avenue of improvement is to devise more robust training methods, so that simultaneous models can perform as well as their offline counterparts when carrying out inference at higher latencies. the segmentation model, though proved useful in a streaming setup, adds complexity and can greatly affect translation quality. thus, the development of segmentation-free streaming mt models is another interesting research topic.