Segmentation-Free Streaming Machine Translation (2024)

Abstract: Streaming Machine Translation (MT) is the task of translating an unbounded input text stream in real-time. The traditional cascade approach, which combines an Automatic Speech Recognition (ASR) and an MT system, relies on an intermediate segmentation step which splits the transcription stream into sentence-like units. However, the incorporation of a hard segmentation constrains the MT system and is a source of errors. This paper proposes a Segmentation-Free framework that enables the model to translate an unsegmented source stream by delaying the segmentation decision until after the translation has been generated. Extensive experiments show how the proposed Segmentation-Free framework has better quality-latency trade-off than competing approaches that use an independent segmentation model.1

## introduction streaming machine translation (str-mt) is a specific task of machine translation (mt) that consists in translating an unbounded input text stream in real-time. str-mt systems are typically used in a cascaded setting following a streaming automatic speech recognition (asr) system. this task has many applications for scenarios such as live broadcasting, parliamentary debates, live lectures, etc. where the input speech to be translated is potentially several hours long, and the system must provide accurate and real-time translations over the live session. however, conventional mt systems are not well prepared to work in the conditions described above. training samples for conventional mt systems are sentence-aligned pairs, so there is a length mismatch between the training (a few hundred tokens at most) and inference conditions (thousands of tokens for live sessions). conventionally, some sort of segmentation model is used in order to split the incoming text stream into sentence-like units, so that they can be translated by the mt system. each sentence-like unit, or segment, is typically translated in isolation, although techniques from document mt can be used to provide additional context to a conventional mt model beyond the current sentence @xcite @xcite @xcite @xcite . these techniques can be adapted to the streaming case using the concept of streaming history @xcite , which keeps a limited context of the previously seen source segments and their corresponding translations generated by the mt model. we collectively refer to all approaches that use an independent upstream segmenter model as the segmented setting. the main downside of the segmented setting is that the translation quality is very dependant on the quality of the segmenter, and forcing a hard decision without involving the mt system conditions the resulting translation quality. this is particularly relevant in scenarios where we have a downstream system. on this line of work, our final goal is to build a streaming speech-to-speech translation system using an additional downstream streaming tts model. the decisions of the tts model cannot be changed once the output has been sent to the user, and therefore it is not possible to change the output of the segmenter/mt systems. this paper proposes a segmentation-free (segfree) approach that does not rely on an upstream segmenter. instead, the mt model receives an unsegmented stream of source text and generates a continuous sequence of translated words. the segfree model jointly generates the translation and its target segmentation by inserting a special token ('' @xcite '') into the translation stream. whereas in the segmented setting, the segmentation decision is incorporated into the source side independently from the mt model, in the segfree setting this decision is taken by the mt model considering both, the source and target streams. that is, the segmentation decision has been moved from an upstream segmenter into the target translation stream. delaying this segmentation decision allows the segfree model to take into account additional target-side information, which is the crucial component that enables the segfree model to significantly outperform its segmented counterpart. the rest of this article is organized as follows. first, the related work is described in section 2. then, section 3 defines the statistical framework of our proposed segmentation-free model. next, section 4 introduces the datasets involved, how they were processed and the instantiation of the models. lastly, section 5 reports the results achieved and conclusions are drawn in section 6. ## related work the problem of automatic segmentation has limited the mt community for many years, and many solutions have been proposed, ranging from simple length-based heuristics @xcite , using language model probabilities @xcite , to introducing segmentation with a monolingual mt system @xcite @xcite . @xcite propose a data augmentation technique that introduces segmentation errors during training in order to make the model more robust to this type of errors. mt of unsegmented inputs has received relatively little attention. @xcite propose a stream decoding algorithm for phrase-based statistical mt, which is able to translate unsegmented input by using a continuous translation lattice that is updated during the translation process. new input words extend the lattice, and the output is committed whenever a predefined latency is exceeded. in the case of neural mt systems, schneider and waibel (2020) use a transformer-xl @xcite encoder for longer context, combined with a monotonic encoder-decoder attention head. the model translates unsegmented input text using a rolling window over the source stream with a fixed offset. their training procedure is a multi-stage method that involves multiple training phases. when translating in a streaming setup, the latency of the model is quite significant when compared with the speaker @xcite . more recently, @xcite propose a method for the translation of unsegmented input by using a small window over the source stream that is re-translated each time a new input token is received. the overlapping translations of each window are then merged together to form the output stream. the downside of this approach is that it introduces flickering into the output, so it is difficult to apply to setups such as speech-to-speech translation that require non-flickering output. with regard to the translation models themselves, a distinction can be made between those that use fixed policies, policies which are pre-defined and do not depend on the current content being translated, and adaptive policies, which are those that adapt their decision based on the current translation status. wait-k @xcite is the most popular fixed translation policy. this policy first waits for k source words to arrive, and then alternates between writing a new target word and waiting for a new source word. adaptive policies are much more varied, although they can be classified along some general trends. one popular approach is to try to first detect meaningful units or chunks that must be translated together, and only generate a translation once an entire chunk has been received @xcite @xcite @xcite . in contrast, in some approaches the policy is derived from the model itself, either from the output probabilities @xcite @xcite or from some internal state of the model @xcite . ## segmentation-free statistical framework under the proposed segfree framework, the translation system receives an unsegmented, continuous stream of source words, and produces a translation stream in a real-time fashion. unlike in the segmented setting, the system is not constrained by the pre-existing segmentation, so it decides how to delimit the segments of the output stream by taking into account both source and target information. moreover, a significant advantage of this approach is that it removes and target (bottom) streams. in the first row (@xmath0, the mt system has just generated the last target word y @xmath1. in the second row (@xmath2, the translation model generates the ''[sep]'' token, which indicates the end of a target segment. at that point, the memory mechanism is activated and decides â = j + 5 with x â = ''that'', so "there is no doubt about that" is moved to the streaming history along with the current translation. in the third row, the translation continues, but the streaming history has grown too large, so the memory mechanism discards the oldest chunk. the dependency on the intermediate segmentation step, which is an additional component that needs to be trained, as well as being a source of cascaded errors. a naive approach to segfree translation consists in using a sliding window over the source and target streams, which is moved/updated following a fixed schedule. for instance, every time a new source word is received, it is added to the source window and the oldest source word is discarded. however, @xcite show that the target-to-source ratio of the words generated by an mt system is not constant during the translation of a source stream, and a system using this approach will end up with source and target windows whose content is out of sync, as a result of having over or under-estimated the writing rate of the system. our proposed segfree system solves this issue by replacing the fixed update schedule of the sliding windows by a memory mechanism. this mechanism keeps track of which parts of the source stream have already been translated, along with the associated translations. in addition, it manages the streaming history by discarding the oldest, already translated words of the stream, which can be forgotten without affecting the current translation. as a result, each of the sliding windows of the segfree model contains two disjoint chunks: a chunk of history words, which has been fully processed, and can therefore be discarded in the future, and the active chunk that needs to be translated. once the maximum capacity of the streaming history has been reached, the oldest source part and its corresponding translation are discarded. the proposed memory mechanism uses a probabilistic model in order to decide which part of the source stream has already been translated and should be moved to the streaming history together with its translation. formally, let @xmath3, x 2 , . . . , x j } be the source stream and @xmath4, y 2 , . . . , y i }, the target stream. let x j j be a chunk of active source words and ŷi i a partial translation of that chunk. every time the translation model generates the end-of-segment token (''[sep]''), the memory mechanism is invoked and selects a source position â ∈ [j, j ] based on the current status of the translation. after this decision, x â j and ŷi i will be moved to the streaming history, and the translation will continue with x j â+1 . this is graphically shown in figure 1 . for this work, the memory mechanism takes the current active source and target chunks, @xmath5, and uses a log-linear model composed by a series of feature functions h f (a, x, ŷ) which provide a score for every position a of the active source chunk. the probability of position a being the last position that has been translated is estimated as after applying the log-transformation, the position of the last source word â to be moved to the streaming history is chosen as the weights of the feature functions, λ ∈ r f , are optimized using gradient descent by minimizing the conventional cross-entropy loss over a set of samples. specifically, each sentence pair can be understood as a classification sample for a task with classes @xmath6, 2, . . . , |x|} and correct label ĉ = |x|. the abbreviated pseudocode for segfree inference is presented in figure 2 . the segfree model presented in this work uses a memory mechanism based on the following feature functions foot_1 : the score for each source position is given by a reverse translation model that computes the probability of the partial translation x a j followed by the end-of-sentence symbol (''</@xmath7. that is, h f (a, x, ŷ) = p @xmath8, </@xmath9, computed as the product of token-level probabilities @xmath10. for this work, our reverse model uses the same architecture and training data as the forward model, but the translation direction has been switched during training. apart from the two aforementioned features, we also tried some other approaches that were not able to improve the results, either in isolation or in combination with other features. specifically, we tried predicting both the mean and the variance rather than only the mean with the linear model as well as replacing the linear models with higher order models. furthermore, we also collected counts of source lengths for every different target length and used them to estimate the segmentation probability, as well as using the source length, target length and length ratios as features. lastly, we also tried a neural-based regressor that predicts the mean and variance. additionally, we also compare our log-linear approach with the naive segfree approach (naive), which uses a sliding window with a fixed offset r. the median target-to-source length ratio is used as the offset. then, during inference time, whenever the ''[sep]'' token is emitted, the source sliding window is moved a fixed number of positions based on this pre-computed ratio. specifically, â = max( |y|/r , 0). results are presented for four translation directions: english to german, spanish, and french, and german to english. the models involving german were trained using datasets available for the iwslt 2022 shared tasks @xcite . the specific datasets are reported in in order to enable simultaneous translation, the prefix-training data augmentation technique @xcite ) is used. one partial translation pair is generated for each sentence pair in the original corpus (which already includes the streaming history) by randomly selecting a partial prefix of both, source and target sentences. if a given sentence pair contains streaming history information, the streaming history is left unchanged, and prefix generation is only applied to the current sentence pair. the model is trained on the concatenation of both, the original training data and the partial prefix data. figure 3 shows a graphical overview of how the training data was constructed. the source side of the dataset is lowercased and punctuation marks are removed in order to simulate the output of a streaming asr system. sentencepiece @xcite is used to learn 50k subword units. the sentence-piece whitespace meta symbol '' '' is used as a suffix instead of a prefix, so that a full word can be written once its last subword has been written, without having to wait for the model to generate the next subword. both segmented and segfree systems are trained exactly with the same data, with the only difference between the two being that the training data of the segfree model has been processed to remove ''[sep]'' tokens from the source side of the data, in order to mimic the inference condition in which no end-of-chunk information is available. the baseline segmenters and the segfree feature functions were trained and the availability of document boundaries, which used for constructing samples with streaming history. note that europarl @xcite is excluded from the training data, in order to avoid overlap with europarl-st. with the must-c v2 train set. the feature function weights were then optimized with the samples of the must-c v2 dev set. ## results segfree and ds-based models follow a wait-k translation policy. we report 10 results for each system, one for each @xmath11, 10], in order to explore the latency-quality tradeoff. each video belonging to the evaluation set is translated independently from the other videos in the set. because both the ds and the segfree approaches create their own segmentation that does not match the reference one, the hypotheses are re-aligned with the reference translation using minimum edit distance @xcite before computing the quality measure bleu @xcite . 3 likewise, stream-level latency @xcite is computed using minimum edit distance so that both approaches can be compared. the average of the average lagging (al) @xcite value of each individual video is reported. the quality-latency tradeoff of the window-retrans approach is controlled using two hyperparameters: w, which is the size of the window that is re-translated at each step, and r, the match threshold that needs to be reached by a hypothesis to be considered a match. similarly to @xcite , we test @xmath12, 12, 16, 20} and @xmath13.1, 0.2, . . . , 0.7}. windowretrans does have flickering, unlike the other systems. we follow the conventional practice of evaluating on the final text @xcite ) and disregard any flickering for quality evaluation. this enables us to compare the windowretrans system with the other proposed systems, assuming an ideal situation in which flickering can be safely discarded. however, in practice we would not be able to do this, as the output of the downstream tts system cannot be changed once it has been received by the listener. figure 4 shows bleu vs. al of segfree systems when using two different combinations of feature functions (reverse-mt and reverse-mt + linreg) compared with the naive segfree system, evaluated on the english to german europarl-st dev set. unsurprisingly, the naive approach underperforms the other two systems. the use of a fixed offset in the naive approach is a limiting factor for the translation quality, as both the source and target streams are assumed to progress at the same rate, irregardless of their actual content. every time a target sentence is produced, a fixed number of source words are considered to have been translated. this means that on some occasions the actual writing rate may be underestimated, and on other occasions it may be overestimated. in this case, the results suggest that the writing rate might have been underestimated, which in turn causes high latency even for low values of k. in contrast, our proposed segfree system with reverse-mt feature works significantly better than the naive baseline, because it can dynamically update the streaming history based on the source and target streams, instead of being constrained by a fixed rate. thus, if a source chunk containing many high fertility words is translated, the system can take this into account when updating the streaming history. this avoids the problem of marking untranslated words as already translated, which is what would have happened in the naive approach. on top of this, combining the reverse-mt feature with the linear regression feature (reverse-mt + linreg) further improves the results, as the linear regression feature smooths the probabilities given by the reverse-mt model. based on this result, the reverse-mt + linreg system is selected for further experimentation. we test the previous hypothesis by taking the translations generated with the ds-oracle and feeding the memory mechanism of the naive and reverse-mt models with the appropriate source context. because the ds-oracle tells us which source words have actually been used to generate the translation, we can test if the memory mechanism is able to correctly identify these words. figure 5 shows the difference in length between the hypothesis generated by the memory mechanism and the ds-oracle. it can be observed how the segfree system is very good at detecting the correct position, except for some cases in where the length is underestimated. in contrast, the naive approach cannot adapt its prediction depending on the content of the actual translation, and as a result it performs significantly worse at selecting the right position to update the source stream. figure 6 shows a comparison between the segfree approach and the selected baselines on the english to german europarl-st dev set. the ds-roberta quality/latency trade-off is very dependant on the size of the future window @xmath14, 1, 2, 4}. it can be observed how @xmath15, but it quickly plateaus between 28 and 29 bleu points. the lack of a future window means that the segmentation decisions are less informed, and the translation quality does not greatly increase even if the translation model is given more context. moving from @xmath16, and the model is able to reach 31.7 bleu points. larger future window values (@xmath17, reaching a maximum of 32.3 and 33.3 bleu points, respectively, but the additional latency introduced by the segmenter does not make them competitive choices. this is consistent with the results of other works that use the ds segmenter @xcite . once the ds-roberta model has one or two future context tokens, it is better to allocate additional latency to the mt model in order to avoid diminishing returns. based on this, @xmath18. there is a gap of around 3 bleu points between the ds-roberta systems and the ds-oracle across all latency regimes. this gap illustrates the loss of performance incurred when using an imperfect segmentation, as well as the upper bound of performance that could be achieved using a perfect segmenter. the proposed segfree system clearly outperforms the ds-roberta systems at mid and high latencies, and performs similarly to the best ds-roberta system at low latencies. the segfree system achieves this quality improvement by having access to the original source stream and letting the mt system take the decision where the segment delimiter should be placed. moreover, the segfree system achieves these results consistently, whereas the ds approach needs multiple segmenters with different w in order to stay competitive. this highlights another advantage of moving beyond a segmenter system, as the latency of the translation only depends on the policy of the mt system. the windowretrans system is far behind the performance of both the segfree and the best ds-roberta system. the results for windowretrans with @xmath19. the configuration with @xmath20. after performing hyperparameter exploration on the europarl-st dev set, the ds-roberta, ds-oracle, windowretrans, and segfree systems were evaluated on the selected test sets. figure 7 reports bleu vs. al results, from left to right, on the english to german europarl-st and must-c test sets, and the german to english europarl-st test set. statistical significance tests using bootstrap resampling @xcite were conducted to test whether differences between systems were significant, with 1000 bootstrap resamples per test. each system was compared with each other system @xmath21.3 al. for the english to german europarl-st test set, the segfree outperforms the ds-roberta system by a wide margin. there is a gap of around 2 bleu points across all latency regimes, and this gap grows up to 2.6 bleu when comparing the best results (34.6 bleu points for segfree and 32.0 bleu points for ds-roberta). the segfree is 1.1 bleu points behind the ds-oracle system at medium latencies (al 5), and this difference decreases at higher latencies (0.5 bleu points for al 9). bleu differences across systems were statistically significant. on the must-c test set, both the segfree and the ds-roberta systems perform similarly at low and medium latencies. the segfree system does significantly outperform the ds-roberta system for @xmath22.6, reaching a maximumof30.1bleu points, whereas the ds-roberta system provides 28.8 bleu points. the ds-oracle is significantly better than both the segfree and ds systems. lastly, the german to english europarl-st test set results show that the segfree system significantly outperforms the ds-roberta system across all latency regimes. for example, there is a gap of 1.9 bleu points for al 4.5, and a gap of 1.1 bleu points for al 8.0. when comparing the ds-oracle and the segfree system, the ds-oracle is not significantly better for @xmath23, 6, 7}. the windowretrans system shows a similar trend to the one that was observed on the dev set. as the value of r is increased, so does the quality of the translation and the latency. the quality plateaus when @xmath24.5 or @xmath25.6 is reached, and further increases on r tend to degrade the performance. the difference in quality between this approach and the ds-roberta model is statistically significant. figure 9 reports the results of the different translation systems when evaluated with the bleurt-20 @xcite neural measure. we observe no significant differences when compared with the evaluation carried out using bleu. figure 10 reports the results for the english to french system, evaluated on the europarl-st and must-c test sets. the results show a similar pattern on both test sets: the ds-roberta system outperforms the windowretrans baseline across all latency ranges, and is in turn surpassed by the proposed segfree system. a small gap remains between the segfree system and the ds-oracle, and this gap is not statistically significant at some latency conditions. for al 9, there is a gap of 0.4 bleu between the ds-oracle and the segfree system on the europarl-st test, and a gap of 3.2 bleu between the segfree system and the ds-roberta system. bleu scores for windowretrans were too low (32.8) and omitted in figure 10 for the sake of clarity. for the must-c test, there is a larger gap of 1.7 bleu between the ds-oracle and the segfree system, and a gap of 2.9 bleu between the segfree and the ds-roberta system. next, figure 11 reports the results for the english to spanish system. the results follow a similar trend to previous test sets. for the europarl-st test (al 8), there is a gap of of 1.7 bleu between ds-oracle and segfree systems, and a gap of 4.0 between the segfree and the ds-roberta systems. the windowretrans result is 1.2 bleu lower than that of ds-roberta. for the must-c test, these gaps are 1.7, 2.6 and 2.8 bleu, respectively. figures 12 and 13 report bleurt curves rather than bleu for the same datasets and languages pairs. the english to french bleurt results are shown in figure 12 , whereas the english to spanish results are shown in figure 13 . as in previous cases, there are no relevant changes regarding system ordering or gaps between systems. similar conclusions are reached when evaluating with either of the measures, bleu or bleurt. ## conclusions this work introduces a novel segfree approach to str-mt that can directly translate an unbounded text stream without having to rely on an intermediate segmenter. this is achieved by letting the mt system decide where the segment delimiters are placed, and delaying this decision until the translation has been generated. in addition, a memory mechanism keeps track of which parts of the stream have already been translated, and can therefore be forgotten when needed, and which parts remain untranslated and must be kept. the segfree approach avoids the performance degradation introduced by a segmenter model, and is able to take into account additional information from both, source and target streams, when generating the output translation and segment delimiters. the experiments have shown how the segfree system is able to significantly outperform the competing ds-roberta approach across six of the seven test sets. furthermore, the segfree approach is able to match the performance of the oracle segmenter in the europarl-st german to english test set. these results validate the performance of the segfree approach across multiple domains and translation directions. more importantly, the segfree approach eliminates the need of an intermediate segmenter system in a cascaded system. as a result, the segfree approach is not only better in terms of quality, but it also lets the mt system retain full control over the translation policy. as a future work, the proposed segfree memory mechanism has been instantiated with a reverse-mt feature and a linear regression feature, but the generic formulation allows for any arbitrary feature function to be used. likewise, the segfree approach has been tested with static translation policies, but it could also be applied to a dynamic translation policy. the segfree approach opens the doors to further research that moves away from local, sentence level translation with limited context, into a fully-fledged contextual translation system augmented with a dynamic history that keeps the appropriate context. 1120 figure 13: bleurt vs. al on the english to spanish europarl-st (left) and must-c (right) test sets.