Don’t Discard Fixed-Window Audio Segmentation in Speech-to-Text Translation (2022)

Abstract: For real-life applications, it is crucial that end-to-end spoken language translation models perform well on continuous audio, without relying on human-supplied segmentation. For online spoken language translation, where models need to start translating before the full utterance is spoken,most previous work has ignored the segmentation problem. In this paper, we compare various methods for improving models’ robustness towards segmentation errors and different segmentation strategies in both offline and online settings and report results on translation quality, flicker and delay. Our findings on five different language pairs show that a simple fixed-window audio segmentation can perform surprisingly well given the right conditions.

## introduction end-to-end spoken language translation (slt) has seen considerable advances in recent years. to apply these findings to real online and offline slt settings, we need to be able to process continuous audio input. however, most previous work on end-to-end slt makes use of human-annotated, sentence-like gold segments both at training and test time which are not available in real-life settings. unfortunately, slt models that were trained on such gold segments often suffer a noticeable quality loss when applied to artificially split audio segments @xcite . this also highlights that a good segmentation is more important for slt than for automatic speech recognition (asr) because we need to split the audio into "translatable units". for a cascade system, a segmenter/punctuator can be inserted between the asr and machine translation (mt) model @xcite in order to create suitable segments for the mt model. however for end-to1 we publicly release our code and model outputs here: https://github.com/zurichnlp/window_ audio_segmentation end slt systems, it is still not clear how to best translate continuous input. solving this problem is very much an active research field that has mainly been tackled from two sides: (1) improving slt models to be more robust towards segmentation errors @xcite @xcite and (2) developing strategies to split streaming audio into segments that resemble the training data more closely @xcite . both types of approaches were successfully used in recent years for the iwslt offline slt shared task @xcite @xcite to translate audio without gold segmentations. however, they have not yet been tested systematically in the online slt setup where translation starts before the full utterance is spoken. recent editions of the iwslt simultaneous speech translation shared task focused more on evaluation using the gold segmentation rather than unsegmented audio @xcite . segmenting streaming audio is especially interesting in online slt because aside from effects on translation quality, different segmentations can also influence the delay (or latency) of the generated translation. in this paper, we aim to fill this gap and focus on the end-to-end online slt setup. we suspect that there is an interplay between more robust models and better segmentation strategies and that an isolated comparison may not be informative enough. consequently, we explore different combinations of these two approaches for two different slt models and present results in five language pairs. figure 1 shows the four segmentation methods we study in this work (see also section 3.3). our experiments follow the popular retranslation approach @xcite @xcite where a partial segment is retranslated every time new audio becomes available. retranslation has the advantage of being a simple approach to online slt, which can use a standard mt inference engine. as a side-effect, the previous translation can change in later retranslations and the resulting "flicker" (i.e. sudden translation changes in the output of previous time steps) is also considered in our evaluation of different strategies. our main contributions are: ## related work in recent years, the iwslt shared task organisers have stopped providing gold segmented test sets for the offline speech translation task which has lead to increased research focus on audio segmentation @xcite @xcite . one obvious strategy to segment audio is to create fixed windows of the same duration, but previous research has mostly relied on more elaborate methods. typically, methods with voice activity detection (vad) @xcite were employed to identify natural breaks in the speech signal. however, vad models do not guarantee breaks that align with complete utterances and can produce segments that are too long or too short which is why hybrid approaches that also consider the length of the predicted utterance can be helpful @xcite @xcite . most recently, tsiamas et al. (2022b) finetune a wav2vec 2.0 model @xcite to predict gold segmentationlike utterance boundaries, an approach which outperforms several alternative segmentation methods and was widely adopted in the 2022 iwslt offline slt shared task @xcite @xcite . apart from improving automatic audio segmentation methods, previous research has also focused on making slt models more robust toward segmentation errors. @xcite and @xcite both explore context-aware end-toend slt models and show that context can help to better translate vad-segmented utterances. similarly, training on artificially truncated data can be beneficial to segmentation robustness in cascaded setups @xcite but also in end-to-end models @xcite . while this approach can introduce misalignments between source audio and target text, such misalignments in the training data are not necessarily harmful to slt models as @xcite recently showed in an evaluation of the must-c dataset @xcite . both of these approaches -improving automatic segmentation and making models more robust toward segmentation errors -can be combined. for example, @xcite show that continued finetuning on artificial segmentation can help narrow the gap between hybrid segmentation approaches and manual segmentation. however, a combination of both methods is not always equally beneficial. @xcite repeat @xcite 's analysis with the segmentation model proposed by @xcite and show that for this segmentation strategy, continued finetuning on resegmented data does not lead to an improvement in translation quality. in our work, we aim to extend these efforts and test various combinations of segmentation and model finetuning strategies. we are especially interested in fixed-window segmentations which have largely been ignored in slt research but are attractive from a practical point of view because they do not require an additional model to perform segmentation. to the best of our knowledge, we are the first to perform such an extensive segmentationfocused analysis for online slt, considering delay, flicker and translation quality for the evaluation. ## discussion based on our results in section 4, we believe that fixed-window segmentation should not be disregarded in future slt research on unsegmented audio. given the right setup with a context-aware model and a merging window algorithm, this segmentation can outperform current state-of-the-art automatic segmentation models and in some cases even the gold segmentation in terms of translation quality. moreover, in an online slt setup, a fixed-window approach brings the additional benefit that no dedicated segmentation model needs to be loaded at inference time and run every time new audio becomes available. while there is currently no solution to bring flicker down to biased beam search levels without hurting quality (see appendix b) or increasing delay (see appendix c), this should not be a reason to disregard fixed-window segmentation as it opens exciting opportunities for future research. ## conclusion in this paper, we explored several combinations of segmentation-robust finetuning and different automatic segmentation strategies in an online slt setup. we focus on a retranslation-based approach to slt and we run experiments on five different language pairs based on two different slt models. considering the evaluation of translation quality, flicker and delay, we discuss several issues that arise when comparing different segmentations and propose a fix to an existing toolkit for evaluating delay. our results show that a simple fixed-window segmentation can perform surprisingly well if an algorithm is used for merging overlapping windows and a context-aware slt model is used. in terms of translation quality, this segmentation performs comparably to shas -the current state-of-the-art segmentation method -and in some cases even outperforms the gold segmentation, showing potential for future application to offline slt. in terms of flicker and delay, the results of the merging windows approach are comparable to the other segmentations if biased beam search is not enabled but future work is needed to reduce flicker in the merging windows approach to similar levels as biased beam search for other strategies without hurting translation quality.