End-to-End Evaluation for Low-Latency Simultaneous Speech Translation (2023)

Abstract: The challenge of low-latency speech translation has recently draw significant interest in the research community as shown by several publications and shared tasks. Therefore, it is essential to evaluate these different approaches in realistic scenarios. However, currently only specific aspects of the systems are evaluated and often it is not possible to compare different approaches. In this work, we propose the first framework to perform and evaluate the various aspects of low-latency speech translation under realistic conditions. The evaluation is carried out in an end-to-end fashion. This includes the segmentation of the audio as well as the run-time of the different components. Secondly, we compare different approaches to low-latency speech translation using this framework. We evaluate models with the option to revise the output as well as methods with fixed output. Furthermore, we directly compare state-of-the-art cascaded as well as end-to-end systems. Finally, the framework allows to automatically evaluate the translation quality as well as latency and also provides a web interface to show the low-latency model outputs to the user.

## introduction in many applications scenarios for speech translation, the quality of the translations is not the only important metric, but it is also essential to provide the translation with a low latency. this is for example the case in translations of presentations or meetings. therefore, we observe an increasing interest in the field of low-latency speech translations, as shown by numerous published techniques and the organization of a dedicated shared task as part of the international conference on spoken language translations (iwslt) @xcite . in order to enable further progress in the field as well as a wide adoption of the technique a framework to evaluate different approaches is essential. however, the current evaluation only considers a limited number of aspects or techniques. in contrast, for an overall evaluation of different architectures (end-to-end and cascaded) and presentation style (revision and fixed) a general evaluation framework is needed. this should also consider the computational latency as well as the ability to process several sessions in parallel. motivated by this, we present a new framework to apply and evaluate low-latency, simultaneous speech translation. thereby we focus on a framework that can evaluate the different approaches in as realistic conditions as possible. the system is able to simulate different load conditions as well as compare systems using different design choices. finally, we also provide a web interface foot_0 to present the low-latency model outputs to the user. the main contributions of our paper are: in the next section, we describe the overall architecture of the framework. the two following sections explain the streaming algorithms for the speech and text processing components. after that, we illustrate how we evaluate our framework and then how the experimental setup looks like. in section 7 we present the results. then, we review the related work. at the end we describe the limitations and conclude our work. ## dynamic framework for low-latency speech translation motivated by previous work @xcite , we use a central mediator that coordinates the interaction of the different components (see figure 1 ). the user sends data to an api component which then sends the data to the mediator. the mediator forwards all arriving data to the corresponding component(s), e.g., the audio signal from the user to the speech processing component, the resulting transcripts to the text processing component and the output (through the api) to the user. in order to allow a flexible processing, for each session a graph dynamically defines how the data is sent to the different components. we process different requests at each component using the existing streaming framework kafka foot_2 . each component consists of a middleware and a backend with the processing separated into three steps: encoder decoder [m1] @xmath0: audio or text stable output [s] model forward model output [h] [m1, m2] s1 @xmath1, @xmath2, m2, m3] s2 @xmath3, we are able to share the backend and use batching of the requests. our framework supports two modes for lowlatency speech translation. first, a revision mode @xcite where the component (automatic speech recognition (asr) or machine translation (mt)) can send stable and unstable outputs. given more context at a later time step, the component can revise the unstable outputs. second, a fixed mode @xcite where the component is only allowed to send stable output. for fixed mode (and the revision mode of the asr component), the component needs to perform a stability detection (see sections 3 and 4 and figure 2 ), i.e., determine which parts of the output should be considered stable. note that for our streaming algorithms the backend models need to support prefix decoding, i.e., one can send a prefix which is then forced in the output. our framework is easily extendable by deploying additional backend models for different languages, adding new streaming algorithms in the middleware or adding custom components (e.g., speaker diarization as a preprocessing step before the asr) and including them in the session graph. ## low-latency speech processing the speech processing component receives a stream of audio packets and sends chunks of text (transcript or translation) to the mediator. for this two steps are run: input processing: first, a voice activity detec-tion generates a speech segment that can be extended when new packets of audio arrive. for this we use webrtc voice activity detector (wiseman, 2016). each audio frame (30ms) is classified if it contains speech or not. then a moving average is calculated. if it exceeds a certain threshold, a new segment is started. new audio is added to this segment until the moving average falls below a certain threshold and the segment ends. second, the backend model (asr or speech translation (st)) is run. if there exist speech segments that already ended, they are processed only once and the output is sent as stable text, other segments are constantly processed until they end. stability detection and output processing: we use the method local agreement two (la2) from @xcite . the intuition is that if the prefix of the output stays the same when adding more audio, the prefix should be considered stable. let c denote the chunk size hyperparameter (la2_chunk_size). the fixed mode works as follows (see figure 2 ): it waits until the segment contains (at least) c seconds of audio (denoted by m 1 ) and then runs the model but does not output any stable text. let's denote this first model output by h 1 . after the segment contains (at least) c more seconds of audio (denoted by m 2 ) the model is run again with all the audio and outputs h 2 . then the component outputs the common prefix of h 1 and h 2 as stable output s 2 . after the segment again contains (at least) c more seconds of audio (denoted by m 3 ) the model is run again with all the audio. however, now s 2 is forced as prefix in the asr/st model decoding. the model outputs h 3 and the common prefix from h 2 and h 3 is the next stable output s 3 . this procedure is continued until the speech segment ends. note that the asr/st model has a certain maximum input size due to latency, memory and compute constraints. therefore, if this limit is reached, the input audio to the model as well as the corresponding forced prefix is cut away. the revision mode differs from the fixed mode in that the last hypothesis except the common prefix is sent as unstable output. furthermore, in the time period until the speech segment contains again c more seconds of audio, the currently given audio is run through the model and the hypothesis except the last stable output is sent as unstable output. ## low-latency text processing the text processing component receives a stream of (potentially revisable) text messages and sends chunks of text (translation) to the mediator. input processing: first, all input text that arrived is split into sentences by punctuation. then, the backend model (mt) is run. stability detection and output processing: all sentences containing only stable text are processed once and the output is sent as stable text. for the other sentences containing unstable text the behavior depends on the mode. if text is stable or not is given by the speech processing component. the revision mode works as follows: all sentences containing unstable text are processed by the backend model and the output text is sent as unstable text. a similar approach is not possible in the speech processing revision mode (see section 3) since speech segments are not limited in size but the model input size is. for the fixed mode we use the method local agreement from @xcite . the processing is similar to the speech processing. the difference is that the backend model is run when at least one new word is given instead of at least c seconds of audio. in our preliminary experiments, up to at least five words but the results were basically identical since the input is extended by a few words most of the time. furthermore, only the stable part of the sentences containing unstable text is used as input. this restriction is not necessary in the speech processing component since there is no unstable audio input. ## evaluation framework we evaluate our system in an end-to-end fashion. that is, given an input audio, we send it to the system and evaluate the final returned transcript and translation. we provide an evaluation framework foot_3 that assess the system in different aspects and logs the results to categorized experiments on an ui board using mlflow @xcite . we consider different evaluation metrics as follows. bleu: in order to assess the translation quality, we use case-sensitive bleu score, calculated using sacrebleu @xcite . we extract the final stable translation, align it sentence-wise with the gold reference using mwersegmenter @xcite before calculating the bleu score. wer: in order to assess the transcription quality of the asr component in the cascaded setting, we use the case-sensitive word error rate (wer) calculated using jiwer 5 . similar as before, we extract the final stable transcription, align it sentencewise with the gold reference using mwersegmenter @xcite before calculating the wer. latency: we evaluate the latency of the system in an end-to-end manner. factors such as network latency influence our latency metrics. however, our experiments are conducted locally, thus such factors are constant and negligible. we define the end-to-end latency of the system as the average time (in seconds) it takes since an utterance is spoken until its first-unchanged translation is returned by the system. note that the firstunchanged translation is not necessarily already marked as "stable" by the system. for each message returned by the system, we have the stable/unstable flag along with three timestamps, t s , t e , t r . the timestamps t s and t e are the start and end time of the audio segment that aligns to the message. the timestamp t r is when the message was received. we collect the first unchanged messages as follows. we split the received messages into blocks of messages marked from "unstable" to "stable". in each unstable-to-stable block, from the last stable message, we backtrack the previously received unstable messages to find the first ones that has prefix-overlaps with the final stable message. the illustration is shown in figure 3 . once we have collected the first-unchanged messages, we can calculate the latency. we use the same definition of delay as @xcite , where the average delay of the i th message is: then we calculate the latency as the weighted average of the delays of all m first-unchanged messages based on their length: note that the timestamps t s and t e in our latency formula are calculated by the used streaming algorithm. therefore, we also tried another modelindependent latency metric that only uses t r . this metric approximates the segment-message alignment by assuming that each word output by the system has the duration of 0.3 second in the audio. due to the strong assumption, this metric does not represent well the perceived latency. we only use this metric in order to verify our main modeldependent latency metric. we find that the model-independent latency metric and our model-dependent metric provide the same relative ranking of the systems. this indicates that the timestamps t s and t e provided by the model itself are reliable to measure latency. flickering rate: the flickering rate is the average number of flickers per reference word. we count the number of flickers by looking at every pair of consecutive messages in a message block. if two words in the same position in the two messages differ, then it is counted as a flicker (see figure 4 ). the flickering rate is calculated as the total number of flickers divided by the total number of words in the reference. ## experimental setup 6.1 evaluation data we test our system using datasets from different language pairs. test datasets includes: the detailed statistics of the test data is shown in ## related work simuleval @xcite ) provides an evaluation framework for low-latency simultaneous speech translation with a decoupled client-server architecture allowing to plug-in translation models and stability detection policies. as the main difference we leave the audio segmentation up to the model whereas @xcite rely on a pre-segmentation of the audio, we factor in the computational latency in addition to the model latency and explore the scaling behavior in multi-session scenarios, both for a more realistic deployment scenario. similar to this work @xcite implement a low-latency speech translation pipeline, however, their architecture does not scale well to multiple sessions and is not well suited for end-to-end evaluation. ## limitations and conclusion since we run and evaluate the experiments in a realistic real-world scenario, it is difficult to exactly reproduce the results. the experiments are nondeterministic, e.g., because of network latencies. furthermore, the results depend on the speed of the used hardware, especially the used hardware for the backend models. additionally, we expect that each streaming algorithm implemented returns start and end timestamps. this may not be the case for all streaming algorithms one could want to compare. in conclusion, this paper presented a framework for running and evaluating low-latency speech translation under realistic conditions. the research opens up new possibilities for advancing lowlatency translation systems and serves as a resource for researchers seeking to improve the latency and quality of real-time speech translation applications by being able to properly evaluate different models and streaming algorithms.