Going Beyond Your Expectations in Latency Metrics for Simultaneous Speech Translation (2025)

Abstract: Current evaluation practices in Simultaneous Speech Translation (SimulST) systems typically involve segmenting the input audio and corresponding translations, calculating quality and latency metrics for each segment, and averaging the results. Although this approach may provide a reliable estimation of translation quality, it can lead to misleading values of latency metrics due to an inherent assumption that average latency values are good enough estimators of SimulST systems’ response time. However, our detailed analysis of latency evaluations for state-of-the-art SimulST systems demonstrates that latency distributions are often skewed and subject to extreme variations. As a result, the mean in latency metrics fails to capture these anomalies, potentially masking the lack of robustness in some systems and metrics. In this paper, a thorough analysis of the results of systems submitted to recent editions of the IWSLT simultaneous track is provided to support our hypothesis and alternative ways to report latency metrics are proposed in order to provide a better understanding of SimulST systems’ latency.

## introduction in recent years, there has been a growing demand for real-world applications that use simultaneous speech translation (simulst) systems to provide real-time translation across languages. current use cases include live broadcasts of news, lectures, and debates, where the continuous audio stream mainly consists of spoken speech. these applications require systems that do not only deliver high-quality translations consistently, but also maintain low latency to ensure effective communication and keep the audience engaged with the audiovisual content. current evaluation of latency in simulst relies on automatic or reference segmentation of datasets @xcite to split the input audio and its translations, computing metrics for each segment, and averaging the results. however, this latency estimation for simulst systems has significant limitations @xcite , and reported latency figures may differ from the actual behavior of systems in a real-world scenario. beyond the segmentation-related issues identified in these previous work, we argue that a major cause of the observed discrepancies may be due to the exclusive reliance on the mean when reporting results. while mean latencies allow to simplify system comparison to speed up their development, we hypothesize that by relying solely on the mean, we may be overlooking spurious or faulty system behaviors, as well as anomalies in the current latency metrics. while the presence of outliers is relatively common when evaluating machine learning systems with any metric, their significance and impact in latency evaluation of simulst systems are currently being greatly underestimated. thus, mean values of latency metrics may result in misleading conclusions when comparing simulst systems. in this paper we demonstrate that latency metrics, as currently reported by their average values, are not a sufficiently accurate characterization of simulst systems' response time in the presence of latency distributions that do not follow a normal distribution. to this purpose, latency metrics of systems submitted to recent editions of the iwslt simulst track were thoroughly analysed. our findings reveal that average latency metrics can mask undesirable systems' behavior, potentially resulting in misleading conclusions. this highlights the need for more robust evaluation methods for latency in simulst systems. our contributions are summarized as follows: simulst systems submitted to iwslt in terms of latency metrics. the evaluation of simulst systems is performed in two dimensions: translation quality and latency. while translation quality is typically evaluated by using conventional translation metrics such as bleu @xcite and comet @xcite @xcite , multiple metrics have been developed for measuring the latency of simulst systems. earlier proposed metrics such as average proportion (ap) @xcite and consecutive wait length (cw) @xcite have been mostly superseded in usage by average latency (al) @xcite and proposed variants such as differentiable average lagging (dal) (cherry and foster, 2019) and length-adaptive average lagging (laal) @xcite which try to remedy several limitations in the original al metric definition. another more recent metric which has received a fair amount of adoption is average token delay (atd) (kano et al., 2023; ?), which tries to fix several limitations underlining al based metrics. additionally, wein et al. (2024) and makinae et al. (2024) have proposed metrics tailored towards the evaluation of the quality and latency of translations closer to human interpretation. as characterized in iranzo-sánchez et al. ( 2021 ), current latency measures for simulst can be defined as a normalisation of a latency cost (in terms of words or milliseconds) required to generate a translation ŷ provided a source sentence x and its corresponding reference translation y: with z being a normalisation function, i an index over the target positions and c i a cost function for each target position i. depending on the latency metric, c i is defined as papi et al. (2024) analyzed the behaviour of current simulst systems surveying the current literature to define standardized terminology and taxonomy across different simulst papers, while also identifying overlooked challenges in current simulst systems and recommendations for future work in the field. related to this, xu et al. (2024) identified how current computationally-aware metrics are incorrectly calculated in current standard simulst evaluation toolkits. finally, macháček et al. (2023a) showed how mt quality metrics have good correlation with human ratings for iwslt 2022 and sperber et al. (2024) analyzed the iwslt 2023 evaluation campaign across different aspects. ## limitations of mean latencies in this section, the limitations of mean latencies are illustrated with a simplified example computed on real data from iwslt competitions to show how if we were to pick between the two systems considering conventional latency metrics based on the mean, system a would be considered as good as system b. however, by looking into latency distribution across samples, the values of system a on approximately 10% of the samples significantly differs from the mean more than system b. by deploying system a instead of b in a real streaming st scenario, the probability that latency spikes appear is high enough to lead to an accumulation of delays that causes a desynchronization between the audio stream and the translation text being generated. in other words, the content of the target stream will not be up to date with the source stream. this behaviour is highly undesirable for the end-user experience, and system a would not be an acceptable choice in a real streaming scenario, while system b with a more consistent latency would have been selected on the basis of its percentile scores. ## iwslt as a case study results for iwslt 2022 shared tasks were available for the must-c 2.0 tst-common partition @xcite and various segmentations of the official test set, for which the reference segmentation of the datasets was selected. for iwslt 2024, results were only available on the official test sets. we focused on the speech-to-text track logs and left out the available text-to-text logs from our study. for the iwslt 2022, the delays available in the evaluations logs were used to recalculate al and dal scores using the latest version of the simuleval toolkit 2 . in addition, laal and atd scores, which were still not available at the time of the shared task, were calculated. for each team the corresponding submitted system was selected. for those systems that did not comply with the latency constraints, we treat them as if they were in the nearest latency band. finally, the cuni-kit participation in the en-de track was left out due to tokenization issues, while the xiaomi participation in the en-zh track was not available. ## iwslt analysis first of all, the evaluation logs were checked to detect the possible errors in the dataset, such as mismatched source-target pairs and misaligned segmentations. these errors would jeopardize the analysis of latency solely explained by the performance of simulst systems. as a result of this initial error analysis, we decided to focus on the official iwslt 2022 and 2024 test sets, as these proved largely error-free in contrast to the iwslt 2022 must-c sets. results for the must-c partition can be found in appendix a. ## recommendations in the previous section, a series of tools for descriptive statistics have been presented and illustrated on real data to study and characterize the latency of simulst systems. the results presented in the previous section has allowed us to reflect on effective ways to report latency for simulst systems in order to gain insight into their actual behavior in a real scenario. first of all, as already encouraged in the evaluation of translation quality @xcite , it is strongly recommended to report, in addition to the evaluation tool, the software version of the tool in order to guarantee reproducibility. our analysis shows how reporting a measure of central tendency such as the mean is not able to properly capture the underlying latency of a simulst system and misleading comparisons across systems could be drawn. for this reason, it is convenient to provide descriptive statistics that offer an overall view of the system latency beyond the mean. in this sense, violin and specially normal probability plots were found significantly useful and enlightening to consistently compare systems and detect undesirable system latencies. in addition, normal probability plots proved to be an effective tool for assessing the normality of latency distributions, while also capturing other key descriptive statistics such as skewness and kurtosis, as well as the differences in percentile values across different systems. complementarily, figures reporting latencies for higher percentiles, along with mean and median, are also recommended to prove the robustness of the system latency. finally, over-wait scores are very valuable to identify the percentage of samples in which a simulst system is exhibiting a degenerated offline behavior. ## conclusions in this paper, we have critically examined the current practices for evaluating latency in simulst systems, focusing on the limitations of relying solely on mean latency metrics. through a detailed analysis of systems submitted to recent editions of the iwslt simulst track, mean latency metrics have demonstrated to fail to provide a complete view of simulst systems' response time, particularly in the presence of skewed latency distributions and high latency values. our findings reveal that the mean latency can mask unacceptable latency values which are critical for understanding the performance of these systems in a real scenario. alternative methods have been proposed for reporting descriptive statistics of latency metrics, emphasizing the importance of considering the entire latency distribution rather than just the mean value. specifically, violin and specially normal probability plots were recommended to graphically report latency values per percentile in order to provide a more comprehensive view of the system behavior. our analysis underscores the need for more robust evaluation practices in simulst research. we strongly believe that by adopting the recommendations outlined in this paper, researchers and practitioners can gain a deeper understanding of system performance, leading to more reliable and consistent simulst systems in real-world applications. future work should continue to explore the development of new metrics and evaluation methodologies that better align with the challenges of real-time translation, ensuring that simulst systems meet the demands of end-users in dynamic and continuous speech scenarios. in particular, the more realistic stream-level latency metrics @xcite must be revisited taking into account the lessons learned in this work.