Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translation (2022)

Abstract: Simultaneous speech translation (SimulST) systems aim at generating their output with the lowest possible latency, which is normally computed in terms of Average Lagging (AL). In this paper we highlight that, despite its widespread adoption, AL provides underestimated scores for systems that generate longer predictions compared to the corresponding references. We also show that this problem has practical relevance, as recent SimulST systems have indeed a tendency to over-generate. As a solution, we propose LAAL (Length-Adaptive Average Lagging), a modified version of the metric that takes into account the over-generation phenomenon and allows for unbiased evaluation of both under-/over-generating systems.

## introduction simultaneous speech-to-text translation (simulst) is the task in which the generation of the textual translation in the target language starts before the entire audio input in the source language has been ingested by the model. the need to have high quality translations in the shortest possible time therefore becomes the main objective of simulst systems, which have to satisfy specific time requirements depending on the application scenarios. these requirements are usually expressed in terms of latency, that is the elapsed time between the pronunciation of a word and the generation of its textual translation. how latency is measured hence plays a crucial role in systems evaluation. the simulst task was initially addressed using cascaded models @xcite that divide the translation process into two steps: simultaneous automatic speech recognition @xcite , and simultaneous machine translation @xcite . for this reason, the first latency metrics were designed to evaluate simultaneous machine translation (simulmt) systems @xcite @xcite . among them, average lagging -al - @xcite is the most popular one, and its adaptation to simulst by @xcite has become a widely adopted @xcite @xcite de facto standard. 1 the adaptation by @xcite sparks from a weakness observed in the original formulation of the metric. being susceptible to under-generation, it results in biased evaluations favouring systems that produce shorter predictions compared to the reference. however, though successful in correcting this behaviour, the proposed adaptation did not consider the opposite case of over-generation, which occurs when the prediction is longer than the reference. to fill this gap, in this paper we introduce laal (length-adaptive average lagging): 2 a simple yet effective extension of al that takes into account also over-generation and allows for fair simulst systems comparisons. after a brief explanation of al calculation (section 2), we expose its incorrect behaviour in presence of over-generation phenomena (section 3) and show that over-generation is actually present in the output of recent simulst systems (section 4). then, we present the new laal metric (section 5), whose computation is adjusted at sentence level by looking at the length of model predictions. through examples, we show that, unlike the previous al formulation, our metric is able to fairly evaluate both under-and overgenerating systems. we conclude our work with a discussion (section 6) about problems that still need to be solved for latency computation, remarking that our proposal represents a first step toward a more reliable assessment of simulst systems performance. ## average lagging the idea behind the al metric is to quantify how much time the system is out of sync with the speaker. in simulst, the input sequence is represented as a stream of audio speech in the source language @xmath0, ..., x |x| ] where each element x j is a raw audio segment of duration t j , the reference as a stream of words in the target language therefore, the delay with which the y i word is emitted is d @xmath1. using this notation, in @xcite average lagging was initially defined as follows: however, the authors noticed that this adaptation was not robust for models that tend to stop generating the hypothesis too early, that is systems that under-generate. this phenomenon is more likely to happen in simulst than in simulmt, for which al was first proposed. for instance, the presence of long pauses in the speech may induce systems to generate the end of sentence token too early, even if the source utterance is not yet complete. as observed by the authors, when this phenomenon occurs, the lagging behind the oracle becomes negative. it follows that relatively good latency-quality trade-offs can be achieved thanks to inappropriate al discounts in case of under-generation, while this does not reflect the reality. thus, in @xcite , equation 1 was redefined as: ## the problem of over-generation in this paper, we point out a major issue of al that arises in presence of over-generation. as we will see, al improperly favors over-generating systems, potentially leading the community to wrong conclusions due to biased evaluations. to illustrate how over-generation affects al computation, we consider a real example from the @xmath2. as shown in figure 1 , the prediction suffers from over-generation, especially in the first part of the sentence where more target words are produced compared to the reference. the system translates "en primer lugar," instead of "primero,", forcing all the predicted words to compare with the successive word in the reference. for instance, "es" in the caat output is computed against "juego" in the oracle and its very low lagging (49ms) is a considerable underestimation of the correct lagging with respect to the "es" word in the oracle (763ms). likewise, "de" is assigned a negative lagging (-62ms) instead of the 652ms delay with respect to the time of "de" in the oracle. finally, all the words generated before the end of the utterance (in our example 5000ms) and exceeding the reference length are compared with the last word of the oracle; the same happens to the first word emitted when the utterance is over, while the other words after the end of the utterance are ignored. as a result, the al of caat output for this sentence is 198ms, an extremely low latency that does not reflect the truth. indeed, if we correctly align and compare the words in the system output and the oracle, we see that lagging is on average 846ms. 3 this represents a problem, since the al metric is rewarding an over-generating system, potentially hindering a fair comparison with other models. in light of these observations, two questions arise. first, what are the conditions leading to biased, i.e. underestimated, al values? the example above shows that the problem arises when: i) the system over-generates, and ii) the over-generated words appear before the utterance ends (the earlier in time, the lower the final al score will be). second, why was this problem overlooked when al was introduced? to answer this question, recall that al for simulst was initially proposed to evaluate systems showing a biased behaviour toward under-generation, with predictions length reaching at most the reference length. indeed, earlier simulst systems @xcite were designed to emit only one word at each time step. consequently, even in the case of over-generation, it was extremely unlikely for the number of words emitted before the end of the utterance to be higher than the number of words in the reference. moreover, additional words (if any) were generated only once the utterance was over. as mentioned above, the current al implementation ignores all but the first word emitted at the end of the utterance. therefore, the over-generation occurring at the end of the utterance does not affect the metric computation. instead, al is not robust to over-generation if it occurs before the end of the utterance, an extremely unlikely behavior in early systems but frequent in more recent ones, as we will see in the next section. ## over-generation frequency to quantify the impact of over-generation on system evaluation, we check how frequently it occurs in the output of three simulst systems: caat, the wait-k model by @xcite , and an offline model with the wait-k policy applied only at inference time @xcite . we run three systems on the en-es section of must-c tst-common by varying the k value at inference time in the range {3, 5, 7, 9, 11}. we measure over-generation in terms of average word length difference (awld) between systems predictions and the corresponding references, that is: ## length adaptive average lagging based on the observations made in sections 3 and 4, we propose laal (length-adaptive average lagging), a modified version of al accounting also for the over-generation phenomena. laal defines the oracle delays by dividing the utterance length by the maximum between the reference and the model prediction length. specifically, we consider the reference length when the prediction is shorter (under-generation), and the prediction length when the prediction is longer (over-generation). this means that the correction is made at sentence level, making the metric applicable to a system disregarding its under-or over-generation tendency. figure 2 shows both the under-generation (in blue) and the over-generation (in green) cases. analyzing the under-generation case, we can clearly see the motivation behind the correction made by @xcite : if we consider the prediction length (|y 1 |) in the al computation and the prediction is too short, the system is favoured since negative delays are summed (the blue straight line is mainly below the blue dashed line). a more reliable evaluation is obtained by considering the reference length (|y * |), since the straight blue line is always above the dashed red line. by analyzing the over-generation case, we observe the opposite problem: if we consider the reference length (|y * |) in the al computation and the prediction is too long (|y 2 | > |y * |), the system is favoured since negative delays are summed (the straight green line stays almost always below the red dashed line). this can be corrected by considering the prediction length (|y 2 |) instead. therefore, to make a more reliable evaluation where neither under-generation nor over-generation are rewarded, we have to take the maximum between |y * | and |y| in the delay computation (the two conditions are: |y * | if |y| ≤ |y * |, and |y| if |y| > |y * |). accordingly, equation 3 can be modified to obtain laal as: the difference between applying al and laal to evaluate our three systems is shown in going back to the example in figure 1 , the latency value computed with laal is 707ms. compared to the al value of 198ms, this is much closer to the real measure of 846ms calculated in section 3. in light of these observations, we can conclude that the laal metric gives a more reliable evaluation of the simulst systems compared to al. ## limitations the proposed laal metric is a first step toward a more accurate evaluation of simulst systems. although in this work we focused on the overgeneration problem, we did not address another limitation of al @xcite . the problem is that, as shown in section 2, al compares the system output with an oracle that emits only one word at each time step, each one with a fixed word duration. 4 this means that, in its computation, we assume that the reference words are uniformly distributed in each utterance. however, considering that the amount of information contained in audio segments of the same length could be extremely different, this represents an unrealistic approximation. for instance, a speech segment can contain silences, long pauses, and the speech rate can vary considerably. as a consequence, the latency scores obtained can still largely differ from the latency experienced by the user. this advocates for the development of more human-centric solutions that go beyond al-like metrics despite their success, accounting for different audio phenomena and their impact on the actual latency perceived by the users, also considering the visualization strategy selected @xcite . we leave this line of investigation for future work. ## conclusions we showed through examples based on real systems that the current average lagging computation is inadequate to correctly measure simulst performance in presence of over-generation phenomena. to overcome this problem, we proposed length-adaptive average lagging (laal), a latency metric that can effectively handle both underand over-generation at sentence level, leading to a more reliable evaluation of simulst systems. asia-pacific chapter of the association for computational linguistics and the 10th international joint conference on natural language processing, pages 582-587, suzhou, china. association for computational linguistics. yusuke oda, graham neubig, sakriani sakti, tomoki toda, and satoshi nakamura. 2014. optimizing segmentation strategies for simultaneous speech translation. in proceedings of the 52nd annual meeting of the association for computational linguistics (volume 2: short papers), pages 551-556, baltimore, maryland. association for computational linguistics. sara papi, marco gaido, matteo negri, and marco turchi. 2022. does simultaneous speech translation need simultaneous models? arxiv preprint arxiv:2204.03783. sara papi, matteo negri, and marco turchi. 2021. visualization: the missing factor in simultaneous speech translation. in proceedings of the eighth italian conference on computational linguistics. matt post. 2018. a call for clarity in reporting bleu scores. in proceedings of the third conference on machine translation: research papers, pages 186-191, belgium, brussels. association for computational linguistics. kanishka rao, haşim sak, and rohit prabhavalkar. 2017. exploring architectures, data and units for streaming end-to-end speech recognition with rnntransducer. in 2017 ieee automatic speech recognition and understanding workshop (asru), pages 193-199. ieee. xingshan zeng, liangyou li, and qun liu. 2021. real-trans: end-to-end simultaneous speech translation with convolutional weighted-shrinking transformer. in findings of the association for computational linguistics: acl-ijcnlp 2021, pages 2461-2474, online. association for computational linguistics.