Stream-level Latency Evaluation for Simultaneous Machine Translation (2021)

Abstract: Simultaneous machine translation has recently gained traction thanks to significant quality improvements and the advent of streaming applications. Simultaneous translation systems need to find a trade-off between translation quality and response time, and with this purpose multiple latency measures have been proposed. However, latency evaluations for simultaneous translation are estimated at the sentence level, not taking into account the sequential nature of a streaming scenario. Indeed, these sentence-level latency measures are not well suited for continuous stream translation, resulting in figures that are not coherent with the simultaneous translation policy of the system being assessed. This work proposes a stream level adaptation of the current latency measures based on a re-segmentation approach applied to the output translation, that is successfully evaluated on streaming conditions for a reference IWSLT task.

## introduction simultaneous speech translation systems just started to become available @xcite @xcite thanks to recent developments in streaming automatic speech recognition and simultaneous machine translation. these systems seamlessly translate a continuous audio stream under real-time latency constraints. however, current translation latency evaluations @xcite are still performed at the sentence-level based on the conventional measures, average proportion (ap) @xcite , average lagging (al) @xcite and differentiable average lagging (dal) @xcite . these measures compute the translation latency for each sentence independently without taking into account possible interactions that lead to accumulated delays in a real-world streaming scenario. additionally, the current measures cannot be used by systems that do not use explicit sentence-level segmentation @xcite . in this work, we first revisit the conventional translation latency measures in section 2 to motivate their adaptation to the streaming scenario in section 3. then, these adapted latency measures are computed and reported on an iwslt task in section 4. finally, conclusions and future work are presented in section 5. ## related work current latency measures for simultaneous translation can be characterised as a normalisation of the number of read-write word operations required to generate a translation y from a source sentence x l(x, y) = 1 z(x, y) with z being a normalisation function, i an index over the target positions and c i a cost function for each target position i. depending on the latency measure, c i is defined as the term in ap normalises the sum over the target sentence of absolute source tokens, while al and dal does over the number of target positions, which in the case of al is limited to those target positions reading new source tokens. indeed, the normalization term of al is referred to as τ . the sentence-level latency measures just described are reported as an average value over an evaluation set of multiple sentence pairs, each one evaluated independently from the others. however, the latency evaluation of a continuous paired stream of sentences has not received much attention, with the exception of the strategy proposed by @xcite . this evaluation strategy considers the straightforward approach of concatenating all sentences into a single source/target pair in order to compute the corresponding latency measure. next section outlines some drawbacks of this strategy (hereafter concat-1) to motivate the discussion on how the current sentence-level latency measures could be adapted to the streaming scenario. ## stream-level evaluation let us consider the translation of a stream of two sentences, the first sentence has two input and two output tokens, while the second one has two input and four output tokens with ratios γ 1 = 1 and γ 2 = 2, respectively. the translation process is performed with a sentence-based wait-k system with catch-up characterised by a function g(i) = k + i-1 γ with @xmath0. ind. sent. these differences in results are magnified when computing latencies on a real streaming evaluation set. on the one hand, al and dal tend to obtain scores that do not reflect the real behaviour of the system when using a concat-1 strategy with a single global γ, since the source-target length ratio varies wildly between different sentences. therefore, the wait-0 oracle will sometimes overestimate the actual writing rate, and sometimes it will underestimate it. moreover, the definition of dal keeps the system from recovering from previously incurred delays, and therefore, every time the writing rate is underestimated, the system falls further and further behind the wait-0 oracle. on the other hand, ap turns out to be little informative when the stream is long enough, since ap always tends to be 0.5 because the delay incurred by a system with a reasonable k is always negligible compared with the total source length. the accuracy of al and dal could be improved if sentence-level estimations for γ would be available somehow in a streaming scenario. with the availability of these estimations in mind, we formulate a streaming version of the cost functions in eq. 2 based on a global g(i) function, which returns the number of source tokens (including those from previous sentences) that have been read as in thus, the global delay is converted to a local representation so that it can be compared with the local sentence oracle. if we use a segmentation-free model whose output is a single text stream, stream-level latency measures can be still computed by re-segmenting the output into sentence-like units (chunks). formally, a segmenter takes an input stream y and a set of reference sentences to compute a re-segmentation ŷn 1 of y . once the re-segmentation is obtained, stream-level latency measures are estimated by considering paired input-output segments (x n , ŷn ). in our case, we re-segment by minimizing the edit distance between the stream hypothesis and the reference translations, analogously to the translation quality evaluation widely-used in speech translation @xcite . likewise, we can resegment the output to compute latency measures if our system uses a different segmentation than the reference. moreover, stream-level al and dal measures computed for a wait-k system are coherently close to k with two caveats. first, there can be deviation from the theoretical value of k due to a inaccurate estimation of the writing rate. given that the wait-k policy uses a fixed γ, there will be some sentences in which this results in lower or higher writing rates than desirable. this is a feature inherent to the fixed policy itself. second, a deviation could also occur due to re-segmentation errors. for instance, a word that is part of the translation of the n-th segment can be wrongly included into the previous n -1-th segment causing an increase of the latency. both sources of latency are illustrated in figure 1 . these two caveats given the definition of dal imply that a system can never recover from previous delays, which might be an acceptable solution when computing latency measures at the sentence level, but it seems too strict and unrealistic when computing latency measures for streams comprising tens of thousands of words. to alleviate this problem, we propose to multiply the cost of a write operation 1 γn in g n (i) by a scaling factor @xmath1, 1]. in practice, for values of s close to 1, this means that the write operation costs slightly less for the real system than for the oracle. we believe this is an acceptable practical solution given that there are many ways that this could be achieved in realworld tasks, such as rendering subtitles slightly faster or, in the case of cascade speech-to-speech, slight reducing the duration of tts segments or increasing the playback speed. finally, the scaling factor s can be also understood as a hyperparameter that bridges the gap between al (@xmath2. ## experiments the stream-level latency measures proposed in section 3 are now computed and evaluated on the iwslt 2010 german-english dev set @xcite . to simulate a streaming scenario, all source sentences are concatenated into a single input stream. then, they are segmented into sentences and translated with a wait-k fixed policy. as a result, it is expected that a well-behaved latency measure should rank the systems by increasing order of k. our streaming simultaneous translation sysleft: writing rate error with @xmath3; the model uses γ = 1, but the actual value is γ = 1.5. right: segmentation error with @xmath4; the first translated word of the second sentence is wrongly assigned to the first sentence during resegmentation, i.e. ŷ1 = (y 1,1 , y 1,2 , y 1,3 , y 1,4 , y 2,1 ). al ( now that we have experimentally shown that the concat-1 approach is unable to properly compute latencies, we move onto computing the streamadapted version of the latency measures. the computation of stream-level ap (left), al (center) and, dal (right) with @xmath6.0 and @xmath7.95 (dashed lines) as a function of k in the multi-k approach are shown in figure 2 . the behaviour of ap and al is that expected for the four experimental setups defined above, but the conventional dal measure (@xmath8.0) abruptly suffers the effect of not being able to recover from accumulated delays due to the cost of write operations. in contrast, dal with @xmath9.95 exhibits a smooth interpretable behaviour as a result of compensating for re-segmentation errors. moreover, the gap between "in. seg." and "in. seg. + out. seg." is not significant, therefore we believe that, if the translation quality is good enough, the automatic re-segmentation process is an acceptable way of computing stream-level latencies. lastly, as expected, if we use an oracle system that outputs the reference translation with the appropriate writing rate for each sentence ("policy + in. seg. + out. seg."), the obtained al and dal values are very close to the theoretical value k. if we compute dal using @xmath10.95, we obtain similar values without the need of using any oracle, while accounting for the additional cost of write operations. thus, unlike the concat-1 approach, our streamlevel approach is highly effective for providing interpretable and accurate latency measures. ## conclusions in this work, an adaptation of the current latency measures to a streaming setup is proposed moti-vated by the lack of interpretability of sentencelevel latency measures in this setup. this adaptation basically consists in the modification of the conventional latency measures to move from a sentence-level evaluation based on a local delay function to a stream-level estimation by using a global delay function that keeps track of delays across the whole translation process. at the same time, a re-segmentation approach has been proposed to compute these latency measures on any arbitrary segmentation of the input stream used by the translation model. the resulting measures are highly interpretable and coherent accounting for the actual behaviour of the simultaneous translation system in a real streaming scenario.