Continuous Rating as Reliable Human Evaluation of Simultaneous Speech Translation (2022)

Abstract: Simultaneous speech translation (SST) can be evaluated on simulated online events where human evaluators watch subtitled videos and continuously express their satisfaction by pressing buttons (so called Continuous Rating). Continuous Rating is easy to collect, but little is known about its reliability, or relation to comprehension of foreign language document by SST users. In this paper, we contrast Continuous Rating with factual questionnaires on judges with different levels of source language knowledge. Our results show that Continuous Rating is easy and reliable SST quality assessment if the judges have at least limited knowledge of the source language. Our study indicates users’ preferences on subtitle layout and presentation style and, most importantly, provides a significant evidence that users with advanced source language knowledge prefer low latency over fewer re-translations.

## introduction simultaneous speech translation (sst) is a technology that assists users to understand and follow a speech in a foreign language in real-time. the users may need such an assistance because of limited knowledge of the source language, the speaker's non-native accent, or the topic and vocabulary. the technology can be used for the target languages, for which human interpretation is unavailable, e.g. due to capacity reasons. candidate systems for simultaneous speech translation differ in quality of translation, latency and the approach to stability. some are streaming, only adding more words @xcite @xcite @xcite @xcite , some allow re-translation as more input arrives @xcite @xcite @xcite . finally, subtitle presentation options figure 1 : a detail of the default layout with the video document "dinge erklärt: impfen...". 1 the video is at the top, overlaid by two lines of subtitles in czech, followed by buttons for continuous rating. the button labels are: 1: worse; 2: average; 3: good; 0: i do not understand at all. the evaluation of the traditional, text-to-text machine translation (mt) has been researched for many years (see e.g. han, 2018 or developments and discussion within the series of wmt, @xcite . it targets only the translation quality. sst evaluation faces new challenges: simultaneity, latency, and readability to humans. evaluating only selected aspects in isolation is reasonable (as mt quality in @xcite @xcite , however, a complete evaluation must be end-to-end, from sound acquisition to subtitling, and take into account the intent of communication. we generalize the intent to passing pieces of information from the speaker (sender) to a participant in an online session (receiver). ## evaluation campaign in our evaluation, we simulate live events on which participants need assistance with understanding the spoken language. the source and target languages in our study are german and czech, respectively. this is an interesting example of two neighbouring countries, distinct language families and yet a relatively well studied pair with sufficient direct training data. ## results first, we analyzed the comprehension levels (section 4.1) and presentation layouts (section 4.2). then, we selected the most preferred layout and used it for examining the impact of flicker on comprehension in flicker experiments (section 4.3). 6 ## conclusion we proposed a novel and effective method for end-to-end user evaluation of simultaneous speech translation sst called continuous rating, publishing an open source evaluation tool for the future use. we showed that this method can be used for measuring comprehension and evaluating subtitling parameters. we demonstrated how user comprehension differs from offline mt to online mt. we showed that the users with a knowledge of the source language prefer low latency despite higher instability. we demonstrated that continuous rating can be used as a time-efficient human evaluation metric when employing judges with at least b2 (or, preferrably, c1) level of source language proficiency.