MT Metrics Correlate with Human Ratings of Simultaneous Speech Translation (2023)

Abstract: There have been several meta-evaluation studies on the correlation between human ratings and offline machine translation (MT) evaluation metrics such as BLEU, chrF2, BertScore and COMET. These metrics have been used to evaluate simultaneous speech translation (SST) but their correlations with human ratings of SST, which has been recently collected as Continuous Ratings (CR), are unclear. In this paper, we leverage the evaluations of candidate systems submitted to the English-German SST task at IWSLT 2022 and conduct an extensive correlation analysis of CR and the aforementioned metrics. Our study reveals that the offline metrics are well correlated with CR and can be reliably used for evaluating machine translation in simultaneous mode, with some limitations on the test set size. We conclude that given the current quality levels of SST, these metrics can be used as proxies for CR, alleviating the need for large scale human evaluation. Additionally, we observe that correlations of the metrics with translation as a reference is significantly higher than with simultaneous interpreting, and thus we recommend the former for reliable evaluation.

## introduction the current approach to evaluate simultaneous speech translation (sst, @xcite systems that have text as the output modality is to use automatic metrics which are designed for offline text-to-text machine translation (mt), alongside other measures for latency and stability. researchers tend to use offline metrics, such as bleu @xcite , chrf2 @xcite , @xcite , comet @xcite and others @xcite in sst despite no explicit evidence that they correlate with human ratings. however, simultaneous speech-to-text translation has different characteristics compared to offline text-to-text mt. for example, when the users are following subtitles in real-time, they have limited time for reading and comprehension as they cannot fully control the reading pace by themselves. therefore, they may be less sensitive to subtle grammar and factual flaws than while reading a text document without any time constraints. the human evaluation of sst should therefore reflect the simultaneity. the users may also prefer brevity and simplicity over verbatim word-for-word translation. even if the reference is brief and simpler than the original, there may be lots of variants that the bleu score and other mt metrics may not evaluate as correct. furthermore, sst and mt differ in their input modalities. mt sources are assumed to originate as texts, while the sst source is a speech given in a certain situation, accompanied by para-linguistic means and specific context knowledge shared by the speaker and listener. transcribing speech to text for use in offline evaluation of sst may be limiting. in this paper, we aim to determine the suitability of automatic metrics for evaluating sst. to this end, we analyze the results of the simultaneous speech translation task from english to german at iwslt 2022 @xcite , where we calculate the correlations between mt metrics and human judgements in simultaneous mode. there are five competing systems and human interpreting that are manually rated by bilingual judges in a simulated real-time event. our studies show that bleu does indeed correlate with human judgements of simultaneous translations under the same conditions as in offline text-to-text mt: on a sufficiently large number of sentences. furthermore, chrf2, bertscore and comet exhibit similar but significantly larger correlations. to the best of our knowledge, we are the first to explicitly establish the correlation between automatic offline metrics with human sst ratings, indicating that they may be safely used in sst evaluation in the currently achieved translation quality levels. additionally, we statistically compare the metrics with translation versus interpreting reference, and we recommend the most correlating one: translation reference and comet metric, with bertscore and chrf2 as fallback options. we publish the code for analysis and visualisations that we created in this study. 1 it is available for further analysis and future work. ## related work we replicate the approach from text-to-text mt research (e.g. @xcite that examined the correlation of mt metrics with human judgements. the strong correlation is used as the basis for taking the metrics as reliable. as far as we know, we are the first who apply this approach to sst evaluation in simultaneous mode. in this paper, we analyze four metrics that represent the currently used or recommended @xcite types of mt metrics. bleu and chrf2 are based on lexical overlap and are available for any language. bertscore (zhang et al., 2020) is based on embedding similarity of a pre-trained bert language model. comet @xcite is a neural metric trained to estimate the style of human evaluation called direct assessment @xcite . comet requires sentence-tosentence aligned source, translation and reference in the form of texts, which may be unavailable in some sst use-cases; then, other metric types may be useful. another fact is that bertscore and comet are available only for a limited set of languages. ## human ratings in sst as far as we know, the only publicly available collection of simultaneous (not offline) human evaluation of sst originates from iwslt 2022 @xcite english-to-german simultaneous translation task, which is described in "findings" @xcite , see highlights of it we discuss in appendix a). the task focused on speech-to-text translation and was reduced to translation of individual sentences. the segmentation of the source audio to sentences was provided by organizers, and not by the systems themselves. the source sentence segmentation that was used in human evaluation was gold (oracle). it only approximates a realistic setup where the 1 github.com/ufal/mt-metrics-in-simst segmentation would be provided by an automatic system, e.g. @xcite , and may be partially incorrect and cause more translation errors compared to the gold segmentation. the simultaneous mode in simultaneous translation task means that the source is provided gradually, one audio chunk at a time. after receiving each chunk, the system decides to either wait for more source context, or produce target tokens. once the target tokens are generated, they can not be rewritten. the participating systems are submitted and studied in three latency regimes: low, medium and high. it means that the maximum average lagging @xcite between the source and target on validation set must be 1, 2 or 4 seconds in a "computationally unaware" simulation where the time spent by computation, and not by waiting for context, is not counted. one system in low latency did not pass the latency constraints (see findings, page 44, numbered 141), but it is manually evaluated regardless. computationally unaware latency was one of the main criteria in iwslt 2022. it means that the participants did not need to focus on a low latency implementation, as it is more of a technical and hardware issue than a research task. however, the subtitle timing in manual evaluation was created in a way such that waiting for the first target token was dropped, and then it continued with computationally aware latency. ## correlation of cr and mt metrics in this section, we study the correlation of cr and mt metrics bleu, chrf2, bertscore and comet. we measure it on the level of documents, and not on the test set level, increasing the number of observations for significance tests. there are 60 evaluated documents (17 in the common subset and 43 in non-native) and 15 system candidates (5 systems, each in 3 latency regimes), which yields 900 data points. we discovered that cuni-kit system outputs are tokenized, while the others are detokenized. therefore, we first detokenized cuni-kit outputs. then, we removed the final end of sequence token (</@xmath0. finally, we calculated bleu and chrf2 using sacrebleu @xcite , bertscore and comet. see appendix b for metric details and signatures. ## conclusion we found correlation of offline mt metrics to human judgements of simultaneous speech translation. the most correlating and thus preferred metric is comet, followed by bertscore and chrf2. we recommend text translation reference over interpreting, and single sequence alignment for neural, and mwersegmenter for n-gram metrics.