Barriers to Effective Evaluation of Simultaneous Interpretation (2024)

Abstract: Simultaneous interpretation is an especially challenging form of translation because it requires converting speech from one language to another in real-time. Though prior work has relied on out-of-the-box machine translation metrics to evaluate interpretation data, we hypothesize that strategies common in high-quality human interpretations, such as summarization, may not be handled well by standard machine translation metrics. In this work, we examine both qualitatively and quantitatively four potential barriers to evaluation of interpretation: disfluency, summarization, paraphrasing, and segmentation. Our experiments reveal that, while some machine translation metrics correlate fairly well with human judgments of interpretation quality, much work is still needed to account for strategies of interpretation during evaluation. As a first step to address this, we develop a fine-tuned model for interpretation evaluation, and achieve better correlation with human judgments than the state-of-the-art machine translation metrics.

## introduction simultaneous interpretation is an especially difficult type of translation because it requires the system or human to convey the ideas from one language to another in real time. due to the cognitive load and constraints on memory associated with the act of human interpretation, the number of errors increases exponentially after only minutes of interpreting @xcite . to compensate for these challenges, interpreters often make use of a range of strategies, such as summarization and segmentation @xcite , to concisely provide the gist of what is being said in the source language. despite the prevalence of both human simultaneous interpretation and automatic interpretation * work completed while interning at google. models, investigations into how to effectively evaluate the quality of interpretation data are extremely limited. 1 recent work suggests that standard automatic machine translation metrics are appropriate for interpretation, due to a correlation of select mt metrics (namely bleu @xcite , nist @xcite , and meteor @xcite ) with human judgments of interpretation quality @xcite and the use of meteor for interpreter quality assessment @xcite . recent work has also argued that simultaneous interpretation evaluation systems should be trained and tested on interpretation data as opposed to translation data @xcite . in support of this argument, @xcite demonstrate that there is a sizable difference in bleu score (13.83 points) when evaluating based on interpretation or translation data. given the strategies unique to human interpretation and indications in prior work as to the potential utility of machine translation (mt) metrics, our goal in this work is to investigate the applicability of both (1) interpretation data as references, and (2) existing machine translation metrics for evaluation of interpretation. we argue that the strategies that interpreters leverage to be able to perform live interpretation are critical to the task and should not be penalized by the evaluation metric. thus, we pose three primary questions: to carry out these research questions, we analyze and evaluate both human interpretations and machine translations, identifying potential interpreter strategies that may degrade metric effectiveness (section 3.4). for meta-evaluation, we conduct a human evaluation on the quality of both human interpretation and machine translation to see how those metrics correlate with human judgments (section 4.1). we then conduct a study to assess the sensitivity of the metrics when these strategies are present in an interpretation (section 4.2). finally, in order to further improve the correlation with human judgments, we adapt the method from met-ricx @xcite and create a fine-tuned model using our interpretation data and human annotations (section 4.3). we demonstrate that our new metric is better at assessing interpretation quality, achieving higher correlation with human judgments, suggesting that fine-tuned neural metrics can be valuable tools for assessing interpretation. ## related work common strategies in interpretation include segmentation, passivization, generalization, and summarization @xcite . @xcite also show that interpretations are consistently simpler than their translated counterparts, having lower lexical density, lower mean sentence length, and greater use of frequent words. regarding the use of interpretation data as references, @xcite show that there is a 13.83 gap in bleu score when evaluating simultaneous machine translation output against interpretation transcripts versus the revised text translation. the decrease in system performance when evaluating against interpretation data can also be observed in @xcite and @xcite . the differences between how translators and interpreters translate speech is notable; still, there is no consensus on how to use automatic metrics to evaluate interpretation. within the realm of interpretation evaluation, @xcite adapt a framework developed for human interpreter assessment and perform a human evaluation of both interpreters and machine translation systems. they find that interpreters perform better in intelligibility than machine translation systems, but worse in terms of informativeness. @xcite recommends comet @xcite as a metric for assessing automatic simultaneous speech translation, though the systems considered do not mimic interpreter strategies such as summarization. recent work has also perturbed machine translation data in order to investigate the sensitivity of mt evaluation metrics to different types of errors @xcite . we adapt this idea in our work to investigate the sensitivity of mt metrics to different interpretation strategies. per the results of wmt22, metricx and comet are the highest ranked automatic mt evaluation metrics when ranked via agreement with human judgments of machine and human translations @xcite . a number of multilingual interpretation corpora have been developed in prior work. @xcite collect an english↔japanese interpretation corpus and show that the most experienced interpreter achieves the highest bleu score. @xcite present the naist dataset, which is a larger english↔japanese interpretation corpus, and using a similar setup as @xcite , show that the most experienced interpreter also has a higher bertscore @xcite . however, they point out that bertscore fails when interpreters use a strategy like summarization. the voxpopuli corpus includes simultaneous interpretation data of european parliament event recordings in 24 languages @xcite . @xcite also collect a chinese to english interpretation corpus with three experienced interpreters. depending on whether the interpreters' performance is based on human judgments or bleu scores, the interpreters rank differently in terms of performance. ## methodology in order to assess the presence of barriers to effectively evaluating interpretation data, we leverage comparisons between simultaneous interpretation data and machine translation data (as described in section 3.1); we perform a human evaluation study on the interpretations and machine translation data (section 3.2) to collect human judgments of both fluency and adequacy. we use five machine translation metrics (section 3.3) to assess the applicability of existing metrics in evaluating interpretation data, and identify features in the interpretation data which may impact metric correlation with human judgments (section 3.4). ## results in the subsections that follow, we address each of our research questions. namely, in section 4.1 we address whether human interpretations (collected for other purposes) have sufficient quality to be considered for use as references in evaluation. then, in section 4.2, we ascertain whether we can use existing machine translation metrics-as they are-to evaluate interpretation data. finally, in section 4.3, we develop a refined automatic metric which achieves higher correlation with human judgments of interpretation quality and accounts for common features of interpretations. ## conclusion in this work, we have performed extensive qualitative and quantitative experimentation to measure the impact of common features of interpretation on metric scores. we have studied the sensitivity of mt metrics to interpretation features, including disfluency, segmentation, summarization, and paraphrasing. we argue that common interpreter features should not be penalized if the original gist is successfully conveyed, and we find that off-the-shelf mt metrics are indeed sensitive to disfluency and summarization. our evaluation shows that the quality of human interpretations is worse than machine translations according to both automatic mt metrics and human evaluation. the low scores are caused not only by the sensitivity of mt metrics to interpretation features (as demonstrated in section 4.2), but also by persistent errors made by interpreters (as illustrated in section 4.1). given this finding, though recent work has argued that human interpretations should be used as references in simultaneous interpretation evaluation @xcite , we advise against using existing interpretations as references for evaluation. better data collection procedures and annotations are required to ensure that the interpretation data is of high quality. ultimately, though prior work has assumed the functionality of mt metrics for evaluating interpretation data, our findings reveal that minor disfluencies and summarization are unduly punished by existing metrics. in order to perform an accurate evaluation of interpretation data, these features must be addressed. we propose using fine-tuned learned metrics to assess interpretation quality. with human annotations, even flawed interpretation data can be used to fine-tune a model. as our results show, we are able achieve higher correlation with human judgments using our fine-tuned models than the state-of-theart mt metrics.