Exploring the Correlation between Human and Machine Evaluation of Simultaneous Speech Translation (2024)

Abstract: Assessing the performance of interpreting services is a complex task, given the nuanced nature of spoken language translation, the strategies that interpreters apply, and the diverse expectations of users. The complexity of this task become even more pronounced when automated evaluation methods are applied. This is particularly true because interpreted texts exhibit less linearity between the source and target languages due to the strategies employed by the interpreter.This study aims to assess the reliability of automatic metrics in evaluating simultaneous interpretations by analyzing their correlation with human evaluations. We focus on a particular feature of interpretation quality, namely translation accuracy or faithfulness. As a benchmark we use human assessments performed by language experts, and evaluate how well sentence embeddings and Large Language Models correlate with them. We quantify semantic similarity between the source and translated texts without relying on a reference translation. The results suggest GPT models, particularly GPT-3.5 with direct prompting, demonstrate the strongest correlation with human judgment in terms of semantic similarity between source and target texts, even when evaluating short textual segments. Additionally, the study reveals that the size of the context window has a notable impact on this correlation.

## introduction the assessment of interpreting quality is a common practice in both professional interpreting and academic contexts. the results of these evaluations offer valuable insights for a wide range of stakeholders, including interpreter's clients, users, practitioners, educators, certification bodies, and researchers @xcite . assessing quality in interpretation is a complex endeavor. quality is not only challenging to measure, but it manifests also an "elusive nature" @xcite making it difficult to define. the notion of quality in fact may vary from one user to another, introducing a substantial degree of subjectivity in determining what constitutes a good translation of speech. furthermore, the criteria for quality are contingent upon the type of interpretation involved. for instance, in conference interpreting, the emphasis is generally on the quality of the interpreter's output, encompassing aspects such as content, language, and delivery. in contrast, within community settings like social and healthcare interpreting, interactional competencies and discourse management play a crucial role in determining what quality is @xcite . traditionally, the assessment of interpreting performances has been carried out manually, a methodology that comes with its own set of pros and cons. on the positive side, human evaluations offer a holistic view of quality by taking into account various facets of the communication process, thereby delivering a more nuanced understanding of interpreting performance @xcite . conversely, manual assessment comes with its own set of challenges, including being labor-intensive, time-consuming and costly @xcite . furthermore, the results often have limited generalizability due to either the restricted scope of the data sampled or the inherent complexities associated with evaluating spoken translation. in light of the limitations, there has been a growing interest to apply automatic metrics to the evaluation of interpreting performances. while traditional statistical metrics like bleu have shown limited efficacy in capturing translation quality from a user's perspective, the emergence of semantic vectors and pre-trained, large-scale generative language models has yielded promising results, especially in the domain of written translation @xcite . the application of these metrics is gradually extending to the field of spoken translation as well @xcite . however, it must be mentioned that orally translated texts possess certain characteristics that might restrict the efficacy of employing metrics designed for written texts. interpreters, especially in the simultaneous modality, tend to alter the text more extensively than translators, modifying the structure and omitting parts deliberately as a strategy rather than a deficiency. for example, interpreters may omit part of the original when experiencing cognitive overload, when they cannot comprehend the original message, to name just a few @xcite . this non-linearity between source and target texts renders the task of automatic evaluation even more challenging. the adoption of easy accessible and robust automatic evaluation in interpreting offers several potential applications that could benefit a wide range of stakeholders. firstly, the ability to provide instant feedback to trainees and practitioners would enable them to quickly assess their performance and pinpoint areas for improvement, also in realtime, creating a faster feedback loop that could substantially accelerate autonomous skill development. secondly, automatic evaluation might aid organizations in consistently and objectively monitoring the quality of their multilingual services. thirdly, automatic metrics that correlate with human judgments might serve as a useful tool for the continuous evaluation of machine interpretation. this study addresses two primary questions: first, is there an automatic metric that aligns closely with human judgment and can thus be used to automate the accuracy evaluation of spoken language translation? second, do these metrics evaluate human-generated translations, machinegenerated translations, or both more effectively? the rest of the paper is organised as follows. in section 2 we present an overview of research in the field of automatic evaluation of interpreting performances. in section 3, we illustrate our research methodology, our data and the experimental design. section 3.1 describes the dataset created for this task. section 3.2 describes the process for human evaluation of the translations while section 3.3 delves on the process followed for the automatic evaluation. section 4 presents the results. section 5 introduces some ethical implications. finally, section 6 concludes the paper with some discussion and remarks. ## related work the evaluation of translation quality and in particular of accuracy or information fidelity, i.e. the correspondence between source-language and target-language renditions, has traditionally differed between computer science, with its tradition of abundant use of automatic metrics, and translation and interpreting community, with its focus on manual evaluation as perceived by experts and users. in computer science, evaluation metrics such as bilingual evaluation understudy (bleu) @xcite , national institute of standards and technology (nist) @xcite , metric for evaluation of translation with explicit ordering (meteor) @xcite , and translation edit rate (ter) have been foundational in establishing benchmarks for machine translation quality estimation (mtqe). bleu and nist emphasise n-gram precision, with nist uniquely weighting distinct n-grams. meteor integrates both recall and precision, while ter quantifies requisite edits for optimal translation alignment. however, recent scholarly discourse have suggested that these metrics, while valuable, may possess intrinsic limitations in encapsulating the multifaceted subtleties and overarching context of linguistic structures @xcite . this acknowledgement has precipitated the exploration of advanced, data-driven methodologies for mtqe without references. neural networks, characterized by their bio-inspired architectures, emerge as a compelling alternative. these computational structures excel in managing voluminous datasets, discerning intricate patterns, and, crucially, accounting for the inherent complexities associated with linguistic phenomena. in the field of neural network architectures, the potential of recurrent neural networks (rnns), convolutional neural networks (cnns), and the groundbreaking transformer for semantic similarity computations has been explored. of these, transformer-based models like bert and gpt have gained considerable academic traction due to their outstanding performance across numerous natural language processing (nlp) tasks @xcite @xcite @xcite @xcite @xcite . central to these architectures is the concept of embeddings: dense vector representations that capture the semantic essence of words or textual segments. within this high-dimensional space, vectors situated closely denote semantic relatedness. in translation evaluation, embeddings offer a mediating semantic layer, enabling comparisons between source and target linguistic structures. however, the embeddings landscape is complex. models from the universal sentence encoder multilingual (usem) to the generative pretrained transformer (gpt) produce embeddings with varied purposes. for example, usem is geared towards retrieving semantically aligned entities, while gpt emphasizes generating contextrich linguistic constructs. these nuances highlight the need for researchers to thoughtfully choose models aligned with their specific research goals. in interpreting studies, a subdomain of translation studies dedicated to oral translation, the traditional practice has been to assess accuracy manually, with or without references. many of the evaluation methodologies are derived by written translation. for the assessment based on references, also known as intra-lingua assessment, the notion of "tertium comparisons" stands as a pivotal benchmark within a particular language @xcite . tracing the historical evolution of this methodology, @xcite stands out as a foundational contributor. he experimented with lay people to ascertain accuracy in translations. building on carroll's scale, @xcite refined the process, integrating references to spoken language and interpreting for intra-lingual assessment. these approaches, while receiving considerable acceptance from seasoned interpreters, are not without limitations. the impact of cognitivelinguistic factors can potentially alter evaluation results @xcite . moreover, intralingual evaluations face challenges in adapting to changing contexts and demographics and may lack a universally acknowledged point of reference @xcite . academic discussions prioritise evaluation methods for gauging accuracy in inter-language interpreting. these methods range from error analysis, as seen in works by @xcite and @xcite that identify translation inaccuracies, to propositional analysis, endorsed by researchers like @xcite and @xcite @xcite , which examines textual accuracy. however, these methods present challenges in addressing linguistic subtleties and differing interpretations. more recent research emphasizes grading rubrics, tracing back to @xcite , which outline performance across competency tiers and are validated in multiple studies @xcite @xcite . yet, even with proven reliability, this rubric-based evaluation faces hurdles like the development and validation of rubric descriptors and evaluator inconsistencies. a few studies have explored the effectiveness of various metrics in evaluating the translation quality or interpreting performances. @xcite , for instance, pinpointed the strong alignment between human evaluations and scores determined by bleu and meteor for german-to-korean translation. subsequent studies by @xcite and @xcite reinforced the merit of these automated tools. @xcite discerned that meteor's sentence-level evaluations resonated more with human assessments than broader, text-level evaluations. conversely, lu and han's (2022) exploration, fortified with the integration of the bert model @xcite , showcased substantial correlations between human and automated evaluations, underlining the potential of these metrics in assessing interpreting performances. a recent study by @xcite employed large language models like gpt to evaluate translation quality across three language pairs, concluding that only models gpt3.5 and above possess the capability for such translation quality assessment. while initial studies have underscored positive and moderate-to-strong correlations for mt met-rics such as bleu and meteor, to the best of our knowledge no research has been conducted so far on the use of language models for referencefree interpreting assessment. our study aims to fill this gap. ## results to analyse the data, we devised charts from three perspectives, including the comparison of correlation values among evaluation methods, a comparison between translation h and translation m, and correlation values based on window size. in figure 1 we compare the distribution of correlation values across all machine evaluation methods, namely gpt-3.5, all-minilm-l6-v2, gpt-ada, and umse, for both translation h and translation m. the correlation with gpt-3.5 displays the highest median correlation value. the interquartile range (iqr) is also quite narrow, indicating that the correlation values for this method are consistently high and well-aligned with human evaluations. the correlation with all-minilm-l6-v2 has a wide iqr, showcasing varied performance. the median value is close to zero, but there are negative outliers, indicating instances where the machine evaluation is inversely related to human judgment. the correlation values with gpt-ada are relatively consistent, with a narrow iqr. the median is slightly above 0.3, which indicates moderate alignment with human judgments. umse's performance seems to be close to gpt-ada with a median slightly above 0.3. the iqr is a bit larger, suggesting a bit more variability in the correlation values. for the comparison between translation h and translation m in figure 2 , paired bar charts elucidate the average correlation disparities for each machine evaluation method. gpt-3.5's measurements exhibit robust correlation values with human judgments for both translations, although translation h marginally outperforms translation m. for all-minilm-l6-v2, the correlation of translation h gravitates towards zero, whereas translation m registers a negative value, implying a potential misalignment of the all-minilm-l6-v2's evaluations with human perspectives, predominantly for translation m. gpt-ada embeddings yield nearly identical correlation values for both translations, but with translation h slightly edging out. intriguingly, umse's embeddings produce a higher correlation value for translation m compared to translation h. turning to the third perspective, which examines the shift in correlation values based on window size, line charts in figure 3 and figure 4 offer insights into this dynamic for each machine evaluation method. for translation h (human translated) in figure 3 , gpt-ada correlation with human ratings sees a mild fluctuation across window sizes, initially decreasing from window size-1 to size-2, then slightly rising in the following window size2-5. the all-minilm-l6-v2 model, in contrast, exhibits a downward trend, indicating reduced agreement with human evaluations as window size grows. umse consistently maintains a stable correlation with human ratings, showing only minor variations across different window sizes. gpt-3.5 presents a distinct pattern; while its correlation initially drops from size-1 to size-2, it surges notably in the subsequent window, outperforming the other models. in observations for translation m (machine translated) in figure 4 , gpt-ada begins with a positive correlation with human ratings with size-1, but this declines as the window size expands, hinting at potential metric inconsistencies for broader contexts. the all-minilm-l6-v2 model's correlation, on the other hand, commences positively by size-1 and consistently rises with the window size, pointing to a more aligned evaluation with human judgment for larger translation segments. umse's performance mirrors its evaluation with human translations, maintaining stability across all window sizes and showcasing its consistent metric evaluation. in contrast, gpt-3.5's correlation fluctuates considerably across window sizes -experiencing a drop, a subsequent rise, and then another decline -indicating a variable level of concurrence with human assessments depending on the window size. ## ethical considerations the adoption of automatic evaluation also raises ethical concerns that warrant careful consideration. one potential issue is the possibility to continuously monitor interpreters, which might infringe on privacy rights and create a sense of constant surveillance, negatively impacting job satisfaction and professional autonomy. additionally, decisions regarding the employability of individuals could be blindly given to mechanical means, potentially leading to unjust or biased outcomes if the algorithms fail to account for contextual nuances or other essential aspects of human communication. as such, it is crucial for the language industry to carefully weigh the benefits and challenges of automatic evaluation, ensuring that ethical considerations are addressed as advancements in ai technology continue to reshape the landscape. ## conclusions this study aimed to analyze the correlation between automated and human evaluations of translated content. the peculiarity of this experiment is that we focused on a specific form of translation: the simultaneous interpretation of english speeches into spanish. this mode of translation introduces unique challenges to assessment due to the nonlinear nature of the output (in spoken translation, the differences between the source and target can be more pronounced than in written translation) and varying user expectations regarding interpretation quality. we evaluated both interpretations provided by professional interpreters and those produced by a machine interpretation system. the objective was to develop a metric reflecting interpreting quality in a manner consistent with human judgment. the direct prompting of gpt-3.5 for quality estimation on a likert scale exhibits the highest median correlation with human evaluation. this finding establishes gpt-3.5 as the most promising tool among the evaluated methods to gauge translation quality, both for interpretations produced by humans and machines. gpt-3.5 benefits from a larger context, performing better with larger segment windows. this suggests that the model can capture and evaluate long dependencies more effectively. contrary to expectations, gpt-3.5's correlation with human judgment is somewhat stronger for translations produced by professional interpreters than for machine-generated ones. this implies that gpt might be subtly more attuned to the linguistic nuances of human translation, even though it remains adept at evaluating speech translation. one possible explanation for this is that human interpreters often introduce subtle contextual, tonal, and idiomatic adjustments that are more aligned with gpt-3.5's training on diverse data, whereas machine translations might adhere more strictly to equivalences. looking forward, further research could explore deeper into the characteristics of the datasets used for training such models and their alignment with real-world interpretation tasks. this study presents several limitations. the observed low interrater agreement suggests potential inconsistencies in human evaluations, possibly affecting correlation values, and generability of the results. furthermore, the limited scope of the sampled translations might not capture the full range of linguistic complexities inherent to interpretation. future research should consider evaluations for higher window sizes. in light of gpt-3.5's performance in this study, future research might explore its ability to delineate nuances of typologies of errors rather than merely providing aggregate scores this study is considered a preliminary attempt to test the feasibility of applying automatic metrics to evaluate inputs from both human and machine interpreters. before these metrics can be used in production, more research needs to be conducted.