Assessing the Role of Context in Chat Translation Evaluation: Is Context Helpful and Under What Conditions? (2024)

Abstract: Despite the recent success of automatic metrics for assessing translation quality, their application in evaluating the quality of machine-translated chats has been limited. Unlike more structured texts like news, chat conversations are often unstructured, short, and heavily reliant on contextual information. This poses questions about the reliability of existing sentence-level metrics in this domain as well as the role of context in assessing the translation quality. Motivated by this, we conduct a meta-evaluation of existing automatic metrics, primarily designed for structured domains such as news, to assess the quality of machine-translated chats. We find that reference-free metrics lag behind reference-based ones, especially when evaluating translation quality in out-of-English settings. We then investigate how incorporating conversational contextual information in these metrics for sentence-level evaluation affects their performance. Our findings show that augmenting neural learned metrics with contextual information helps improve correlation with human judgments in the reference-free scenario and when evaluating translations in out-of-English settings. Finally, we propose a new evaluation metric, Context-MQM, that utilizes bilingual context with a large language model (LLM) and further validate that adding context helps even for LLM-based evaluation metrics.

## introduction automatically estimating the quality of machine or human-generated translations has received a lot of attention over the past two decades from the nlp community @xcite , specifically via shared tasks organized by wmt from 2014- * work partially developed during internship at unbabel. present @xcite @xcite @xcite @xcite @xcite . a variety of evaluation metrics have been developed for this purpose, encompassing lexical matching approaches such as bleu @xcite and chrf @xcite ; embedding-based methods like bert-score @xcite and word mover distance @xcite ; learned metrics like comet @xcite , and bleurt @xcite ; and metrics that employ prompting techniques with large language models (llms) like gemba-mqm @xcite or automqm @xcite . among these metrics, neural metrics have gained widespread acceptance @xcite as they are directly trained to predict sentence-level translation quality assessment scores @xcite @xcite , word-level error annotations collected by professional linguists @xcite , or post-editing efforts as measured by hter @xcite @xcite . however, the reliance on human-written reference translations and judgments collected predominantly from structured domains like news or wikipedia as training data raises questions about their adaptability and reliability in detecting errors in other domains @xcite , for example, in evaluating translation quality in more informal settings. unlike news articles, which involve carefully authored and well-formatted text, and which current translation systems are well equipped for, chat conversations are often synchronous and short, and involve formal language, colloquial expressions, and slang that may not have direct equivalents in the target language (gonc ¸alves et al. @xcite . an example of such a conversation is presented in figure 1 . although chat messages are relatively easier to translate, errors introduced by machine translation (mt) systems might go unnoticed by end users if not detected properly, potentially leading to miscommunication, conversation breakdowns, or even more serious implications on high-risk domains (e.g., patient-physician chats) @xcite . identifying these errors as they occur can potentially help the users ask clarification questions and correct any misunderstandings due to the mistranslation of the intended information @xcite . moreover, conversational texts rely heavily on context, meaning that the interpretation of a text is largely influenced by the surrounding contextual information. hence, mt systems for such domains are often trained with contextual in-formation and this has been shown to improve translation quality, lexical inconsistency, and coherence of the generated outputs @xcite . however, it is unclear to what extent context plays a role in estimating translation quality for machine-translated conversations. in this work, we first systematically analyze the nature and the frequency of the errors in real bilingual chat translations from customer support and contrast them with the structured news domain ( § 2). we find that translation errors are 21% less frequent in chat relative to the news domain and that the nature of the errors introduced by mt systems in the two domains is also different. this underscores the importance of understanding and evaluating existing automatic metrics for chat translation. due to the infrequent error occurrences at the turn level, mt systems for chat translation tasks might receive higher scores, necessitating a more nuanced evaluation of the metrics. motivated by the same, we present a metaevaluation of automatic metrics, primarily tested on news translation tasks, in their ability to gauge the quality of machine-translated chats at turn and dialogue level ( § 3). we use the multidimensional quality metrics (mqm) annotations collected by the wmt22 chat shared task on systems submitted for bilingual customer chat translations @xcite , where experts were asked to rate the quality of translation at each turn given the bilingual context, independently for all language directions. we evaluate the sentence-level metrics across two scenarios: on all translation pairs as well as imperfect translations as judged by humans. we then study the impact of augmenting a subset of the learned sentence-level neural metrics with different types of conversational contextual information ( § 4). our findings are summarized below: foot_0 finally, we present context-mqm, an llmbased evaluation metric that uses context for chat translation quality estimation ( § 5.3). our preliminary experiments with context-mqm show that adding bilingual context to the evaluation prompt helps improve the quality assessment of machinetranslated chats on imperfect translations. ## errors in chat vs. news: a case study to better understand how the error types differ in these domains, we present an analysis of the nature and the frequency of errors using mqm annotations (conversation: 7120, news: 4800 translation pairs) collected for english-german as a part of the wmt22 metrics shared task @xcite . errors are less frequent. we calculate the percentage of translations with a perfect mqm score from the news and conversational subset of the wmt22 dataset: only 46.4% of news translations have perfect mqm scores, whereas this percentage is 57.8% for the conversational domain. this suggests that errors are less frequent in the conversational domain, likely due to the relatively short (see figure 2 ) and probably less complex text dominant in conversations. however, it is important to note that these errors do not occur in isolation and can escalate into larger communication issues. moreover, if quality estimation metrics fail to accurately detect less frequent errors, they may misrepresent the true quality of the mt systems. most frequent error types across domains are different. errors related to fluency, such as issues with spelling, consistency, and register, occur more frequently in conversations compared to accuracy-based errors like mistranslation, which are more common in the news domain (figure 3 ). this underscores the idea that the nature of observed errors is also influenced by the specific domain context. the variation in error types and their frequencies across the two domains underscores the need for a systematic study of the suitability of current automatic metrics within the chat domain. ## meta evaluation of automatic metrics we now investigate to what extent automatic mt metrics capture chat translation quality. we ## context-aware translation evaluation let agent (a) and customer (c) represent the two participants in a bilingual chat. given a text x generated by a or c, the goal is to predict the quality of its translation, ŷ, given an optional reference translation, y. we extend comet-based metrics to utilize conversational context as detailed below: incorporating context comet uses pooled token-level representations extracted from pretrained language models (parameterized by f θ ) like bert @xcite or xlm-r @xcite to encode source sentences (x), hypotheses (ŷ) and the reference translations (y) into sentence-level contextual vectors ([f θ (x), f θ (ŷ), f θ (y)]). the concatenated feature vectors are then passed through a regression layer to generate a quality score. following @xcite , we obtain the token-level contextual representations of the current source, x, and the reference (y)/hypothesis (ŷ) sentences by prepending up to k sentences of context preceding to it. however, only the representations of the current instance are pooled before passing them to the regressor module that generates the quality score. we illustrate this for the reference-free comet-20-qe that uses source and hypothesis to produce a quality score in figure 4 for a context window of size 1. for comet-22, the regression layer can also access the reference vector (h y ). this simple approach was used to evaluate document-level translation quality and was shown to be more effective over sentence-level counterparts @xcite , whereas, in our work, we study and extend its applicability to contextualized sentence-level chat quality estimation. furthermore, this approach can be utilized with any metrics that aggregate or use contextualized token-level representations to generate a sentence-level representation, e.g., bertscore. ## results we first present the results of the meta-evaluation of existing automatic mt metrics. then we discuss the impact of adding context to a subset of sentence-level metrics with ablations on how context impacts translation quality evaluation. finally, we present a preliminary study on utilizing context with llm-based mt evaluation. ## conclusion and discussion estimating translation quality in diverse domains is crucial to ensure that the metrics employed in mt evaluation accurately reflect the mt system's quality across various types of content. we show that the nature and the type of errors in the conversational context are different from the generally evaluated news domain. hence, designing robust metrics that can capture these errors is very important. our work presents a step in that direction by systematically benchmarking existing automatic mt metrics on machine-translated chats. given the highly contextual nature of the chat domain, we extend and evaluate contextbased reference-free and reference-based metrics. best practices given our findings, we recommend the community adopt the following practices: alternatively, our proposed reference-free context-mqm metric based on gpt-4, can also be used as it outperformed reference-based comet-22 on the small-scale preliminary evaluation ( § 5.3). though, in the light of recent studies showing potential biases with llm-based evaluation -llms might favor their own outputs -we recommend using llm-based evaluation only in scenarios where the evaluating llm does not generate the translations @xcite .