Evaluation Briefs: Drawing on Translation Studies for Human Evaluation of MT (2024)

Abstract: In this position paper, we examine ways in which researchers in machine translation and translation studies have approached the problem of evaluating the output of machine translation systems and, more broadly, the questions of what it means to define translation quality. We explore their similarities and differences, highlighting the role that the purpose and context of translation plays in translation studies approaches. We argue that evaluation of machine translation (e.g., in shared tasks) would benefit from additional insights from translation studies, and we suggest the introduction of an ‘evaluation brief” (analogous to the ‘translation brief’) which could help set out useful context for annotators tasked with evaluating machine translation.

## introduction the evaluation of translation quality remains a challenge in the fields of machine translation (mt) and translation studies (ts). evaluation methods relying on human judgement have changed and developed alongside advances in machine translation technology. in mt, the longstanding goal of these evaluation approaches has been to provide a standardized and possibly even "objective" evaluation process. in this work, we will draw on complementary perspectives from mt and ts. 1 we will show that there are similarities and connections between the fields' views on evaluation, as well as areas where insights from ts could be used to inform and improve approaches to human evaluation of mt. controversies resulting from claims that mt quality has reached "parity" with humans @xcite as well as problems with human evaluation campaigns at the workshop on machine translation (wmt) have led to mt researchers shin-ing a spotlight on evaluation protocols and their challenges @xcite @xcite )this has also piqued the interest of researchers in ts (e.g., @xcite . while many previous works in the mt literature on problems in human mt evaluation have examined questions like how to set up the evaluation process, how to incorporate context, and how to standardize annotator scores, it is rarer for them to focus specifically on the definition at the core of this process: what is translation quality? in this paper, we investigate how perspectives on that question differ between researchers in the fields of mt and ts. we begin with mt researchers' perspectives on current methodologies in human evaluation, focusing on what attributes of quality these evaluations prioritize. we then present a view of theoretical and practical dimensions of translation quality assessment (tqa) within ts in academia and industry. we explore tqa models in academia such as house's tqa model, williams' argumentation-centered approach, colina's approach, and industry-driven approaches like multidimensional quality metrics (mqm). following a comparative analysis of these methodologies, we highlight one particular facet of quality evaluation that is present in the tqa models but frequently absent from the mt research approaches: an acknowledgement of the purpose of a given translation and the context in which it is produced and expected to be used. we take the position that mt research could benefit from incorporating these ts perspectives, and we conclude this work by considering how this could be done in practice. translators are sometimes given a "translation brief" describing the goals of the translation, the intended audience of the translation, and other important contextual information; we propose an analogous "evaluation brief" to serve a similar role in human evaluation of translation. we then discuss how this additional context could be implemented in practice in mt evaluation, including the importance of being aware of how the annotator population differs or is similar to the expected end users of the mt system being evaluated (e.g., in terms of subject area knowledge, dialect, context, etc.), as problems in evaluation could arise due to a mismatch. ## mt researcher perspectives since early mt experiments, human evaluation has been positioned as the ideal form of evaluation, with automatic metrics seen as a necessary standin. even bleu ("bilingual evaluation understudy"), in its name, considers automatic metrics an "understudy" to human evaluators @xcite . how those human evaluations should be produced has been an open question, with a rotating cast of proposed methodologies and definitions of quality. the methodologies, procedures, and interfaces used for collecting evaluations include sliders with a continuous scale, discrete scales, ranking, annotations of the text, among others. these are often discussed in conjunction with the aspects of quality being considered (e.g., discrete scales for adequacy and fluency 2 ), with terminology surrounding methodologies and interfaces blurring the line between the interface itself and the questions that annotators are being asked about quality. but in practice these are orthogonal concerns; various interfaces could be paired with any number of questions about different aspects of quality. this paper categorizes human evaluation methodologies used in mt evaluation into three broad (and sometimes overlapping) groups: manual scoring, semi-automatic (or, from another perspective, this could be viewed as semimanual), and task-based. we also touch on how these groups of evaluation methodologies typically address questions of quality, and which aspects of quality are regularly considered in mt human evaluation. we define "manual scoring" evaluations as evaluations in which an annotator directly provides a score or ranking to one or more systems. early evaluations at shared tasks asked annotators to judge adequacy and fluency on 5-or 7-point scales @xcite @xcite and this approach is frequently revisited in other proposed variants, such as the rating of semantic faithfulness to source text in @xcite . in later system ranking tasks @xcite @xcite , annotators were asked "to rank the translations from best to worst (ties are allowed)" @xcite , without specific guidance about what aspects would make one translation better or worse than another. ence in appendix a. in addition to their similarity in terms of annotators directly providing some sort of score for a translation, these approaches have all been challenged at various points due to issues relating to inter-and intra-annotator consistency. we define our second category of "semiautomatic" evaluations as ones in which an annotator provides some sort of annotation (or transformation like post-editing) to the text, and then a score for the mt is computed (automatically) based on the annotations. this includes approaches such as humantargeted translation edit rate (hter; @xcite , where the edit rate (an automatic measure of the number of changes/difference) of mt output is measured against the (human) post-edited version, rather than a generic reference, with the expectation that that mt output with higher translation quality requires fewer edits in order to produce an acceptable post-edited translation. other approaches involve the annotation of errors using an error typology (e.g., multilingual quality metrics, mqm; burchardt, 2013) followed by computing a score based on the number and severity of errors. similar approaches, such as hmeant @xcite and hume @xcite , involve annotating the shallow semantic structures/units and translation correctness of each semantic unit in the mt output, followed by aggregating these correctness annotations into a score for the translation quality at sentence level. semi-automatic approaches tend to have more well-defined instructions for annotators. however, even these may have ambiguities in the interpretations of the evaluation task. al sharou and specia (2022) described challenges in consistency of annotating critical errors using an error typology, noting the importance of annotator training while also acknowledging that ambiguities and confusion may nevertheless persist. @xcite and @xcite both showed that there are compounding disagreements between annotators at the end of the evaluation task using hmeant and hume. the third group consists of task-based evaluations. in this type of extrinsic evaluation, annotators are asked to use mt output to perform a task, e.g., template filling in @xcite , question answering in @xcite , semantic pars-ing in @xcite , etc.; the performance on the downstream task is scored. these scores are interpreted as the usefulness of the mt output for the downstream task and used to form a score or ranking of the translation quality of the underlying mt system. these task-based approaches typically do not ask annotators to directly judge aspects of mt quality. instead, they emphasize the utility/usefulness aspect of the translation and implicitly ask "is the quality of the mt good enough for the annotator to perform the requested task?", "does one mt system better enable annotators to complete the task than another mt system?", or similar questions. as far as we can tell, the specific form of the questions posed/directions given to annotators are (with a few exceptions) rarely studied by mt researchers in order to ensure their validity or reliability. in general, despite its goals of producing "objective" scores, human evaluation in mt research has tended to focus either on high-level and potentially undefined or underspecified aspects like generic "quality", divided quality into adequacy and fluency, defined mt performance based on downstream task performance, or used error typologies. @xcite raised the question of whether identifying the "components" of quality that annotators used in their decisions could help to improve the reliability and validity of future evaluations. ## translation studies (ts) perspectives we now explore perspectives on translation quality and translation quality assessment (tqa) foot_0 from ts academia and industry. academic research in ts often explores theoretical frameworks, pedagogical implications, and methodological innovations @xcite , and the translation industry tends to focus on operational efficiency, quality assurance, and client satisfaction, frequently employing quantitative measures and standardized processes to ensure consistency and reliability in translation outputs @xcite @xcite . @xcite and @xcite note the challenges of tqa in practical settings, with an eye toward realworld applicability, often within the constraints of tight timelines and specific client needs. although there is a significant body of research within ts that is process-oriented @xcite , in this paper we focus on product-oriented aspects of translation, as these align more with the areas of mt evaluation research that we also examine. ## overlaps and differences in mt and ts the fields of mt and ts do have points of commonality when it comes to human evaluation and assessment of mt quality, while the areas where they differ may have their origins in the underlying objectives and methodologies of the fields. this is noted in @xcite , who point out that many researchers in ts "have argued that evalua-tion is directly associated with the underlying translation theory that one subscribes to," citing in particular the quote that "different views of translation lead to different concepts of translational quality, and hence different ways of assessing it" @xcite . we also note that even within each research community-and in their areas of overlap-there is not a broad consensus on how to define quality or which aspects ought to be considered most important. there are two main forms of evaluations that overlap between the two fields: error typologybased and task-based evaluations. mqm, developed in the translation industry and ts and recently adopted by mt for some evaluations @xcite @xcite , breaks mt quality down into a typology of errors. task-based and recipient evaluations have also been used in both fields. in task-based evaluations, we see the use of mt presented in a particular context, with the users asked to either perform a task or evaluate it from the perspective of their use case. these may come the closest to examining whether or not translations are appropriate for the situations and contexts for which they are intended. both fields have recognized the challenge of defining and assessing translation quality, though they have largely taken different approaches in exploring this. at various points in mt evaluation history, we have seen quality broken down into component parts at different levels of granularity (e.g., adequacy and fluency, or error typologies like mqm). perspectives from ts provide other ways of categorizing the components that come together to make up notions of "quality". these include extra-textual factors that influence quality, as well as borrowing and incorporating understandings of quality from different disciplines, such as functionalism, industry, and management. but in all of these efforts, we see that quality is multidimensional (i.e., made up of various contributing aspects) and situationdependent; there is not a straightforward simple or universal definition. mt research often seeks a single "objective" metric for mt "quality", which can be used to compute a simple ranking of systems. this is connected to the leaderboard and competition aspects that are common to mt and other areas of machine learning. it also directly relates to the fact that system optimization is a major focus in mt research: optimization towards a single objective is substantially easier than optimization towards multiple (potentially conflicting) objectives. in mt, the response to observing annotator variation in evaluation has often been to modify the evaluation protocols (e.g., changing from a rating scale to ranking to direct assessment) or to seek ways of standardizing annotator scores. both mt and ts have considered the question of who should perform annotations. in largescale mt evaluations, this has often been constrained by the cost of annotation, with interest in crowdsourcing @xcite @xcite and comparing crowdsourced results against language and translation experts. in ts, @xcite also examines this question of who should perform annotations, finding greater levels of inter-annotator agreement within homogenous groups (e.g., groups of all professional translators or groups of bilinguals who are not translators). in ts, we see more attention paid to the meaning of quality itself and how best to define that, influenced by definitions and descriptions of quality from different disciplines. ts also tends more towards exploring the notion of subjectivity, with a greater focus on the specific use cases and users of a particular translation and how that translation serves its purpose. this focus on a specific use case can be seen as a difference from mt research, which often purports to aim for a broad or universal use case (see, e.g., the framing of tasks at wmt, such as "news" or "general" translation, without reference to a specific audience for the news/general translations). 7 in ts, there is a significant focus on who the translation is for, what it is intended to do, and the specific circumstances surrounding its creation and use @xcite @xcite . this approach ensures that translations are tailored to meet the needs and expectations of their target audience. this contrasts with some mt research proposals of a translation that can be used in any context by anyone. this mt perspective may be tied to underlying assumptions of invertibility as a desired component of mt (since round-trip translation performance has frequently been used as a benchmark of success by mt researchers), a view which is not shared in all of ts. a recent concept from the mt perspective assumes that a single translation can meet all purposes or that there exists a general-purpose translation, which is often unrealistic given the diversity of language use and cultural contexts. 8 while the concept of a universal translator has long been a goal of some researchers, we note that, over time, mt research has taken various views on how best to approach translation. early mt successes such as the meteo systems @xcite ) occurred through focused efforts on limited and specific domains: purpose-built mt. the late 1990s and early 2000s saw the widespread availability of free public online mt systems, such as altavista babel fish @xcite , allowing anyone with an internet connection to (attempt to) translate anything within a limited set of language pairs. mt research has seen both these research tracks-the purposebuilt task-specific translation system and the goal of a universal system-pursued in parallel. when researchers or users treat online mt systems, for example, as a box into which any source text can be placed with the expectation of receiving the desired translation, conflict and disappointment are likely to arise. users of mt technologies are in fact using mt with a purpose, and two users of the same mt system may be using it with two different and 7 we do note some exceptions to this, such as the specification that the 2024 english-spanish task is intended to translate into latin american spanish, specifically (wmt, 2024), though one could argue that this still covers a wide range of language variants. we argue that mtr research should be considering these purposes and specificities when performing evaluations, whether by explicitly highlighting specific use cases, language variants, and so on, or by being clear about how to handle conflicting preferences in translation quality. ## evaluation briefs in this work, we have looked at how both mt and ts have explored questions of what it means to evaluate the "quality" of a translation. while we have seen that mt has explored some aspects of quality (e.g., adequacy and fluency), ts has enumerated a wider range of aspects that contribute to perceptions and judgments of mt quality; tqa involves decisions that take into account many factors beyond the source and target text, such as the intended target audiences and their linguistic and cultural background, the purpose of translation, and the medium of reception. without access to these relevant details, human evaluators are reasoning under uncertainty. this leads us to ask: can insights from ts suggest to us aspects that are missing from many of the current implementations of human evaluation of mt from the mt research side? we argue that the concepts of the purpose and intended audience of a translation are some of the central aspects that have been underexplored in the mt literature. this is also one of the major research areas identified in human-centered mt evaluation by @xcite . trying to incorporate this into mt evaluation (e.g., of the sorts performed at wmt or other large-scale evaluations) will require mt researchers to first settle on more concrete and well-defined goals for their mt systems. that includes the considerations of the intended use case, the language variants, and the intended audiences. this is certainly not a new call; @xcite pointed out that "if the application is not clearly identified (or worse, if the application is poorly chosen), then it is often very difficult to find a satisfying evaluation paradigm." that claim was made in an era of "crummy" mt, but we argue for its continued relevance in an era of improved mt. among several other goals, @xcite argued that an appropriate application should "set reasonable expectations" and "should be attractive to the intended users". now that we have access to much-improved mt for many language pairs and domains, how should we push forward? we propose being explicit with a "translation brief" (for the use of both the translators producing reference translations and the researchers building mt systems 9 ) as well as expanding this to an "evaluation brief". an "evaluation brief" would provide the human evaluators with a wider context and detailed instructions about how to evaluate the translation. this is similar to the "role" or "persona" described in @xcite , which annotators are asked to take on when evaluating mt output; that work also highlights the importance of taking great care with the design of such instructions. with the evaluation brief, human evaluators could situate themselves in the use case of the translation and as the intended users of the translation to consider the users' needs and expectations. as for what to include in an evaluation brief, we could draw inspiration from the translation brief: source and target languages (including language variants), relevant information about both the author/speaker and the audience, purpose, style guide, and so on @xcite . for example, we can consider two different types of medical texts: medical information that is intended for healthcare workers (domain experts) and medical information in public health announcements that is intended to be accessible to a broad audience (non-experts). 10 an appropriate evaluation 9 while human translators will make use of the translation brief directly, i.e., deciding on levels of formality, language variants, technical language, and so on to use in their translations, mt researchers are likely to use this more indirectly, such as by selecting which data sources to train on, deciding whether to incorporate model features such as tagging (e.g., for multi-domain or multilingual systems), considering issues such as robustness to input variations, and so on, with the goal of producing a translation system that in turn will follow the translation brief. we could also imagine employing translation briefs when experimenting with large language model-based translation, as part of the instructions provided to the large language model. 10 we consider here primarily the case in which the expertise level of the audience is held consistent from the source to the target (i.e., translating text for domain experts from a source language into target language text also intended for domain experts); the transformation of text from expert to non-expert (or vice versa) introduces additional challenges. ## conclusion in this work, we have examined perspectives on both mt quality and how to evaluate mt from the perspectives of mt research and translation studies. we argue that future mt evaluation could benefit from drawing on insights from translation studies. in particular, this includes an increasing focus on the purpose, intended audience, and context of translation. more broadly, we encourage mt researchers to seek collaborations and conversations in ts and beyond. in order to better design the questions that mt researchers ask of evaluators, the field would likely benefit from more interactions with research best practices in measurement theory, survey research methods, human-computer interaction, and more.