Opportunities for Human-centered Evaluation of Machine Translation Systems (2022)

Abstract: Machine translation models are embedded in larger user-facing systems. Although model evaluation has matured, evaluation at the systems level is still lacking. We review literature from both the translation studies and HCI communities about who uses machine translation and for what purposes. We emphasize an important difference in evaluating machine translation models versus the physical and cultural systems in which they are embedded. We then propose opportunities for improved measurement of user-facing translation systems. We pay particular attention to the need for design and evaluation to aid engendering trust and enhancing user agency in future machine translation systems.

## introduction evaluation of machine translation (mt) system performance seeks to close the gap between automated measurements and human judgments of translation quality. the mt evaluation literature contains a variety of work to quantify how closely human and automated measurement are linked. the models being evaluated in this literature are often complex combinations of traditional nlp and neural components. in this paper, we consider these systems as major but separate components of larger user-facing translation systems embedded in larger social contexts. these translation systems serve diverse user needs that go beyond simple translation of text to text. grounded in research across domains, we motivate new research in three specific areas. first, translation system users, their motivations, and usage practices deserve further inquiry. second, we argue that end-users (both individual and institutional) evaluate system performance differently from the system developers. end-users necessitate evaluation of the entire system, rather than just the model. finally, we argue that significant research is necessary around nurturing trust in mt. ## machine translation users web-and app-based translation services, as well as translation apis, are widely available. people who use these systems span geographic, socioeconomic, and occupational categories. we first discuss the contexts of consumers, a group of people who encounter machine translation as part of transacting their daily lives, but not as part of their vocation. then we discuss the needs of translation professionals, people who use machine translation as part of their occupational workflows. ## evaluating translation systems evaluating translation systems used by humans is difficult because the task, system design, and model performance are intertwined. ## trust in translation systems although some argue that mt has reached "human parity," many challenges remain. output translations still contain errors, especially for lower resource languages. while minor errors in low-stakes contexts may be inconsequential, the unpredictability and lack of control over mt output raises concern regarding these systems' reliability in real world settings. researchers and users have also documented systematic errors and biases in mt system output, which can result in longer term, representational harms to minoritized and oppressed groups @xcite . in this section, we review important concerns in the design and use of safe, reliable, and useful machine translation systems. in the following section we identify open research questions at the intersection of hci and mt that can begin to address some of these challenges. systematic errors and biases have been documented across nlp models and tasks. for example, bolukbasi et al. demonstrated that word embeddings can learn stereotypical associations between gender and occupations @xcite . machine translation models also struggle with semantic gender; for example, difficulties arise when translating a sentence from a language with genderneutral pronouns to one with gendered pronouns while avoiding reinforcement of gender stereotypes @xcite . in 2021, a major web translation site addressed this by allowing users to select from multiple target translations with different genders. foot_0 this gives the user agency to control the output of the system, compensating for issues with the underlying model. ## implications on future hci+mt research in this paper, we have examined groups of translation users and the role that mt plays in their lives. common needs that cut across these user groups include reliability, trust, and agency, and we examine these in more detail here. ## conclusion human communication is subtle and complex. for high-resource languages, machine translation models can successfully map many phrases across languages with high fluency and adequacy. we believe that the next frontier of research in machine translation is, through a human factors lens, to bring modern themes of trust and agency to the table. first, more robust qualitative and quantitative research is necessary to understand how people interact with mt systems today, and what improves the quality of those interactions. machine translation users vary from professionals incidentally using mt as part of their workflow, to immigrants with low language proficiency who rely on mt systems to survive. from a design and implementation standpoint, we acknowledge that mt user interfaces have evolved very little. given that people use machine translation to communicate in highstakes scenarios, there is a great opportunity to