Quality in Human and Machine Translation: An Interdisciplinary Survey (2023)

Abstract: Quality assurance is a central component of human and machine translation. In translation studies, translation quality focuses on human evaluation and dimensions, such as purpose, comprehensibility, target audience among many more. Within the field of machine translation, more operationalized definitions of quality lead to automated metrics relying on reference translations or quality estimation. A joint approach to defining and assessing translation quality holds the promise to be mutually beneficial. To contribute towards that objective, this systematic survey provides an interdisciplinary analysis of the concept of translation quality from both perspectives. Thereby, it seeks to inspire cross-fertilization between both fields and further development of an interdisciplinary concept of translation quality.

## introduction translation quality has been a source of debate in translation studies for decades @xcite , since it is considered highly subjective and dependent on how translation and quality are defined. one common denominator is the central role played by accuracy and fluency @xcite , a view shared by the field of machine translation @xcite ). an accurate semantic correspondence between source and translation as well as an adequate degree of fluency in the latter are expected. aside from these shared notions, approaches to define, assess and measure translation quality differ substantially in the field of translation studies and machine translation. this interdisciplinary survey analyzes literature on translation quality from both perspectives. the idea to join the theoretical basis of translation studies with the operationalized quality definitions of machine translation is not new @xcite . however, existing surveys on the topic focus either on machine translation @xcite , post-editing @xcite or the perspective of translation studies @xcite . from a theoretical perspective, @xcite present key quality theories from both fields and argue that the line between human and machine translation is increasingly blurring, especially in post-editing. the multidimensional quality metric (mqm) @xcite proposes a comprehensive catalog of quality issues, which can be used to calculate a score for evaluating translations. inspired by the prisma method @xcite and guidelines by @xcite , this paper presents a systematic literature review on translation quality in the field of translation studies and machine translation. resulting publications are deduplicated and ranked by a keyword rating method that takes the number of occurrences across platforms and keywords into account. the resulting top 41 publications are presented based on the authors' fields and translation quality perspective. thereby, the present survey contributes an overview of types of translation quality per field and interdisciplinary publications in the result set. it seeks to provide a basis for more crossfertilization between human and machine translation quality analysis. as a basis for the following discussion, we provide a very brief introduction to selected concepts of translation quality in translation studies and machine translation (see e.g. @xcite for a more complete overview). an initial criterion of equivalence in translation studies, that is, a very close correspondence between source text and translation, was soon found too vague for a targeted quality assessment. thus, a functionalist approach, the skopos theory @xcite , proposed to focus on preserving the purpose of the source text in the translation. @xcite deems it difficult to exactly determine the purpose and proposes to divide a text into register and genre, each further subdivided, for a detailed analysis of categorybased equivalence. with more attention on the recipient of the translation, criteria such as readability and comprehensibility were introduced. for instance, @xcite proposes several dimensions of comprehensibility, that is, concision, correctness, motivation, structure, simplicity, and perceptibility. in machine translation, the main differentiation is between automated and human quality measurement. in the former, some well-known evaluation metrics based on human reference translations are bleu @xcite and meteor @xcite . other automated methods take linguistic features into account, e.g. syntactic features @xcite and semantic roles @xcite . one major drawback is that these approaches rely on nlp techniques with limited availability for natural languages. with document-level approaches, criteria such as cohesion and coherence @xcite enter the field. to overcome the need for reference translations, machine translation quality estimation (mtqe) @xcite has been proposed, especially for neural machine translation (nmt). mtqe tasks include extracting features from source text and translation, selecting translations fit for post-processing, selecting the best translation between several mt systems, among others. human quality assessment of mt focuses on categorizing segments or parts by specific criteria, e.g. comprehensibility and adequacy @xcite , however, is generally considered subjective and time-consuming and should be conducted by professional translators @xcite . ## method the objective of this systematic literature review is to provide an overview of the state of translation quality research in the field of machine translation and translation studies and suggestions for possible joint approaches and future directions. to this end, the guidelines by @xcite and the prisma method @xcite served as a methodological basis. in a detailed review protocol, the main question, keywords for the search, search platforms, and inclusion/exclusion criteria were defined, which are explained below in the three main prisma stages, that is, identification, screening, and inclusion, illustrated in figure 1 . ## results the number of records per stage of the literature survey is presented in figure 1 . during the identification stage, 12 keyword combinations were utilised to search and rate publications. the number of records returned from these was 13,762. after removal of duplicates, the keyword-ranking procedure produced results with a maximum score of 167 for the highest-ranked paper. the cutoff score for this article was determined at 77 after screening the results and determining their relevance for the research focus, taking into account the limitations caused by the number of experts of this study. in the screening process, 4 records were excluded because they were not peer-reviewed, 5 because they were superseded or results were presented elsewhere and 1 was a book review. the 41 publications included in this review, were then divided into different thematic fields based on two dimensions: (i) background of authors in one or both fields, and (ii) field addressed in the publication. the background of authors was determined by affiliation(s), available biographic and educational descriptions, and their most common publication venues. in order to avoid confusions between the field of machine translation and approaches to machine translation, the former is referred to as computer science/computational linguistics in this section. 4.1 translation studies/languages (ts) out of the 41 works in the result set, 12 were assigned to the field of translation studies by the professional background of the author(s) and/or categorization of their contents. the main thematic fields in this category are (i) translation quality assessment in general; (ii) machine translation quality (assessment); as well as (iii) human translation quality, post-editing and revision. ## discussion this systematic survey showed that translation studies and machine translation have more in common in reference to translation quality than accu-racy and fluency. a growing influence of technological advances has shifted the translation workflow and conceptualizations of translation quality in ts. alongside automated metrics and postediting, the fit-for-purpose idea of translation quality has entered the field, shifting the burden of defining quality from translators to clients. on the other hand, quality criteria such as (crosssentence) context, comprehensibility, and readability have entered the field of mt. furthermore, the substantial number of joint publications by authors from both backgrounds indicates a convergence of both fields. the results show that in both disciplines, new technological developments are of great interest. ts scholars become increasingly aware that mt can be useful in ts. in contrast, mt scholars realize that comparing outputs to a reference translation or without taking the context into account has considerable drawbacks. publications in ts contain more theoretical contributions, ideas on how mt can be integrated in translators' workflows, studies on machine translation quality assessment as well as post-editing and revision. the fairly new concept of (machine or human) translation quality estimation seems to have not yet been considered in ts. in the field of mt, (machine or human) translation quality estimation is the main topic in more than half of the publications. additionally, a continuously strong focus on automated metrics and technological advances can be observed. in a nutshell, ts can still contribute a strong theoretical basis, quality criteria, and especially definitions of translation quality to mt, while mt can facilitate more measurable and (semi-)automated approaches to translation quality to ts. several limitations of the present survey should be acknowledged. first, its scope was limited to 41 included results, which, given the scope of the topic, raises no claims as to completeness. in fact, several important publications, e.g. @xcite and @xcite , were not in the result set. snowballing or considering citation scores should be future amendments of the method to counteract this issue. secondly, categorizing publications by the authors' scientific field is a somewhat unusual and time-intensive approach. we opted for this approach, since we were particularly interested in the number of publications jointly authored by researchers from both fields and the view on quality concepts by each field. an additional subdivision of publications by the main translation quality concept seeks to provide a transparent and comprehensible categorization method. ## conclusion and future research this comprehensive survey on translation quality in the field of translation studies and machine translation showed that the main ideas in both fields still differ slightly, with translation studies still focusing more on theoretical and less measurable concepts and computational linguistics more on conducting studies and developing metrics. while, on the whole, quality concepts in the two fields are converging, the main challenge in the future will still be to design quality assessment metrics including less easily measurable criteria, such as context and purpose. a systematic catalog of translation quality definitions, criteria, and evaluations of their measurability would be interesting in this regard. furthermore, we suggest to include the role of the translation industry and its viewpoint on translation quality in future reseearch.