Proficiency and External Aides: Impact of Translation Brief and Search Conditions on Post-editing Quality (2022)

Abstract: This study investigates the impact of translation briefs and search conditions on post-editing (PE) quality produced by participants with different levels of translation proficiency. We hired five Chinese student translators and seven Japanese professional translators to conduct full post-editing (FPE) and light post-editing (LPE), as described in the translation brief, while controlling two search conditions i.e., usage of a termbase (TB) and internet search (IS). Our results show that FPE versions of the final translations tend to have less errors than LPE ver- sions. The FPE translation brief improves participants’ performance on fluency as compared to LPE, whereas the search condition of TB helps to improve participants’ performance on accuracy as compared to IS. Our findings also indicate that the occurrences of fluency errors produced by experienced translators (i.e., the Japanese participants) are more in line with the specifications addressed in translation briefs, whereas the occurrences of accuracy errors pro- duced by inexperienced translators (i.e., our Chinese participants) depend more on the search conditions.

## introduction post-editing (pe) has become widely used in industrial translation. in some domains, more than 40% of translation practices are conducted as pe (jtf, 2020). however, pe does not mean that translators simply use machine translation (mt) to translate. in practice, mt systems must be integrated into an authentic environment, such as cat tools, with which professional translators can perform pe operations including searches for terminology, concordance, usage of external resources on websites, etc. in addition, pe must often satisfy given quality requirements -as to whether it is full pe (fpe) or light pe (lpe) -which are normally described in the work instruction a translation brief. thus, differences in search conditions -internet search (is) or availability of termbase (tb) -and translation brief (i.e., lpe/fpe) may affect the translator's psychology and working style, which in turn may impact cognitive load during the translation process (effort) and the translation product (quality). while the differences in the work environment and task conditions affect the process and product, few previous studies have taken this into account. there are possibly two reasons for it. the first reason is that an authentic translation environment capable of collecting translation process data was not previously available. for example, translog-ii, which is an experimental tool for researchers to collect translators' keyboard input and gaze data, does not provide the functionality that professional translators are used to in conventional cat tools @xcite . the other reason has to do with translation conditions. the establishment of the international pe guidelines, iso 18587 in 2017 has led to a certain common understanding with respect to what has to be post-edited. however, the industry definition of pe has not always been used which makes it difficult to compare the results of pe studies across the research. for example, it is often unclear whether a study focused on lpe or fpe. given this background, the purpose of this study is to have translators translate in an authentic translation work environment, collect translation process data, and compare and verify differences of translation performance. the term "authentic environment", in our definition, means 1) ensuring that pe is conducted in a professional cat environment (trados studio in our case), as well as that translators are allowed to use is or tb, and 2) providing appropriate work instructions that specify whether the task is fpe or lpe. the variable 1) is referred to as "search condition", and 2) is considered to be the "translation brief". in this way, this study examines differences in product-process interactions, considering these conditions (as variables) that will affect pe processes and products. ## literature review there exist many comparative analyses of translation products and processes under different task conditions (from-scratch translation, pe) and work environments. however, to the best of our knowledge, no previous literature has directly examined the impact of differences in translation briefs (e.g., lpe vs. fpe) on pe quality. a translation brief specifies the intended audience and purpose of the translation in the target language. translation briefs are meant to bias (or prime) translators, to activate particular, but not other, "bodies of thought" @xcite 13) that answer to a specific translation expectations. @xcite points out that the notion of translation brief is "a key point in german-language skopos theorie since 1984". for @xcite translation depends on the "conclusions the translator draws from the brief [. . . ] it is no longer the source-text [alone] that guides the translator's decisions but the overall communicative purpose the target text is supposed to achieve in the target culture." also @xcite ) mentions that "translation briefs and technical guidelines offer indications both about author intentions and the background of the target audience", and by now several translation companies offer guidelines that explain how to draft translation briefs foot_1 . @xcite defines translation from a process-oriented perspective as follows: "translation is the process of creating target language content that corresponds to the source content according to agreed-upon specifications". melby's concern is the relationship of translation to the "specifications" or the required functions of the translation in the given context. this idea applies to pe tasks. @xcite argue that error-category-based specifications should be used to define quality in mtpe projects. the specification that melby et al. promote is mqm (multidimensional quality metrics), developed in the eu-funded qtlaunchpad project @xcite . although the definition of fpe and lpe has been clarified with the development of iso 18587, the international standard for pe (iso, 2017), this definition is ambiguous for use in practice. @xcite provide clearer instructions and methods for post-editors, based on their own industry experience, by correlating instructions of fpe and lpe with mqm error typologies. in the same vein, @xcite propose a risk management method for which each translation task must include the client's requirements which should be broken down into issue typologies that are applied during pe with consideration of their severities. @xcite have described this pre-production process from this risk management perspective. they claim that decision-making processes which include -among other thingsthe understanding of translation brief should be taught during the translation education. it is important, they say, to consider the constraints, conditions and expected translation quality for each task. this has been compiled as pe guidelines @xcite which stress that the translation brief has a significant impact on the translation process and product as well. the following literature deals with interactions between different task conditions or "parameters" and differences in their performance. daems and macken (2020) carried out an experiment with two groups of participants. one group were revisers who usually check/edit human translations. the other group consisted of post-editors who are used to correct mt output. in this experiment, the raw mt output and the human translations were given to both the revisers and the post-editors, but participants were not informed of the type of the texts given to them. the study compared the quality of the translations after each group's edit. the result shows that, the revisers outperformed the post-editors when they edited mt output. however, the post-editors outperformed the revisers when they edited human translations. while in every case accuracy errors remain undereditted, this outcome suggests that different conditions influence the performance of the revisers and post-editors. in connection with this, "search conditions" are also important. for example, is the cat tool available for the pe, is a glossary provided, and/or is plenty of time and pay given for external searches? whether or not sufficient working conditions are prepared is a prerequisite for fulfilling the expected requirements. search conditions for external resources such as is relevant to the subject matter in the course of translation are vital to the translator, and this will also affect translators' performance. @xcite compared search behaviors of professional and novice translators. they found that professional translators devote a higher percentage of time and operations into searches. they found a high correlation between accuracy errors and the frequency and depth of searches. these findings suggest that is during translation will greatly affect translators' quality. ## experimental design in our study, we investigate how the two controlled parameters, i.e., translation brief and external search, interact with the participants' different levels of translation proficiency. ## quality assessment two professional translators (one chinese and one japanese) were hired to annotate the translation errors in the simplified chinese and japanese gnmt outputs as well as in the 12 pe versions. because the sts of ata exams are specifically designed to incorporate challenges that may result in translation errors associated with the categories and severity of errors under the grading framework of ata, annotators in this experiment were given guidelines for error annotation based on an ata-adapted annotation schema. 2 errors were divided into six types, "mistranslation", "usage", "terminology", "grammar", "omission/addition" and "other". the former four types were further annotated as "critical" and "minor" errors depending on the severity of errors. as a result, there were altogether ten different kinds of errors, i.e., mistranslation critical, mistranslation minor, usage critical, usage minor, terminology critical, terminology minor, grammar critical, grammar minor, omission/addition, and other. in this experiment, the annotation was conducted on the level of ag. annotators were asked to proceed in two steps: first, they should conduct word-level alignment between the tt and their corresponding st. then in the second step, ags were assigned an error as applicable. when they came across an error that they considered an omission or addition, however, they were not required to do an alignment. in other words, there are only ags for errors excluding "omission/addition" in this research. for the purpose of this study, the occurrences of "mistranslation", "terminology", and "omission/addition" errors were grouped under the label of "accuracy" error, while "usage", "grammar", and "other" errors were grouped under the label of "fluency" error. additionally, all kinds of translation errors were grouped under the label of "critical" and "minor" errors according to their annotated severity. therefore, we gained four subcategories of errors under study, such as "accuracy critical", "accuracy minor", "fluency critical", and "fluency minor". foot_3 5 results ## conclusion this paper aims to investigate the impact of translation briefs (full pe, fpe vs. light pe, lpe) and search conditions (provided termbase tb vs. free internet search is) on pe quality of two groups of participants with varying levels of translation proficiency. to this purpose, four english sts from previous ata certification exams (47 sentences, about 1,000 words) were automatically translated into simplified chinese (zh) and japanese (ja) by google nmt (gnmt), and were post-edited by five chinese student translators and seven japanese professional translators, respectively. the study was thus carried out in two language pairs (en-zh, en-ja) and the 12 post-editors produced a total of 658 segments. keystrokes were logged and gaze data recorded, but these aspects of the experiment are not addressed in this paper. to run the experiment under ecologically valid working conditions of professional translators, we conducted the experiment in the trados workbench using the new trados-translog interface @xcite . we asked participants to post-edit four texts under two types of translation briefs, i.e., fpe (f) and lpe (l), and two types of search conditions, i.e., tb (t) and is (s). therefore, we had four different pe tasks for each participant, i.e., pfs, pft, pls, plt. the chinese and japanese gnmt outputs and the corresponding post-edited versions were annotated for translation errors based on an ata-adapted error taxonomy. we grouped the errors under four labels, i.e., "accuracy", "fluency", "critical", and "minor" errors. we calculated the error count by segment, aggregated them over the four pe tasks, and compared the error distribution in the two raw gnmt outputs (simplified chinese and japanese) and in the twelve post-edited versions. our results show a similar error distribution for gnmt output and the pemt versions. for both, gnmt and pemt, minor fluency and critical accuracy errors were more common than other subcategories of errors. pemt has generally lower percentages of fluency and minor errors than the gnmt, but a higher percentages of critical accuracy errors. looking into the error distribution for each of the tasks we see that, overall, fpe versions tend to have less errors than lpe versions. the translation brief of fpe improves in particular the participants' performance on fluency as compared to lpe, and the provided tb seems to improves the participants' performance on accuracy as compared to is. across the four pe tasks, there are notable more accuracy and critical errors for chinese than for japanese participants. these results are to be expected, to the extent that the more experienced japanese translators ought to deliver more frequently translations of higher quality than our inexperienced chinese translators. our findings also shows that inexperienced translators have significantly fewer accuracy errors in the tb condition as compared to searching online (is). we assume that this is the case since less experienced translators typically possess less research skills @xcite @xcite . experienced translators, on the other hand, seem to better realize implications of the translation briefs: with respect to accuracy errors, there is no significant difference across the four pe tasks. however, in the fpe condition, experienced translators produce less fluency errors as compared to lpe condition. this difference has not been observed for the less experienced translators, which suggests that experience leads to more awareness of the variations between different translation briefs. due to the restrictions of accessibility to the translators of our experimental language pairs, we only recruited twelve participants for this study. therefore, there are certain limitations in the statistical results due to the relatively small sample size. however, we are currently collecting more data and intend to look into other aspects of translation process and translator behavior in future studies. the datasets are publicly available in the tpr-db @xcite and the reported results replicable.