Predicting the number of errors in human translation using source text and translator characteristics (2022)

Abstract: Translation quality and efficiency are of great importance in the language services industry, which is why production duration and error counts are frequently investigated in Translation Process Research. However, a clear picture has not yet emerged as to how these two variables can be optimized or how they relate to one another. In the present study, data from multiple English-Japanese translation sessions is used to predict the number of errors per segment using source text and translator characteristics. An analysis utilizing zero-inflated generalized linear mixed effects models revealed that two source text characteristics (syntactic complexity and the proportion of long words) and three translator characteristics (years of experience, the time translators spent reading a source text before translating, and the time translators spent revising a translation) significantly influenced the number of errors. Furthermore, a lower proportion of long words per source text sentence and more training led to a significantly higher probability of error-free translation. Based on these results, combined with findings from a previous study on production duration, it is concluded that years of experience and the duration of the final revision phase are important factors that have a positive impact on translation efficiency and quality

## context in the language services industry, prompt delivery of an accurate translation is greatly appreciated. however, time and quality are often a trade-off, which is a substantial concern for many translators @xcite . although neither human nor machine can create a "perfect" translation instantly, it is important to identify which factors lead to speedy production and high quality. translation process research (tpr) can shed light on such an essential aspect of translation. in tpr, efficiency has often been investigated with respect to source text (st) difficulty and the different levels of expertise possessed by translators (i.e., what distinguishes professional translators from non-professionals, such as student translators or language learners). for example, @xcite revealed that more difficult texts attracted longer gaze time, and @xcite found that difficult sts slowed down production time. interestingly, in dragsted's study, professionals tended to fall back on more novice-like behavior when they were engaged in difficult sts, while professionals exhibited exceptional performance when sts were easy. moreover, research has shown that professional translators produce translations faster than student translators @xcite . although the differences in translator behavior based on expertise and st difficulty are not always statistically significant (see @xcite , the findings in tpr in general support that time efficiency is influenced by the nature of the st and certain translator characteristics. while efficiency is relatively easy to define, translation quality is not due to its multifaceted nature. product quality can be measured in various ways, for which @xcite formulated different approaches: the transcendent, product-based, user-based, manufacturingbased, and value-based approaches. in addition to the quality inherent in the product itself, how clients perceive the product is crucial in translation. indeed, some clients prioritize cost and time over quality. such being the case, it is hard to reach a consensus as to which aspect of translation quality should be prioritized, although this topic is actively debated in translation industry @xcite . quality measurement also poses problems in translation research, though scholars have attempted to take industry perspectives into account. for instance, @xcite introduces a functionalist translation assessment tool that focuses on user points of view. within the critt tpr-db community, foot_1 the multidimensional quality metrics (mqm) framework (lommel, 2018) is often utilized. the critt tpr-db makes it possible to annotate errors using a scheme based on mqm, on a platform called yawat (see @xcite . although quality measurement based on error typologies such as those made available through mqm has some disadvantages (see @xcite , the error-based assessment of translation can be useful when accuracy is seen as vital @xcite . such an assessment is also extremely beneficial to tpr in that it offers clarity and consistency to the field. the complexity of investigating translation quality makes it difficult to fully capture the trade-off or interplay between production time and quality, especially in human translation (ht). however, some interesting findings have been reported. for example, @xcite examined the use of external resources during ht and post-editing (pe) and found that the overall production time of ht was significantly higher than pe due to the increased time spent on external resources in ht. they also revealed that the overall quality was influenced by the time spent using external resources and that, in ht, the overall error score was lower when the participants consulted external resources for a longer period of time @xcite . in this specific experiment, it seems that time and quality were in fact a trade-off. however, it is still unknown at this point whether this is the case with ht without external resources and/or in different language pairs. the present study attempts to further elucidate the relationship between production time and translation quality using english-japanese translation. the research question is: can we predict the quality of translation based on characteristics of the st and of individual translators? here, the quality of translation is operationalized as number of errors, which has been correlated with several process metrics used as indicators of cognitive effort @xcite . a statistical method called zero-inflated generalized linear mixed models (ziglmms) will be utilized, which nicely handles count data skewed by a large number zeros. by doing so, this study aims to identify which characteristics of a st or a given translator potentially influence translation quality and efficiency. in the following, section 2 contains a description of the data; section 3, the results of statistical analyses. in section 4, the overall result will be discussed along with some findings from @xcite , where production duration was predicted by text and translator characteristics, in order to gain a better understanding of the relationship between st and translator characteristics and translation quality and efficiency. ## data and methodology the data used here was originally extracted from the enja15 project from the critt tpr-db, in which 39 participants translated two out of six sts from scratch. in the present study, there were approximately 13 different translations for each st. 2 the ".sg tables" from the critt tpr-db were utilized, where the participants' textual and behavioral (i.e., typing and gaze) data is organized in a way that researchers can analyze it at the segment (i.e., sentence) level. errors were manually annotated and counted. 3 in doing so, although the unit of analysis was at the segment level, st and tt did not necessarily have segment-level equivalence. in fact, some translators did divid one st segment into two tt segments or combine two st segments into one tt segment. the number of errors was approximated by the number of content words (i.e., nouns, verbs excluding auxiliary verbs, adjectives, and subordinating conjunctions) in the alignment group on the st side. for example, a participant translated "was imprisoned" as 逮 捕 さ れ まし た (literally "was arrested"). this japanese translation was morphologically analyzed and divided, as marked by underscores, into five tokens, and yet was aligned to "was imprisoned" as a group. in this case, there was only one content word on the st side of this alignment group, and therefore, only one error was counted despite multiple tt tokens. this method of counting errors roughly but consistently quantified the severity of errors without judging them subjectively and dichotomously (e.g., minor versus critical). the resulting variable, named totalerrorcount, ranged from 0 to 14 errors. 243 out of 520 segments had zero errors, and 117 segments only had one. as figure 1 shows, the data was zero-inflated and overdispersed (i.e., the variance is greater than the mean). ziglmms were utilized to handle this skewed data, which have two separate parts. the first part is a count model, which can be interpreted in the same manner as general linear mixed effect models. the count model explains what increases the number of errors. the other part is called a zeroinflated (zi) model, whose interpretation is equivalent to a logistic regression. it calculates the chance of contributing to excessive or structural zeros among all the zeros in the data. in this case, the zi model tells us what affects the probability that a segment will have zero errors. for this statistical analysis, packages called glmmtmb @xcite and dharma @xcite were used in rstudio @xcite . a variable called text, which identifies the six sts in critt tpr-db, was used as a random effect for both count and zi models. a backward step-wise selection method was adopted to build models; that is, a model was created with all the st characteristics included in fixed effects, and one independent variable was removed at a time until all the fixed effects in the model were statistically significant (@xmath0.05). another set of models was created for translator characteristics in the same manner. the two different types of characteristics (st and translator) were not combined in a single model so that the methodology would be identical to that of @xcite . the following is a sample model: the independent variables tested in this study are described in ## discussion the check marks in figurative and l1 were not significant in any models. figurative expressions have been discussed and identified as a source of translation difficulty (e.g., @xcite . a preliminary analysis on totalerrorcount indeed indicated that figurative produced a significant result in the count model, though only if it was the sole fixed effect in a model. using the backward step-wise selection method may have lowered the explanatory power of figurative when other independent variables were involved in a model. this might also be true for l1, which showed a significant result in the zi model in a single fixed-effect model. there was no st characteristic that significantly influenced both totalerrorcount and dur, but two translator characteristics (i.e., experience and endrevision) were important factors for both dependent variables. @xcite revealed that dur was negatively influenced by experience and positively influenced by endrevision. that is, more experienced translators translated faster, and participants who spent more time on end revision had longer production duration as a result. 8 combined with the finding in the present study, it can be concluded that i) years of experience is a good predictor of translation quality and time efficiency and that ii) the time translators spend on end revision inevitably increases production duration but in return increases quality. dur was also significantly influenced by prob1norm; participants spent more time translating as they encountered a greater number of less familiar or less frequently used words. of course, each individual has different linguistic knowledge, and prob1norm is a simplistic operationalization of word familiarity. that being said, if prob1norm truly impacts dur but not totalerrorcount, it might be the case that translators can improve time efficiency by further familiarizing themselves with the source language. familiarization would particularly matter when it comes to different genres or domains, where words are used as terms with different meanings than when they are used in general texts. ddepth and lwratio increased the error count while experience and endrevision decreased it, as discussed in the previous section. the former two are st characteristics that are fairly easy to quantify. it is not clear at this point whether pre-editing sts in such a way that ddepth and lwratio values will be lower leads to higher-quality translation. these characteristics may nevertheless be used to compare different texts and/or caution translators about potential difficulty in advance. the effects of experience and endrevision were also straightforward. this evidence may encourage translators to gain more experience and keep in mind that the end revision phase is critical to translation quality even when translators feel the need to prioritize time. it may be worth mentioning that no clear relationship was observed between endrevision and experience. recall that, in initialorientation, the head-starter group was expected to produce the least number of errors and that the quick-planner group the most. this can be explained by the fact that these two groups had the highest and lowest average years of experience respectively. in contrast, the short group in endrevision had the highest average years of experience, followed by the long and none groups in this order. moreover, participants who did not spend any time reviewing their draft (i.e., the none group) were most prevalent in the headstarter group, and none of them belonged to the quick-planner group. although there was no clear relationship between endrevision and initialorientation in this study, previous research has found that head-starters and quick-planners tended to prefer online revisions (i.e., revising as they produce a tt) while scanners and systemic-planners carried out end revision @xcite . perhaps, totalerrorcount may be better analyzed if participant revision preferences, including online revision as well as end revision, are taken into consideration. ## future directions this paper has revealed that some st and translator characteristics significantly contribute to the number of errors per segment in english-japanese from-scratch translation. combined with findings from a previous study, evidence was found that translators' years of experience make a difference in terms of translation quality and time efficiency, and that the length of end revision has a positive effect on quality even though it may take some extra time. however, the examination of translators' initial orientation phase with a st suggested that there may be a complex interplay between the length of end revision and translator style (i.e., translator preferences for revisions and initial st reading). this should be further scrutinized in future research. this study was limited to english-japanese translation, and hence, the result should be corroborated by similar studies using other languages. in doing so, methodology needs to be discussed in two respects. firstly, the way of quantifying translation quality is of utmost importance since it can produce very different results. error count is relatively easy to use as a quantification of translation quality but fails to recognize fine-grained differences in quality (e.g., it does not distinguish excellent from adequate quality). some researchers have tried evaluating the quality of ht using metrics primarily used for machine translation (mt) output, such as bleu (e.g., @xcite , and produced interesting results. at the same time, however, research has found that those metrics cannot fully capture errors in ht because ht errors are different from mt errors @xcite . translation evaluation methods call for further discussions in tpr. even if the number of errors is used as a primary measure of translation quality, multiple evaluators and calculations of inter-rater reliability may need to be considered. our study was limited to errors annotated by a single researcher, which admittedly is the biggest weakness of this paper. furthermore, there may be a better way of counting errors. the method utilized (see section 2) seems justifiable since it allows us to quantify errors regardless of the target language and to compare different studies in the critt tpr-db. however, it does require significant manual work and may not be viable when multiple evaluators are involved. the other methodological factor that demands attention is the use of ziglmms. this is a fine-tuned statistical method that can deal with zero-inflated count data, but the interpretation of zi models requires more discussion in tpr. some researchers may find it implausible to assume excessive zeros in conducting this line of analysis. since many researchers in tpr use linear mixed effect models, it might be time for us to discuss what is considered a good model in our discipline. in this paper, the r 2 of the best model was 0.29, which means that roughly 30% of the total variance was explained by the model. this seems to be satisfactory as much smaller numbers have been reported (e.g., @xcite , while much greater values have been achieved as well (e.g., @xcite . of course, it is risky to solely rely on r 2 as if it were the only criteria that could be used to validate an analysis. such a discussion will surely lead to the advancement of methodology in tpr.