Investigating the Impact of Different Pivot Languages on Translation Quality (2022)

Abstract: Translating via an intermediate pivot language is a common practice, but the impact of the pivot language on the quality of the final translation has not often been investigated. In order to compare the effect of different pivots, we back-translate 41 English source segments via vari- ous intermediate channels (Arabic, Chinese and monolingual paraphrasing) into English. We compare the 912 English back-translations of the 41 original English segments using manual evaluation, as well as COMET and various incarnations of BLEU. We compare human from- scratch back-translations with MT back-translations and monolingual paraphrasing. A varia- tion of BLEU (Cum-2) seems to better correlate with our manual evaluation than COMET and the conventional BLEU Cum-4, but a fine-grained qualitative analysis reveals that differences between different pivot languages (Arabic and Chinese) are not captured by the automatized TQA measures.

## introduction translation via a pivot language has been a common practice for a long time. for instance, the preservation of ancient greek ideas is a major contribution of islamic civilization via arabic as a pivot language. much of aristotle's original work in old greek is preserved to us through muslim scholars who translated the ancient-greek scripts into arabic which was then later translated into latin, and from there into various other languages. still today, pivot translation is an important technique mainly due to a lack of available direct translators. the availability of translators who know two (or more) languages becomes increasingly limited as the number of speakers in those languages decreases. it is, therefore, in particular, translation across smaller languages which requires translation via another, usually a more common language. thus, from the more than 4000 languages in the world that have developed a writing system foot_0 , translators will be available for only a very tiny fraction of the 16 million or so possible language combinations. however, translations into (or out of) the 'big' languages -such as english, french, spanish, russian, or arabic -might be more easily available. similarly, there are 552 language pairs for the 24 official european languages but it might not always be possible to find translators for all of these combinations. as a work-around, often english, french, or spanish are used as an intermediate language in the eu. while pivot translation is commonly used for written and spoken language (e.g., interpretation), not much work exists that assesses the impact of the intermediate language on the translation quality. @xcite indicates that translation studies researchers' interest in pivot translation has grown since the mid-2010s. the first research that focuses on pivot translation, however, can be traced back to 1963 which was in regard to literary works @xcite . zaborov's work reflects the soviets authorial control over book translation by requiring to translate any foreign book into russian before it can be translated into other languages @xcite . translation studies' trend of literature-oriented research focusing on pivot translation carried on through the seventies, eighties and nineties of the twentieth century @xcite @xcite @xcite . starting from 1999 onward, pivot translation research expanded to include two other areas in which translation via an intermediate language is considered a common practice, namely interpreting and audiovisual translation @xcite @xcite . more recently, pivot translation is getting more popular in more areas of research. @xcite for instance, review the applicability of pivot mt systems and recommend incorporating "quality estimation and/or automatic/human post-editing to the intermediate translation of the pivot language" (p. 10). most recently, o'hagan (2022) investigates the challenges, and implications of the use of english pivot translation in game localization. the choice of the pivot language is often based on the available human (and/or electronic) resources, but the quality of the final translation depends crucially on the quality of the pivot language. if there is a mistake or ambiguity in the pivot translation, the source meaning might be erroneously or incompletely reproduced in the target. the pivot language might be lacking (linguistic) constructions and possibilities that the source language has and, therefore, be incorrectly recovered from the pivot language. the pivot language might also favor interpretations that lead to incorrect conclusions in the target. as compared to direct translation, pivot translation proceeds in two step (1: source-to-pivot and 2:pivot-to-target), each of which filters or amplifies the linguistic signal in specific ways. in this study, we use back-translation as a method to assess the impact of different pivot languages in translation. we choose the source and the target to be the same language (english), and we select two quite different pivot languages, chinese and arabic. back-translation into english via two different intermediate languages allows us to clearly assess the impact of the pivot language, since any divergence between the source and the target can be attributed to the intermediate language. we triangulate using monolingual paraphrasing as a tool with which back-translations are compared. section 2 provides a detailed description of the different datasets and their collection processes. in section 3, we explain the different translation quality assessment methods we used. we describe our manual evaluation design and use its result as a reference for the results of the two automatic evaluation metrics we incorporate (bleu and comet). then, we draw quantitative and qualitative comparisons among the three assessment metrics' results. in section 4, we present a statistically backed discussion of the influence of pivot languages on human translation quality in our datasets. we follow this discussion with an in-depth qualitative observations from our datasets in the light of normalization, priming, and shining-through. section 5 gives a summary and conclusions and states future endeavors. ## experimental design this study compares english back-translations via arabic and chinese pivot languages and monolingual english paraphrasing on the segment level. we generated data for from-scratch back-translation (ht) via arabic (ar), chinese (zh), via monolingual english paraphrasing (ph), as well as machine back-translation (mt). a total number of 41 english source segments were first machine translated and postedited by professional translators into arabic and chinese. these arabic and chinese translations served as pivot translation. subsequently, the arabic pivot translations were then back-translated into english by 8 translators (ar) -with arabic as their first language (l1) and english as their second language (l2). the chinese pivot translations were back-translated into english by 4 translators (zh) -with chinese as their l1 and english as their l2. for these human from-scratch back-translations, we collected behavioral data, eye-tracking and key logging. as described in @xcite , we also used two neural machine translation (nmt) systems (i.e., bing and google translate) to generate english nmt back-translations (mt) via the arabic and chinese pivot translations. in addition, 8 computer sciences graduate students with english as their l2 produced monolingual paraphrases of the original english segments (en). a total of 912 translated segments were generated, consisting of: ar 328 segments, zh 164 segments, mt also 164 segments, from which 82 segments for arabic (mt-ar) and 82 segments for chinese (mt-zh), and en 256 segments. figure 1 illustrates this data collection process. it shows how we set up the translation quality assessment (tqa) for three different translation tasks (ht, mt, ph). 2 the data was processed in the critt translation process research database @xcite , critt tpr-db), which includes manual word-alignment of the 912 segments. the quality of all translated segments were manually evaluated, as well as automatic assessment (i.e., bleu and comet). these assessment results were utilized as references for the translation quality. 3 translation quality assessment ## impact of different pivot languages on human translation in this section we look into differences between paraphrasing (en) and different human translations via the pivot languages (ar and zh). ## conclusion this study investigates the quality of translations via different pivot languages. in order to allow for seamless comparison of the source and the target, it compares human back-translation, neural machine back-translation, and monolingual paraphrasing from english via arabic and chinese back into english. six short english texts (together 41 sentences) were translated into arabic and chinese. the two sets of arabic and chinese texts were then back-translated into english by 8 arabic and 4 chinese translation students respectively. in addition, we also produced back-translations with two nmt systems, bing and google translate, and the english original texts were also paraphrased by 8 computer sciences students. this amounts to 25 english versions: 1 english original, 8 arabic and 4 chinese human back-translations, 2 nmt back-translations from arabic and 2 from chinese, and 8 (monolingual) paraphraes. however, as some segments were not translated, paraphrased or lost due to software errors, we were left with 912 translated segments (from potentially 24*41 = 984). we assessed the quality of the 24 reproduced versions (912 translated segments) on a segment level by comparing each of them with the english original, using two automatic measures (bleu and comet) as well as manual evaluation. the manual evaluation was based on a likert scale (see appendix a) in which 20 fluent english speakers assessed the similarity between the original and the reproduced versions. each segment was independently rated by at least five evaluators with good agreement (weighted fleiss's kappa of 0.67). we took the average score of the manual evaluations as a gold standard. we also experimented with different weights for various bleu configurations. our findings indicate that: we explain the higher scores of monolingual paraphrasing (as compared to bilingual paraphrasing) through stronger priming effect, which results in more literal renderings. in contrast, normalization processes effects could be observed in the forward translation when generating the pivot translation, which we explained as cultural differences in the pivot language, while shining-through effects could be observed in the back-translation. both phenomena may be factors which results in back-translations to show more lexical variation than paraphrasing. further studies can be conducted using other automatic metrics and similar methods applied in this study. further research can also include more pivot languages and nmt systems. furthermore, the collected behavioral data of eye tracking and key logging of the human fromscratch back-translations can be utilized for purposes of triangulation and also in future related research.