Designing a Metalanguage of Differences Between Translations: A Case Study for E nglish-to- J apanese Translation (2023)

Abstract: In both the translation industry and translation education, analytic and systematic assessment of translations plays a vital role. However, due to lack of a scheme for describing differences between translations, such assessment has been realized only in an ad-hoc manner. There is prior work on a scheme for describing differences between translations, but it has coverage and objectivity issues. To alleviate these issues and realize more fine-grained analyses, we developed an improved scheme by referring to diverse types of translations and adopting hierarchical linguistic units for analysis, taking English-to-Japanese translation as an example.

## introduction in translation, assuring quality is the primary and indispensable issue. in translation industry, translation quality assessment (tqa) is introduced to ensure a certain level of quality for clients and endusers, whereas in research, tqa is conducted to gauge the differences in quality between different translation processes and systems @xcite . the goals of tqa are diverse depending on situations, but regardless of situations, we need to compare translations as systematically and objectively as possible @xcite . in translation education, learners should acquire competence to analyze and justify their translations, and explain their decisions with appropriate metalanguages and theoretical approaches @xcite . lacking systematically organized concepts and precise descriptions, however, instructors can explain several possible translations and their differences only by using their own languages in an ad-hoc manner, and learners are not able to grasp the whole picture of differences. in the translation production workflow in industry, machine translation (mt) systems are often used with manual post-editing (iso/tc37, 2017). however, no study has analytically assessed how post-edited mt output (mt+pe) and translation produced exclusively by human translators (ht) differ and what cause the differences. these situations suggest the necessity of a comprehensive typology, or metalanguage @xcite , of differences between translations (target documents, henceforth tds) for the same source document (sd), as a scaffold to discuss such differences objectively, analytically, and precisely. there is only one scheme that enables us to describe differences between independently produced tds @xcite . while their scheme has been tailored for analytic and systematic assessment of differences, it has two vital problems. first, the covered phenomena would be limited; the tds they analyzed were all from the same content domain and produced by human translators. another problem is the vagueness and subjectivity of units employed to capture sub-sentential pairs within given tds. this paper presents our scheme for describing differences between tds, which we have developed to alleviate these two problems. to cover a wider variety of phenomena, we used several sds from various content domains and obtained their translations via substantially different methods, i.e., ht and mt+pe. for tangible and objective analyses, we adopted general linguistic units. our scheme has two notable features: (i) it serves as scaffolding metalanguage for discussing differences between tds, and (ii) it can be used as a research tool as well as a learning material. the remainder of this paper is structured as follows. section 2 describes related work. section 3 explains how we have developed the scheme. section 4 presents our scheme. section 5 reports on our intrinsic evaluation, and section 6 discusses the current status of our scheme and remaining issues. section 7 concludes this paper. ## related work many studies have so far addressed analytic and systematic assessment of translation quality. existing evaluation schemes, e.g., mqm @xcite , focus on translation errors (or issues) @xcite . however, none of them can be used to describe differences between pairs of issue-free translations: how they differ and what cause the differences. recent mt systems, which cause less translation issues @xcite , will require such schemes sooner or later. there is a large body of studies comparing issue-free translations independently produced by various translators, such as students and professional translators @xcite @xcite @xcite , and mt+pe and ht @xcite . they revealed differences in terms of linguistic features, i.e., translationese @xcite and posteditese @xcite . however, they only observed general tendencies of tds as a whole, and none of them established a means to analytically and systematically explain individual instances that exhibit some kind of differences. unlike above, yamamoto and yamada (2022) made an analytic comparison of draft and final versions of tds. they compiled a typology of manipulations applied to tds during the production process, called translation strategies (chesterman, 2016), foot_0 extending the work by @xcite , and ensuring the coverage and systematicity through analyzing actual revision examples extracted from pairs of draft and final versions of tds. their typology consists of syntactic, semantic, and pragmatic subparts foot_1 comprising 13, 9, and 10 types, respectively. the syntactic and semantic strategies have been adopted from linguistic theories @xcite . the pragmatic strategies are, on the other hand, more specific to translation, e.g., referring to external information and ensuring quality for target readers, performed to produce a td that is more appropriate for the predetermined purposes. however, the typology of translation strategies would not be applicable to the pairs of independently produced tds, since it has been developed only on the basis of revision examples performed during the process of producing tds. @xcite is the pioneer of constructing a scheme for extracting and explaining differences between independently produced tds for the same sd. to identify differences between any pair of linguistic expressions observed in given pairs of tds, they proposed a two-step procedure: decompose given pairs of tds and classify the differences between each constituent pair. the latter is realized with decision lists, consisting of 13, 8, and 4 types of categories for syntactic, semantic, and pragmatic differences incorporated from translation strategies @xcite . their work has two major defects. one is the limited variety of phenomena it covers. they used a set of abstracts of scientific articles and their human translations (ht) produced by different translators. due to the relatively limited range of textual domain, homogeneous text type, and the same method for translation production, they observed only a limited range of differences. the other problem is the intermediate unit called "chunk." they introduced it in between sentence and word, and proposed criteria to extract pairs of chunks from given pairs of tds. however, the vague definition of chunk leads to subjective analyses. ## construction of the scheme we developed an improved scheme for describing differences in tds. in our scheme, we adopted the two-step workflow proposed in the previous work @xcite : top-down recursive decomposition of pairs of tds followed by classification of each constituent pair into pre-defined categories. we also followed @xcite to implement a procedure for the first step and decision lists for the second step. in contrast, we addressed the two problems in @xcite as follows. we developed and refined our scheme through repeating annotation and discussion in order to ensure its systematicity and coverage as much as possible (sections 3.2 and 3.3), taking english-to-japanese translations. ## our improved scheme through the process described in section 3, we developed a scheme for describing differences between pairs of tds. during the process, we also assembled instructional materials for annotators. these documents are made publicly available; foot_4 they are mainly written in japanese, since we have compiled them for analyzing tds in japanese. in this section, we explain their summary, using examples of unit decomposition and classification shown in ## intrinsic evaluation we evaluated whether our scheme meets the criteria of metalanguage of translation @xcite , in particular, consistency of decomposition, consistency of classification, and coverage of categories. the two engaged in the development (a and b) and another one of the authors (c) participated in the evaluation as annotators. annotator a is a graduate student in pedagogy, annotator b is a ph.d in computational linguistics, and annotator c is an ma in translation studies. the annotators first read instructional materials of the scheme described in section 4. they then independently annotated the two pairs of tds reserved unseen for this purpose (doc8 and doc9 in as a result, they extracted 471, 466, and 443 pairs of units from doc8, and 463, 456, and 451 from doc9, respectively. tables 6 , 7 , and 8 respectively show the frequencies of syntactic, semantic, and pragmatic categories labeled by each annotator. ## discussion our intrinsic evaluation confirmed that our scheme enables us to analyze the differences between independently produced translations at a certain level of consistency and coverage. toward improving the consistency of classification further, we plan to refine intensional definitions and enrich examples to delineate extensional definitions. external references, such as lists of functional expressions and named entities, terminology, and style specifications, should also help improve consistency. to ensure the coverage, we plan to introduce new categories. even though the present scheme does not achieve perfect consistency and coverage, we consider that the disagreed examples do not necessarily suggest the defects of the scheme. such examples represent fundamental difficulties in understanding the notions indispensable for analyzing translations, and are thus useful in the practical use of the scheme. for instance, in educational settings, the scheme itself is a subject to learn. through exercises of annotating the same td pairs and discussing discrepancies of the annotation results, learners should be able to improve their competence in translation and grasp underlying concepts, such as syntax, semantics, and pragmatics, referred to in the scheme. the scheme will also help learners explain their specific choice of expressions in the target language. our scheme subsumes the categories of translation strategies @xcite and enables comparisons of arbitrary pair of entire tds. it is thus worth investigating that our scheme can also be used to analyze translation strategies. ## conclusion this paper presented a scheme for analytically and systematically assessing the differences between independently produced translations for the same sd. on the basis of the work in @xcite , we adopted nine types of linguistic/non-linguistic units for analysis and refined the decision lists with a wide variety of categories through annotation and discussion using substantially heterogeneous translations, i.e., ht and mt+pe. unlike previous work in analytic assessment @xcite @xcite , we also conducted an intrinsic evaluation of the scheme, employing multiple annotators. the results show that classification of semantic and pragmatic differences is more difficult compared to decomposing unit pairs and classifying syntactic differences. nevertheless, we believe that our scheme is useful, since it covers a wide range of translationrelated concepts and thus can be a useful metalanguage to talk about differences in translation. our scheme is partly dependent on the target language, i.e., japanese. we thus plan to examine its applicability to translations from other languages than english into japanese. to analyze differences between translations in other target languages than japanese, we need to adapt our scheme to them.