Attainable Text-to-Text Machine Translation vs. Translation: Issues Beyond Linguistic Processing (2021)

Abstract: Existing approaches for machine translation (MT) mostly translate given text in the source language into the target language and without explicitly referring to information indispensable for producing proper translation. This includes not only information in other textual elements and modalities than texts in the same document and but also extra-document and non-linguistic information and such as norms and skopos. To design better translation production work-flows and we need to distinguish translation issues that could be resolved by the existing text-to-text approaches and those beyond them. To this end and we conducted an analytic assessment of MT outputs and taking an English-to-Japanese news translation task as a case study. First and examples of translation issues and their revisions were collected by a two-stage post-editing (PE) method: performing minimal PE to obtain translation attainable based on the given textual information and further performing full PE to obtain truly acceptable translation referring to any information if necessary. Then and the collected revision examples were manually analyzed. We revealed dominant issues and information indispensable for resolving them and such as fine-grained style specifications and terminology and domain-specific knowledge and and reference documents and delineating a clear distinction between translation and what text-to-text MT can ultimately attain.

## introduction translation is not a purely linguistic process @xcite but also the process of producing a document in the target language that plays the same role (has the same effect) as the given source document written in the source language. when translating a given document, translators refer not only to the textual elements in the document, but also to the role of each textual element (e.g., running text, section title, table element, and caption), other non-linguistic elements (e.g., figures and formulae), and their structure. to produce a translation, we also need some extra-document and non-linguistic information, such as the norms specific to the register of the document and corresponding target sub-language @xcite , the objective and the intended usages of translation, i.e., skopos @xcite , and various specifications @xcite designated by the translation client if any. despite the requirements a (proper) translation must satisfy, techniques for machine translation (mt) have been developed by regarding the task of translation as text-to-text transfer. until very recently, most studies have performed a text-to-text mt for each text segment, 1 even though a sequence of perfect segment-level text-to-text translations does not necessarily qualify as a proper translation. recent studies on neural mt (nmt) have addressed issues beyond this formulation, exploiting further information such as document-level textual context @xcite @xcite and other modalities @xcite . there are also several focused studies on exploiting extra-document and non-linguistic information. however, such information has not been extensively discussed. as a result, in translation production workflows at translation service providers (tsps), where mt outputs are treated as draft translations, heavy human labor is necessary to fill the gap between mt outputs and translations in addition to resolving issues at the text-to-text level, for instance, by manual post-editing (pe). to design and establish more practical ways of exploiting mt systems in translation production workflows as well as to discuss how to make mt systems more useful, we need to understand what lies in the gap between a translation that text-to-text processing can attain and a truly acceptable translation. moreover, this should be shared among not only translators but also mt researchers and mt users. from this point of view, this paper presents our analytic assessment of mt outputs, taking an english-to-japanese news translation task as a case study. first, we obtained segment-level text-to-text translation by resolving translation issues in mt outputs. at this stage, a minimal pe was performed referring only to each source segment isolated from any other information, and thus the results represent what segment-level text-to-text mt systems can ultimately attain. then, the document-level full pe (iso/tc37, 2017) in the succeeding stage resolved all the remaining issues, i.e., those issues lying in the gap between acceptable segment-level text-to-text translation and proper translation. finally, the collected revision examples were manually analyzed based on an issue classification scheme. this revealed several dominant issues as well as the information indispensable for resolving them. the remainder of this paper is organized as follows. section 2 summarizes related work in translation studies and mt. section 3 presents the material for our case study. section 4 describes our workflow, designed for collecting translation issues that cannot be solved by textto-text processing. section 5 presents our analytic assessment of translation issues, which relies on an existing issue typology, and explains the dominant issues as well as several types of extra-document and/or non-linguistic information that must be used to solve them. section 6 describes future research directions and advice for non-expert mt users, and section 7 concludes the paper. ## related work in the literature of translation studies, linguistic approaches to translation have been criticized @xcite , and the equivalence of a source document and a target document has been studied from a diverse range of aspects. in a seminal work, @xcite claimed the necessity of equivalence of recipients' reactions when reading source and target documents. @xcite compiled a typology of translation strategies adopted to guarantee the equivalence when producing a translation. his syntactic and semantic strategies can be explained (and potentially realized) referring only to textual information in the source document and linguistic knowledge in general. in contrast, some of his pragmatic strategies, such as cultural filtering and illocutionary changes, require extra-document and/or non-linguistic information. some of the kinds of information that must be referred to for producing a proper translation, including terminologies and style specifications, are mentioned in the translation workflow standard, iso 17100 (iso/tc37, 2015). other items are mentioned in existing criteria for quality assurance, such as the multidimensional quality metrics (mqm) foot_1 and the dynamic quality framework (dqf). 3 reference sources, such as translation memories and bilingual concordancers, and other access to past translations are valuable assets for improving efficiency in personal practices and workflows in tsps. however, there is neither a comprehensive inven-tory of references, nor a common view of the extent of the necessity and availability of each reference depending on the given skopos. recent advances in mt go beyond segment-level and/or text-to-text processing. for instance, @xcite focused on several discourse-level issues, i.e., deixis, lexical cohesion, and ellipsis, occurring in segment-level text-to-text mt. following studies proved that context-aware decoding that refers to several preceding segments better handles these linguistic phenomena @xcite . there are several focused studies on exploiting extra-document and non-linguistic information, including terminologies @xcite , politeness @xcite , domain @xcite @xcite , style @xcite , markups @xcite , and external lexical knowledge @xcite . however, the information indispensable for producing a proper translation have not been thoroughly studied. more importantly, no work guarantees to perfectly reflect such information. the mt community has benefited from manual analyses of translation issues foot_3 caused by mt systems. existing methodologies for analyzing translation issues in mt outputs can be two-fold: (a) comparisons of independent products, i.e., mt outputs and human translations @xcite @xcite , and (b) annotations of the issues in mt outputs according to pre-determined issue typologies, such as mqm and dqf @xcite @xcite . the issues identified in the former approach contain both true errors and preferential differences, i.e., alternative acceptable translations independently selected by mt systems and humans. the latter approach enables us to clearly separate them. for instance, past studies @xcite @xcite analyzed outputs of segment-level text-to-text mt, showed the limitation of that approach, and encouraged the research on document-level mt. however, they discussed only the differences between two text-to-text approaches. issues beyond the text-to-text processing, such as those related to extra-document and/or non-linguistic information, have seldom been mentioned @xcite , and no focused and empirical analysis has been conducted. ## subject of our case study our focus in this paper is to clarify the types of extra-document and/or non-linguistic information that are indispensable for producing a translation. among several translation tasks, this paper takes an english-to-japanese news translation task as a case study and presents our indepth analysis. we chose it for two reasons. first, despite the high demand for it, the task is still very difficult, since the two languages are linguistically distant and used in substantially different cultures (cf. english-to-german studied by @xcite ). the norms for news texts are also substantially different in these languages, making them more difficult to translate than texts in other domains, such as scientific paper abstracts @xcite and patent documents @xcite . the second reason is that we wished to conduct an indepth analytic assessment of translation (see section 5) by ourselves. we have a linguist who is highly competent in both linguistics and translation and has ample experiences in the analytic assessment of both mt outputs and human translations. as material for this case study, we used the documents in the asian language treebank (alt) @xcite . 5 ## data collection to clarify the limitations of the text-to-text approach for mt while acknowledging its status, we began with the outputs of a reasonably strong nmt system and collected examples of translation issues with their revisions through a modified version of the two-stage pe workflow originally proposed by @xcite . our procedure is as follows. stage (1) segment-level text-to-text nmt: given source documents are translated by an mt system, which is preferably the one that can produce a translation of exploitable quality. we regard a segment-level text-to-text nmt as the subject. stage (2) segment-level minimal pe: each segment-level mt output is separately postedited without referring to any information other than the segment itself, for example, other segments in the same document and other reference documents. to avoid introducing any preferences from human workers, this stage allows only minimal edits. ## manual analysis of translation issues our post-edited translation data contain two separate and different types of translation issues: the remaining issues from the segment-level text-to-text mt, and the issues that require information other than the individual segments to resolve. we manually analyzed the latter translation issues resolved during the document-level full pe in stage (3). first, using tercom, we automatically identified the corresponding text spans in the two versions of the translations obtained in stages (2) and (3). then, we manually extracted pairs of text spans: one for an issue in the segment-level pe result, and the other for its revision in the document-level pe result. as a result, we obtained 529 such revision examples. finally, we annotated each revision example with the following three types of labels. ## conclusion to analytically assess issues that cannot be resolved by text-to-text processing, such as textto-text mt, this paper presented our specific constraints incorporated into the two-stage pe pipeline originally proposed by @xcite . in a case study on the english-to-japanese news translation task, we found that translation issues beyond text-to-text processing are caused by a lack of extra-document and/or non-linguistic information, such as fine-grained style specifications, terminology, domain-specific knowledge, and reference documents. the resulted parallel data and annotated revision examples are publicly available. 19 our method is laborious and requires very high competence in both linguistics and translation. nevertheless, it is applicable to other translation tasks where we can build an mt system that can produce translation of exploitable quality. we thus hope other researchers use our method to assess the limitations of text-to-text processing and the remaining issues in a wide range of translation tasks. we plan to introduce another document-level minimal pe stage in order to assess the attainable translation by document-level mt. while clarifying the limitations, we also suggested how we can enable mt systems to explicitly refer to extra-document and/or non-linguistic information. we plan to evaluate the impact of enforcing decoding with external knowledge, such as terminologies and style specifications. an important issue in present-day society was also illuminated: the need to cultivate translation literacy and mt literacy in people to avoid the risk caused by the innocent use of mt services. to tackle this, we are currently compiling educational materials to help people understand translation, mt, and their differences. we will also analyze various levels of competences required for human translators, following the competence framework developed by the european master's in translation @xcite .