A Multilingual Multiway Evaluation Data Set for Structured Document Translation of A sian Languages (2022)

Abstract: Translation of structured content is an important application of machine translation, but the scarcity of evaluation data sets, especially for Asian languages, limits progress. In this paper we present a novel multilingual multiway evaluation data set for the translation of structured documents of the Asian languages Japanese, Korean and Chinese. We describe the data set, its creation process and important characteristics, followed by establishing and evaluating baselines using the direct translation as well as detag-project approaches. Our data set is well suited for multilingual evaluation, and it contains richer annotation tag sets than existing data sets. Our results show that massively multilingual translation models like M2M-100 and mBART-50 perform surprisingly well despite not being explicitly trained to handle structured content. The data set described in this paper and used in our experiments is released publicly.

## introduction a common use case of machine translation (mt) is the translation of structured or formatted documents, such as web pages. the key challenge is to properly transfer markup tags within the translatable content (e.g. bold) from the source to the target language during the translation process. a markup example is shown in figure 1 . although there are various data sets for sentence-and document-level machine translation, apart from @xcite and @xcite we are not aware of any other data sets for evaluating the translation quality of markup annotated sentences. this paper introduces a data set that reflects all those aspects to facilitate and foster research that goes beyond the translation of plain text in isolation. in this paper, we describe the second release of the software documentation data set for machine translation, a high-quality multilingual evaluation data set for machine translation in the it domain. 1 it has been released by sap foot_1 , a large enterprise software company. the contents originate from the sap help portal foot_2 that contains documentation and learning materials for sap products. with this release of the data set, we publish development and test data for mt purposes in the form of complete structured documents that include segment-internal (inline) markup, in a rich xml-based localization format as well as transformations that make it readily usable in many standard machine translation workflows. it consists of 385 documents that contain about 4,000 translatable segments and their translations. with the second release, we focus on the following major asian languages: japanese (ja), korean (ko), and chinese (zh). translations have been produced from the same english (en) source, thus the data is multiway parallel. the multiway document-level nature of this data set enables not only evaluation of multilingual models but also document-level translation approaches (if needed) when translating structured content. additionally, in this paper, we establish base-lines for the released data set for individual segment translation, where we utilize massively multilingual models such as m2m-100 @xcite and mbart-50 @xcite , making use of out-of-the-box publicly available checkpoints already trained in a many-to-many translation fashion with no additional fine-tuning on our end. we show that these models can be used for directly translating structured content despite not being explicitly trained to do so. we observe that the quality of the direct translation approach, where the source text is composed of both lexical and markup content, is comparable to the traditional detag-project approach. we then report translation results according to several metrics targeting not only the translation quality but also tag placement accuracy, allowing us to understand the difficulty of translating structured content into asian languages. ## related work only recently awareness has increased that real world content often resides in structured and formatted documents such as html pages and microsoft office formats, and that the transfer of inline markup tags is a challenge for neural machine translation; correspondingly, little work has been published. @xcite present a data set from the it domain that features inline markup, and corresponding mt results using a constrained beam search approach for decoding. furthermore, @xcite compare different data augmentation methods with a detagproject approach, and evaluate on data from legal documents from the european union. the methods for tag transfer in @xcite are also related, even though they focus on inserting the tags into a fixed human translation. in contrast to the previously mentioned available data sets, with the software documentation data set for machine translation, we publish complete documents of high translation quality, thus allowing for context-sensitive translation, such as in @xcite for example, and incontext evaluation as it has been shown to be vital for accurate evaluation assessments @xcite . furthermore, our data set is multiway multilingual, focuses on asian languages and adds lower resource asian languages to the picture to enable a more comprehensive evaluation of different methods. while @xcite enables evaluation of 14 translation directions to or from english, they do not support non-english translation directions as their evaluation data is not n-way parallel. in contrast, the second release of the software documentation data set is n-way parallel enabling 6 translation directions to and from english as well as 6 translation directions between the asian languages leading to a total of 12 directions. the first release of the software documentation data set for machine translation is described in buschbeck and exel (2020). while it also contains complete documents with rich metadata on the segment level and is therefore well suited to evaluate contextual approaches to mt, it does not feature complete hierarchical document structure. its focus is low-resource language pairs that are typically under-represented in mt research, namely english to hindi, indonesian, malay and thai. in terms of methods, according to our knowledge, we are the first to report results on tag transfer using pre-trained massively multilingual translation models mbart-50 and m2m-100 @xcite . we also compare with the detag-project approach, but leave the exploration of other methods on this data set for future work. ## the structured documents data set we describe the second release of the software documentation data set for machine translation, our data set for structured document translation of asian languages. ## baseline experiments we propose to evaluate the translation performance of out-of-the-box pretrained multilingual neural machine translation (nmt) systems for the english to japanese, korean and chinese translation directions. 6 we focus on segment-level translation and propose to leave document-level approaches for future work. ## conclusion in this paper we have presented our multilingual multiway evaluation data set for structured document translation of three asian languages, japanese, korean and chinese -the second release of the software documentation data set for machine translation. our data set contains rich annotation tag sets and is well suited for multilingual natural language processing tasks such as mt and its evaluation. we have established and evaluated mt baselines using two methods to handle inline markup, namely the direct translation and the detag-project approaches. our results show that massively multilingual translation models like m2m-100 and mbart-50 perform surprisingly well despite not being explicitly trained to handle structured content. this previously unknown capability of mt models used in our experiments deserves further exploration, especially in combination with document-level translation approaches.