NICT ’s Submission to the WAT 2022 Structured Document Translation Task (2022)

Abstract: We present our submission to the structured document translation task organized by WAT 2022. In structured document translation, the key challenge is the handling of inline tags, which annotate text. Specifically, the text that is annotated by tags, should be translated in such a way that in the translation should contain the tags annotating the translation. This challenge is further compounded by the lack of training data containing sentence pairs with inline XML tag annotated content. However, to our surprise, we find that existing multilingual NMT systems are able to handle the translation of text annotated with XML tags without any explicit training on data containing said tags. Specifically, massively multilingual translation models like M2M-100 perform well despite not being explicitly trained to handle structured content. This direct translation approach is often either as good as if not better than the traditional approach of “remove tag, translate and re-inject tag” also known as the “detag-and-project” approach.

## introduction neural machine translation @xcite using transformers @xcite is gradually beginning to reach a saturation point in terms of translation quality for major languages like english, french, japanese, chinese @xcite . most existing work focus on the translation of plain text, where the sentence is translated individually or by considering its context via a document level translation approach @xcite . however, this does not directly address an important real life application: "web page translation". web pages are structured documents containing formatted or annotated text, where the annotation is done via inline tags or xml tags. when translating web pages, care must be taken to translate not only the text but also the xml tags. for example, this is @xmath0. is an example of a sentence in a structured document. its translation in spanish should be: esta es @xmath1. where @xmath2. the structured document translation task foot_0 in wat 2022 aims at evaluating approaches for the translation of text with xml tags or inline tags. for a detailed overview of the task, kindly refer to the overview paper @xcite . since nmt models are sensitive to what they are trained on, it is natural to assume that they should be exposed to examples of how to handle xml tags. unfortunately, there is a scarcity of training data containing xml tags to train nmt models to handle structured content. @xcite provide training data for 7 languages, but this is not possible for all languages. therefore, the most viable solution would be the "remove tag, translate and re-inject tag" approach also known as the detag-and-project approach @xcite shortened to dnp. the main problem with dnp is that it needs high quality word alignments and heuristic algorithms when reinserting the tags into the translation. therefore, poor translations, poor alignments and heuristics lead to compounding errors which can negatively affect the injection process leading to poor transfer of structure. in wat 2022, we participated under the team name "nict-5" where we applied the dnp approach to the structured document translation task for english to japanese/chinese/korean as well as japanese/chinese/korean to english translation. given the large availability of pre-trained translation models, we decided to use the m2m-100 model. in order to compare against the dnp approach, we translated sentences containing xml tags using this model and to our surprise, this approach was able to outperform the dnp approach in some instances. our analyses reveal that the dnp approach is better at transferring the xml tag structure but gives poor automatic evaluation scores in some cases as it fails to handle cases such as non-closing tags and the tag injection for words and phrases for which word alignment fails. in terms of methods, according to our knowledge, we are the first to report results on tag transfer using a massively multilingual translation model like m2m-100 @xcite which surprisingly lead to reasonable automatic evaluation scores. we also submit results for the dnp approach, but find that it does not always outperform the direct translation approach. for evaluation, wat uses the xml-bleu metric in accordance with @xcite but we additionally report on the xml tag structure accuracy to better understand the limitations of the approaches we used. ## approaches we use the direct translation (dit) and the detagand-project (dnp) approaches for our submissions. a. direct translation: in this approach, we directly translate the sentences with xml content in them. note that the following considerations are to be made: ## results we present in comparing the dnp and dit rows for our submissions, it can be seen that except for japanese↔english translation, dit is slightly if not substantially better than dnp. this is quite surprising since the m2m model was never explicitly trained to handle xml tags. it is possible that the model treats the tags as rare or unknown english tokens which are usually copied as is in japanese, chinese and korean translation. we leave this investigation for the future. with regard to the xml structure transfer accuracy, it is interesting that although the xml-bleu is higher for dnp, the structure transfer accuracy is lower. upon some manual investigation we found the following: overall, the dnp approach needs a lot of investment but the returns are not equivalent. future work should focus more on the dit approach which is end-to-end and hence more attractive. ## conclusion in this paper, we describe our submissions as team "nict-5" to the structured document translation task in wat 2022. we used the direct translation and the detag-and-project approaches and to our surprise found that the direct translation approach outperforms detag-and-project approach slightly or substantially depending on the language pair. our analyses reveal that the former approach has poorer tag structure transfer accuracy, but still is better than the latter approach, due to (a.) the latter's inability to handle the transfer of tags for content that can't be aligned with its translation and (b.) the latter's sensitivity to poor alignment. rather than working to improve the detag-and-project approach, we plan to focus more on the direct translation approach with constrained generation and some additional training to handle structured content more effectively.