TMU NMT System with J apanese BART for the Patent task of WAT 2021 (2021)

Abstract: In this paper, we introduce our TMU Neural Machine Translation (NMT) system submitted for the Patent task (Korean Japanese and English Japanese) of 8th Workshop on Asian Translation (Nakazawa et al., 2021). Recently, several studies proposed pre-trained encoder-decoder models using monolingual data. One of the pre-trained models, BART (Lewis et al., 2020), was shown to improve translation accuracy via fine-tuning with bilingual data. However, they experimented only Romanian!English translation using English BART. In this paper, we examine the effectiveness of Japanese BART using Japan Patent Office Corpus 2.0. Our experiments indicate that Japanese BART can also improve translation accuracy in both Korean Japanese and English Japanese translations.

## introduction neural machine translation (nmt) has achieved high translation accuracy in large-scale data conditions. however, translation accuracy of nmt drops in the lack of bilingual data @xcite . there are several approaches such as backtranslation @xcite and transfer learning @xcite to address this problem. furthermore, in addition to these methods, there are some approaches to use pre-trained models using only monolingual data. bert @xcite , which is the most typical pre-trained model, can boost the accuracy of many downstream tasks compared to models without bert via fine-tuning with the task-specific training data. however, applying bert to nmt in fine-tuning form like the other tasks requires two-stage optimization and does not provide significant improvement @xcite . recently, several studies proposed pre-trained encoder-decoder models using a monolingual data. @xcite proposed bart, which is one of the pre-trained encoder-decoder models. they demonstrated that bart works well for not only comprehension tasks such as gleu @xcite and squad @xcite but also text generation tasks such as text summarization and translation. however, they reported only the effect of english bart, so they did not investigate bart trained by monolingual data of another language. furthermore, in the translation task, they experimented with only @xmath0, which have subword overlap. therefore, the effect in translations between language pairs without subword overlapping is not clear. furthermore, they did not experiment in translation direction where the source language matches the language of the pre-trained model. additionally, we consider that fine-tuning pretraining models such as bart in translation task is similar to transfer learning @xcite . transfer learning in nmt is a method that trains the network of the parent language pair (the parent model) as the initial network and then fine-tunes it for the child language pair (the child model). in the terminology of transfer learning, the pretrained bart and fine-tuned model are the parent model and child model, respectively. previous studies have shown that transfer learning works most efficiently when the source languages of the parent and child models are syntactically similar @xcite . therefore, we hypothesize that bart is more effective when the language pair for fine-tuning is syntactically similar to the pre-training language. in this study, we examine the effects of japanese bart on the translation task. we use korean/japanese and english/japanese bilingual data of japan patent office patent corpus 2.0 (jpo corpus) for fine-tuning. we also experiment in both translation directions of ko ja and en ja. ## related work there are some approaches pre-trained encoder models like bert @xcite to the nmt task. @xcite used bert as an encoder and demonstrated the effectiveness of two-stage optimization, which first trains parameters without bert encoder, and then fine-tunes all parameters. @xcite used bert representations as input embedding and showed more effectiveness than using bert as the encoder. recently, several studies proposed pre-trained encoder-decoder models such as mass @xcite and bart @xcite , and these models can improve the translation accuracy via fine-tuning with bilingual data. mass @xcite uses monolingual data from both the source and target languages for pre-training when applying to the nmt. on the contrary, bart @xcite uses only monolingual data of target language, unlike mass. @xcite trained multilingual bart (mbart) using monolingual data of 25 languages. they indicated that mbart initialization leads significant gains in low resource settings. however, @xcite showed that mbart cannot obtain improvements in the patent task. ## discussions we hypothesize that jabart is more effective when the language pair for fine-tuning is syntactically similar to the pre-training language, as in transfer learning. in our experimental settings, korean and english are syntactically similar and different languages with japanese, respectively foot_3 . therefore, we expect that jabart is more effective in the ko ja translations than in the en ja translations. however, ## conclusions in this paper, we described our nmt system submitted to the patent task (ko ja and en ja) of the 8th workshop on asian translation. we compared the baseline and fine-tuned jabart models, and demonstrated that the fine-tuned jabart achieves consistent improvements of bleu scores in language pairs with no subword overlapping, and irrespective of translation directions. contrary to our hypothesis, our experiments indicated no significant difference in the translation accuracy depending on the syntactic similarity. however, we consider that there are some differences in another aspect such as training process per epoch and network representations. therefore, we attempt to analyze bart fine-tuned using language pairs with varying syntactic proximities in detail in the future.