K orean-to- J apanese Neural Machine Translation System using Hanja Information (2020)

Abstract: In this paper, we describe our TMU neural machine translation (NMT) system submitted for the Patent task (Korean→Japanese) of the 7th Workshop on Asian Translation (WAT 2020, Nakazawa et al., 2020). We propose a novel method to train a Korean-to-Japanese translation model. Specifically, we focus on the vocabulary overlap of Korean Hanja words and Japanese Kanji words, and propose strategies to leverage Hanja information. Our experiment shows that Hanja information is effective within a specific domain, leading to an improvement in the BLEU scores by +1.09 points compared to the baseline.

## introduction the japanese and korean languages have a strong connection with chinese owing to cultural and historical reasons @xcite . many words in japanese are composed of chinese characters called kanji. by contrast, korean uses the korean alphabet called hangul to write sentences in almost all cases. however, sino-korean 1 (sk) words, which can be converted into hanja words, account for 65 percent of the korean lexicon @xcite . in addition, several studies have suggested that overlapping tokens between the source and target languages can improve the translation accuracy @xcite . @xcite trained a korean-to-chinese translation model by converting korean sk words from hangul into hanja to increase the vocabulary overlap. in other words, the meaning of a vocabulary overlap on nmt is that each corresponding word's embeddings are the same. @xcite and @xcite improved translation accuracy by making embeddings between source words and their corresponding target words closer. from this fact, we hypothesize that if the embeddings of each corresponding word are closer, the translation accuracy will improve. based on this hypothesis, we propose two approaches to train a translation model. first, we follow @xcite 's method to increase the vocabulary overlap to improve the korean-to-japanese translation accuracy. therefore, we perform hangul to hanja conversion pre-processing before training the translation model. second, we propose another approach to obtain korean and japanese embeddings that are closer to korean sk words and their corresponding japanese kanji words. sk words written in hangul and their counterparts in japanese kanji are superficially different, but we make both embeddings close by using a loss function when training the translation model. in addition, in this study, we used the japan patent office patent corpus 2.0, which consists of four domains, namely chemistry (ch), electricity (el), mechanical engineering (me), and physics (ph), whose training, development, and test-n1 2 data have domain information. our methods are more effective when the terms are derived from chinese characters; therefore, we expect that the effect will be different per domain. this is because korean (hangul) 그래서 n의 함유량은 0.01 %이하로 한정한다. korean (hanja) 그래서 n의 含有量은 0.01 %以下로 限定한다. japanese そのため nの 含有量は 0.01 %以下に 限定する。 english therefore, n content 0.01 % or less limit. in this study, we examine the effect of hanja information and domain adaptation in a koreanto-japanese translation. the main contributions of this study are as follows: ## related work several studies have been conducted on korean-japanese neural machine translation (nmt). @xcite trained a korean-to-japanese translation model using a transformer-based nmt system with relative positioning, back-translation, and multi-source methods. there have been other attempts that combine statistical machine translation (smt) and nmt @xcite . previous studies on korean-japanese nmt did not use hanja information, whereas we train a korean-to-japanese translation model using data in which sk words were converted into hanja words. @xcite proposed byte-pair encoding (bpe), i.e., a sub-word segmentation method, and suggested that overlapping tokens by joint bpe is more effective for training the translation model between european language pairs. zhang and komachi (2019) increased the overlap of tokens between japanese and chinese by decomposing japanese kanji and chinese characters into an ideograph or stroke level to improve the accuracy of chinese-japanese unsupervised nmt. following previous studies, we convert korean sk words from hangul into hanja to increase the vocabulary overlap. @xcite proposed a method to build a bilingual dictionary by aligning the source and target word embedding spaces using a rotation matrix. they showed that word-by-word translation using the bilingual dictionary can improve the translation accuracy in low-resource language pairs. lample and @xcite improved the translation accuracy by fine-tuning a pre-trained cross-lingual language model (xlm). the authors observed that the bilingual word embeddings in xlm are similar. based on these facts, we hypothesize that if the embeddings of each corresponding word become closer, the translation accuracy will improve. in this study, we use hanja information to make the embeddings of each corresponding word closer to each other. some studies have focused on exploiting hanja information. @xcite focused on the linguistic connection between korean and chinese. they used the parallel data in which the korean sentences of sk words were converted into hanja to train a translation model and demonstrated that their method improves the translation quality. in addition, they showed that conversion into hanja helped translate the homophone of sk words. @xcite proposed an approach of training korean word representations using the data in which sk words were converted into hanja words. they demonstrated the effectiveness of the representation learning method on several downstream tasks, such as a news headline generation and sentiment analysis. to train the korean-to-japanese translation model, we combine these two approaches using hanja information for training the translation model and word embeddings to improve the translation quality. ## nmt with hanja information our model is based on the transformer architecture @xcite , and we share the embedding weights between the encoder input, decoder input, and output to make better use of the vocabulary overlap between korean and japanese. we do not use language embedding @xcite to distinguish the source and target languages. we propose two models using the hanja information described below. ## domain adaptation we examine the effect of domain adaptation, which uses domain-specific data for retraining the pretrained model trained by all training data. we translate the test data for each domain using a domainspecific translation model. for training and validation, we split the training and development data into four domains: chemistry, electricity, mechanical engineering, and physics using domain information. we use these data to build domain-specific translation models. for testing, we use the domain information annotated with the test-n1 data. however, the test-n2 and test-n3 data do not have domain information. therefore, we train a domain prediction model by fine-tuning korean or japanese bert @xcite using the labeled training data of the japan patent office patent corpus 2.0 to predict the domain information of test-n2 and test-n3 data. ## conclusions and future work in this paper, we described our nmt system submitted to the patent task (@xmath0: a successful output example of the hanja-loss model. in one form, r1 and r2 include one or more deuterium atoms. source we also demonstrated that the effect of our proposed method is different for all domain data. however, some sk words are polysemous. our proposed method treats embeddings of such sk words the same and cannot address this problem. therefore, the problem of polysemous words is a major challenge for our proposed method. in this study, we focused on vocabulary overlap between korean hanja words and japanese kanji words. in addition, many hanja and kanji words are of chinese origin. therefore, in the future, we will attempt to develop a translation method that takes advantage of the vocabulary overlap among korean, japanese, and chinese.