U nihan LM : Coarse-to-Fine C hinese- J apanese Language Model Pretraining with the Unihan Database (2020)

Abstract: Chinese and Japanese share many characters with similar surface morphology. To better utilize the shared knowledge across the languages, we propose UnihanLM, a self-supervised Chinese-Japanese pretrained masked language model (MLM) with a novel two-stage coarse-to-fine training approach. We exploit Unihan, a ready-made database constructed by linguistic experts to first merge morphologically similar characters into clusters. The resulting clusters are used to replace the original characters in sentences for the coarse-grained pretraining of the MLM. Then, we restore the clusters back to the original characters in sentences for the fine-grained pretraining to learn the representation of the specific characters. We conduct extensive experiments on a variety of Chinese and Japanese NLP benchmarks, showing that our proposed UnihanLM is effective on both mono- and cross-lingual Chinese and Japanese tasks, shedding light on a new path to exploit the homology of languages.

## introduction recently, pretrained language models have shown promising performance on many nlp tasks @xcite @xcite @xcite . many attempts have been made to train a model that supports multiple languages. among them, multilingual bert (mbert) @xcite is released as a part of bert. it directly adopts the same model architecture and training objective, and is trained on wikipedia in different languages. xlm (lample and @xcite is proposed with an additional language embedding and a new training typhoon is a type of tropical depression. to facilitate lexical sharing, we propose unihan language model (unihanlm), a cross-lingual pretrained masked language model for chinese and japanese. we propose a two-stage coarse-to-fine pretraining procedure to empower better generalization and take advantages of shared characters in japanese, traditional and simplified chinese. first, we let the model exploit maximum possible shared lexical information. instead of learning a shared sub-word vocabulary like the prior work, we leverage unihan database @xcite , a ready-made constituent of the unicode standard, to extract the shared lexical information across the languages. by exploiting this database, we can effectively merge characters with the similar surface morphology but independent unicodes, as shown in to verify the effectiveness of our approach, we evaluate on both lexical and semantic tasks in chinese and japanese. on word segmentation, our model outperforms monolingual and multilingual bert @xcite and shows a much higher performance on cross-lingual zeroshot transfer. also, our model achieves state-of-theart performance on unsupervised chinese-japanese machine translation, and is even comparable to the supervised baseline on chinese-to-japanese translation. on classification tasks, our model achieves a comparable performance with monolingual bert and other cross-lingual models trained with the same scale of data. to summarize, our contributions are three-fold: ## unihanlm in this section, we introduce the tokenization, character merging and training procedure for our proposed unihanlm. ## experiments in this section, we compare unihanlm with other self-supervised pretrained language models. all of our baselines (monolingual bert, mbert and xlm) use wikipedia for self-supervised pretraining. note that we do not compare our model to xlm-r @xcite and unicoder @xcite since they are trained with much more data and even on supervised tasks. ## related work multilingual representation learning learning cross-lingual representations are useful for downstream tasks such as cross-lingual classification @xcite , cross-lingual retrieval @xcite and cross-lingual qa @xcite @xcite . earlier work on multilingual representations exploiting parallel corpora @xcite or a bilingual dictionary to learn a linear mapping @xcite . subsequent methods explored self-training @xcite and unsupervised learning @xcite @xcite . recently, multilingual pretrained encoders have shown its effectiveness for learning deep cross-lingual representations @xcite @xcite @xcite . word segmentation word segmentation is often formalized as a sequence tagging task. it requires lexical knowledge to split a character sequence into a word list that can be used for downstream tasks. this step is necessary for many earlier nlp systems for chinese and japanese. recent work on chinese word segmentation @xcite @xcite @xcite and japanese word segmentation @xcite @xcite exploit deep neural networks and focus on building end-to-end sequence tagging models. ## discussion and future work there is still space to improve for our method. first, as we analyze, except for chinese characters, english words often appear in both chinese and japanese texts. in our current model, they are treated as normal characters without any special processing. however, such a rough processing may harm the performance of the model on some tasks. for example, in paws-x, many entities remain untranslated and this may have a negative effect on the performance of our model. also, loan words (i.e., gairaigo), especially from english, constitute a large part of nouns in modern japanese @xcite . these words are written with kanas, instead of chinese characters which makes it inapplicable to be shared with our approach. thus, it may be reasonable to involve english in cross-lingual modeling of asian languages, as well. similarly, chinese characters exist in korean and vietnamese but are now written in hangul (korean alphabet) and vietnamese alphabet, respectively. our future work will explore the possibility to generalize the idea to more asian languages including korean and vietnamese. ## conclusion in this paper, we exploit the ready-made unihan database constructed by linguistic experts and propose a novel chinese-japanese cross-lingual language model trained by a two-stage coarse-to-fine procedure. our extensive experiments on word segmentation, unsupervised machine translation and text classification verify the effectiveness of our model. our approach sheds some light on the linguistic features that receive insufficient attention recently and showcases a novel way to fuse human linguistic knowledge and exploit the similarity between two languages.