C hinese BERT : C hinese Pretraining Enhanced by Glyph and P inyin Information (2021)

Abstract: Recent pretraining models in Chinese neglect two important aspects specific to the Chinese language: glyph and pinyin, which carry significant syntax and semantic information for language understanding. In this work, we propose ChineseBERT, which incorporates both the glyph and pinyin information of Chinese characters into language model pretraining. The glyph embedding is obtained based on different fonts of a Chinese character, being able to capture character semantics from the visual features, and the pinyin embedding characterizes the pronunciation of Chinese characters, which handles the highly prevalent heteronym phenomenon in Chinese (the same character has different pronunciations with different meanings). Pretrained on large-scale unlabeled Chinese corpus, the proposed ChineseBERT model yields significant performance boost over baseline models with fewer training steps. The proposed model achieves new SOTA performances on a wide range of Chinese NLP tasks, including machine reading comprehension, natural language inference, text classification, sentence pair matching, and competitive performances in named entity recognition and word segmentation.

## introduction large-scale pretrained models have become a fundamental backbone for various natural language processing tasks such as natural language understanding @xcite ), text classification @xcite and question answering @xcite . apart from english nlp tasks, pretrained models have also demonstrated their effectiveness for various chinese nlp tasks @xcite @xcite . since pretraining models are originally designed for english, two important aspects specific to the chinese language are missing in current large-scale pretraining: glyph-based information and pinyinbased information. for the former, a key aspect that makes chinese distinguishable from languages such as english, german, is that chinese is a logographic language. the logographic of characters encodes semantic information. for example, "液(liquid)", "河(river)" and "湖(lake)" all have the radical "氵(water)", which indicates that they are all related to water in semantics. intuitively, the rich semantics behind chinese character glyphs should enhance the expressiveness of chinese nlp models. this idea has motivated a variety of of work on learning and incorporating chinese glyph information into neural models @xcite @xcite @xcite , but not yet large-scale pretraining. for the latter, pinyin, the romanized sequence of a chinese character representing its pronunciation(s), is crucial in modeling both semantic and syntax information that can not be captured by contextualized or glyph embeddings. this aspect is especially important considering the highly prevalent heteronym phenomenon in chinese foot_1 , where the same character have multiple pronunciations, each of which is associated with a specific meaning. each pronunciation is associated with a specific pinyin expression. at the semantic level, for example, the chinese character "乐" has two distinctly different pronunciations: "乐" can be pronounced as "yuè [ye 51 ]", which means "music", and "lè [lg 51 ]", which means "happy". on the syntax level, pronunciations help identify the part-of-speech of a character. for example, character "还" has two pronunciations: "huán[xwan 35 ]" and "hái [xai 35 ]", with the former meaning the verb "return" and the latter meaning the adverb "also". different pronunciations of the same character cannot be distinguished by the glyph embedding since the logographic is the same, or the char-id embedding, since they both point to the same character id, but can be characterized by pinyin. in this work, we propose chinesebert, a model that incorporates the glyph and pinyin information of chinese characters into the process of largescale pretraining. the glyph embedding is based on different fonts of a chinese character, being able to capture character semantics from the visual surface character forms. the pinyin embedding models different semantic meanings that share the same character form and thus bypasses the limitation of interwound morphemes behind a single character. for a chinese character, the glyph embedding, the pinyin embedding and the character embedding are combined to form a fusion embedding, which models the distinctive semantic property of that character. with less training data and fewer training epochs, chinesebert achieves significant performance boost over baselines across a wide range of chinese nlp tasks. it achieves new sota performances on a wide range of chinese nlp tasks，including machine reading comprehension, natural language inference, text classification, sentence pair matching, and results comparable to sota performances in named entity recognition and word segmentation. ## experiments we conduct extensive experiments on a variety of chinese nlp tasks. models are separately finetuned on task-specific datasets for evaluation. concretely, we use the following tasks: we compare chinesebert to current state-ofthe-art ernie @xcite , bertwwm @xcite and macbert @xcite models. ernie adopts various masking strategies including token-level, phrase-level and entity-level masking to pretrain bert on largescale heterogeneous data. bert-wwm/robertawwm continues pretraining on top of official pretrained chinese bert/roberta models with the whole word masking pretraining strategy. unless otherwise specified, we use bert/roberta to represent bert-wwm/roberta-wwm and omit "wwm". macbert improves upon roberta by using the mlm-as-correlation (mac) pretraining strategy as well as the sentence-order prediction (sop) task. it is worth noting that bert and bertwwm do not have the large version available online, and we thus omit the corresponding performances. a comparison of these models is shown in ## conclusion in this paper, we introduce chinesebert, a largescale pretraining chinese nlp model. it leverages the glyph and pinyin information of chinese characters to enhance the model's ability of capturing context semantics from surface character forms and disambiguating polyphonic characters in chinese. the proposed chinesebert model achieves significant performance boost across a wide range of chinese nlp tasks. the proposed chinesebert performs better than vanilla pretrained models with less training data, indicating that the introduced glyph embeddings and pinyin embeddings serve as a strong regularizer for semantic modeling in chinese. future work involves training a large size version of chinesebert.