Rethinking Dictionaries and Glyphs for C hinese Language Pre-training (2023)

Abstract: We introduce CDBert, a new learning paradigm that enhances the semantics understanding ability of the Chinese PLMs with dictionary knowledge and structure of Chinese characters. We name the two core modules of CDBert as Shuowen and Jiezi, where Shuowen refers to the process of retrieving the most appropriate meaning from Chinese dictionaries and Jiezi refers to the process of enhancing characters’ glyph representations with structure understanding. To facilitate dictionary understanding, we propose three pre-training tasks, i.e.„ Masked Entry Modeling, Contrastive Learning for Synonym and Antonym, and Example Learning. We evaluate our method on both modern Chinese understanding benchmark CLUE and ancient Chinese benchmark CCLUE. Moreover, we propose a new polysemy discrimination task PolyMRC based on the collected dictionary of ancient Chinese. Our paradigm demonstrates consistent improvements on previous Chinese PLMs across all tasks. Moreover, our approach yields significant boosting on few-shot setting of ancient Chinese understanding.

## introduction large-scale pre-trained language models (plms) such as bert @xcite and gpt @xcite have revolutionized various research fields in natural language processing (nlp) landscape, including language generation @xcite ), text classification @xcite , language reasoning @xcite , etc. the de facto paradigm to build such lms is to feed massive training corpus and datasets to a transformer-based language model with billions of parameters. apart from english plms, similar approaches have also been attempted in multilingual (lample and conneau, 2019) and chinese language understanding tasks @xcite . to enhance chinese character representations, pioneer works have incorporated additional character information, including glyph (character's geometric shape), pinyin (character's pronunciation), and stroke (character's writing order) @xcite . nevertheless, there still exists a huge performance gap between concurrent stateof-the-art (sota) english plms and those on chinese or other non-latin languages @xcite , which leads us to rethink the central question: what are the unique aspects of chinese that are essential to achieve human-level chinese understanding? with an in-depth investigation of chinese lan-guage understanding, this work aims to point out the following crucial challenges that have barely been addressed in previous chinese plms. in this work, we propose cdbert, a new chinese pre-training paradigm that aims to go beyond feature aggregation and resort to mining information from chinese dictionaries and glyphs' structures, two essential sources that interpret chinese characters' meaning. we name the two core modules of cdbert as shuowen and jiezi, in homage to one of the earliest chinese dictionary in han dynasty. figure 1 depicts the overall model. shuowen refers to the process that finds the most appropriate definition of a character in a chinese dictionary. indeed, resorting to dictionaries for chinese understanding is not unusual even for chinese linguistic experts, especially when it comes to ancient chinese (aka. classical chinese) understanding. different from previous works that simply use dictionaries as additional text corpus @xcite , we propose a fine-grained definition retrieval framework from chinese dictionaries. specifically, we design three types of objectives for dictionary pre-training: masked en-1 https://github.com/ymcui/chinese-bert-wwm try modeling (mem) to learn entry representation; contrastive learning objective with synonyms and antonyms; example leaning (el) to distinguish polysemy by example in the dictionary. jiezi refers to the process of decomposing and understanding the semantic information existing in the glyph information. such a process grants native chinese the capability of understanding new characters. in cdbert, we leverage radical embeddings and previous success of clip @xcite model to enhance model's glyph understanding capability. we evaluate cdbert with extensive experiments and demonstrate consistent improvements of previous baselines on both modern chinese and ancient chinese understanding benchmarks. it is worth noting that our method gets significant improvement on cclue-mrc task in few-shot setting. additionally, we construct a new dataset aiming to test models' ability to distinguish polysemy in chinese. based on the baiduhanyu, we construct a polysemy machine reading comprehension task (polymrc). given the example and entry, the model needs to choose a proper definition from multiple interpretations of the entry. we believe our benchmark will help the development of chinese semantics understanding. in summary, the contributions of this work are four-fold: (i) we propose cdbert, a new learning paradigm for improving plms with chinese dictionary and characters' glyph representation; (ii) we derive three pre-training tasks, masked entry modeling, contrastive learning for synonym and antonym, and example learning, for learning a dictionary knowledge base with a polysemy retriever (sec. 3.1); (iii) we propose a new task polymrc, specially designed for benchmarking model's ability on distinguishing polysemy in ancient chinese. this new task complements existing benchmarks for chinese semantics understanding (sec. 4); (iv) we systematically evaluate and analyze the cdbert on both modern chinese and ancient chinese nlp tasks, and demonstrate improvements across all these tasks among different types of plms. in particular, we obtain significant performance boost for few-shot setting in ancient chinese understanding. gograms. at an early stage, devlin et al. (2018); liu et al. (2019b) propose bert-like language models with character-level masking strategy on chinese corpus. sun et al. (2019b) take phraselevel and entity-level masking strategies to learn multi-granularity semantics for plm. cui et al. (2019) pre-trained transformers by masking all characters within a chinese word. lai et al. (2021) learn multi-granularity information with a constructed lattice graph. recently, zhang et al. 2020); zeng et al. (2021); su et al. (2022b pretrained billion-scale parameters large language models for chinese understanding and generation. in addition to improving masking strategies or model size, some researchers probe the semantics from the structure of chinese characters to enhance the word embedding. since chinese characters are composed of radicals, components, and strokes hierarchically, various works @xcite @xcite @xcite @xcite learn the chinese word embedding through combining indexed radical embedding or hierarchical graph. benefiting from the strong representation capability of convolutional neural networks (cnns), some researchers try to learn the morphological information directly from the glyph @xcite @xcite @xcite . sehanobish and song (2020); xuan et al. (2020) apply the glyph-embedding to improve the performance of bert on named entity recognition (ner). besides, polysemy is common among chinese characters, where one character may correspond to different meanings with different pronunciations. therefore, @xcite use "pinyin" to assist modeling in distinguishing chinese words. sun et al. (2021c) knowledge augmented pre-training although plms have shown great success on many nlp tasks. there are many limitations on reasoning tasks and domain-specific tasks, where the data of downstream tasks vary from training corpus in distribution. even for the strongest llm chat-gpt, which achieves significant performance boost across a wide range of nlp tasks, it is not able to answer questions involving up-to-date knowledge. and it is impossible to train llms frequently due to the terrifying costs. as a result, researchers have been dedicated to injecting various types of knowledge into plm/llm. according to the types, knowledge in existing methods can be classified to text knowledge @xcite and graph knowledge, where text knowledge can be further divided into linguistic knowledge and non-linguistic knowledge. specifically, some works took lexical information @xcite @xcite or syntax tree @xcite @xcite to enhance the ability of plms in linguistic tasks. for the nonlinguistic knowledge, some researchers incorporate general knowledge such as wikipedia with retrieval methods @xcite @xcite to improve the performance on downstream tasks, others use domain-specific corpora @xcite to transfer the plms to corresponding downstream tasks. compared with text knowledge, a knowledge graph contains more structured information and is better for reasoning. thus a flourish of work @xcite @xcite @xcite designed fusion methods to combine the kg with plms. dictionary augmented pre-training considering the heavy-tailed distribution of the pre-training corpus and difficult access to the knowledge graph, some works injected dictionary knowledge into plms to alleviate the above problems. @xcite enhance plm with rare word definitions from english dictionaries. @xcite pretrained bert with english dictionary as a pretraining corpus and adopt an attention-based infusion mechanism for downstream tasks. ## the polymrc task most existing chinese language understanding evaluation benchmarks do not require the model to have strong semantics understanding ability. hence, we propose a new dataset and a new machine reading comprehension task focusing on polysemy understanding. specifically, we construct a ## limitations we collect the dictionary from the internet, and although we make effort to reduce replicate explanations, there is noise in the dictionary. besides, not all the words are included in the dictionary. in other words, the quality and amount of entries in the chinese dictionary are to be improved. additionally, our method is pre-trained on the bert-like transformers to enhance the corresponding plms, and can not be applied to llm directly whose frameworks are unavailable. in the future, we will use the retriever for disambiguation and dictionary knowledge infusion to llm. ## conclusion in this work, we leverage chinese dictionary and structure information of chinese characters to enhance the semantics understanding ability of plm. to make chinese dictionary knowledge better act on plm, we propose 3 pre-training objectives simulating looking up dictionary in our study, and incorporate radical or glyph features to cdbert. experiment results on both modern chinese tasks and ancient chinese tasks show our method significantly improve the semantic understanding ability of various plm. in the future, we will explore our method on more high-quality dictionaries (e.g. bilingual dictionary), and adapt our method to llm to lessen the semantic errors. besides, we will probe more fine-grained structure information of logograms in both understanding and generation tasks.