Enhancing C hinese Pre-trained Language Model via Heterogeneous Linguistics Graph (2022)

Abstract: Chinese pre-trained language models usually exploit contextual character information to learn representations, while ignoring the linguistics knowledge, e.g., word and sentence information. Hence, we propose a task-free enhancement module termed as Heterogeneous Linguistics Graph (HLG) to enhance Chinese pre-trained language models by integrating linguistics knowledge. Specifically, we construct a hierarchical heterogeneous graph to model the characteristics linguistics structure of Chinese language, and conduct a graph-based method to summarize and concretize information on different granularities of Chinese linguistics hierarchies. Experimental results demonstrate our model has the ability to improve the performance of vanilla BERT, BERTwwm and ERNIE 1.0 on 6 natural language processing tasks with 10 benchmark datasets. Further, the detailed experimental analyses have proven that this kind of modelization achieves more improvements compared with previous strong baseline MWA. Meanwhile, our model introduces far fewer parameters (about half of MWA) and the training/inference speed is about 7x faster than MWA.

## introduction pre-trained language models (plm) @xcite @xcite have recently demonstrated the effectiveness on a variety of natural language processing (nlp) tasks, such as machine translation and text summarization. for a specific downstream task, the parameters of plms can be fine-tuned with accurately labeled instances or weakly labeled instances of the task to achieve better performance. in recent, there are a series of studies on adapting plms for chinese @xcite @xcite @xcite @xcite . many researchers introduce the chinese-specific linguistics knowledge such as word information into plms by conducting elaborate self-supervised tasks to pretrain chinese plms from scratch. nevertheless, pre-training a plm is computationally expensive and time-consuming since it needs large-scale chinese corpus and heavy computational resources. the high cost makes it difficult for researchers to pre-train a plm from scratch. an alternative way is to integrate the chinesespecific linguistics knowledge into pre-trained plms in the fine-tuning stage in downstream tasks directly. following this idea, the task-free enhancement module is widely used in the fine-tuning stage by adding an additional adapter in plms to integrate external knowledge @xcite . as shown in figure 1 , the enhancement module is inserted between plms and task-specific module, and its inputs are the hidden representations of plms and embeddings of external knowledge. to achieve the goal of integrating external knowledge into plms in the fine-tuning stage, the enhancement module should have the following characteristics. first, as a plug-in adapter module in finetuning stage, it should maintain consistent output formulation with plm. second, it should not introduce unacceptable time or space complexity for training and inference. third, it should improve the performance of downstream tasks universally. ## heterogeneous linguistics graph this section introduces the components of our model hlg which instantiates the enhancement module by exploiting the cws information. we first briefly explain the graph convolutional network as our base encoder, and then describe the graph construction of hlg. finally, we give the details of the multi-step information propagation (msip) to integrate the cws information into plms. ## related works pre-training language models, such as elmo @xcite , bert @xcite , xl-net @xcite and gpt @xcite , have shown their powerful performances on various natural language processing tasks and have been applied in many applications. in recent past, there are studies adapting plms for chinese with chinese-specific features such as word information. glyce @xcite proposed to use the glyph information of chinese characters to enhance plms. ernie 1.0/2.0 @xcite and bertwwm @xcite used the whole word mask to learn the structure of words or entities in the pre-training stage and conducted more and better pre-training tasks to perceive large-scale data. nezha @xcite used a series of methods such as functional relative positional encoding and whole word masking to improve the pre-training tasks, which had brought improvement. zen @xcite adopted n-gram masking to enhance pre-trained encoder and obtained outstanding performance. lattice-bert @xcite introduced word lattice information @xcite into pre-training framework via lattice position attention. as a fundamental feature of chinese, word segmentation information is flexibility, granularity, and easy-to-get @xcite . further, zhang et al. (2018); li et al. (2019, 2020 ) conducted detailed research and experiments on the application of cws in deep learning, and found that cws information can effectively improve the performance of chinese character-based plms. recently, a lot of works have been proposed to prompt nlp applications by constructing graph on text and modeling with graph neural networks. @xcite first constructed word co-occurrence graph between documents and introduced gcn to modeling and aggregating document representation for text classification. @xcite ; @xcite constructed lattice graph to maintain multi-granularity information and external knowledge in chinese short text matching task. nguyen and grishman (2018) proposed performing gcn over dependency trees to extract event trigger. @xcite conducted a character-word interaction graph and performed graph attention network on it to recognize chinese named entities. @xcite introduced a bipartite-graph based transformer plm for integrating hierarchical semantic information. ## conclusion in this paper, we propose hlg which acts as the enhancement module to enhance chinese plms with cws information. the hlg firstly constructs heterogeneous graph based on multiple word segmentations to model the hierarchy of chinese. then, the msip is proposed to model the fine-grained linguistics knowledge of the heterogeneous graph. experimental results on 6 nlp tasks with 10 benchmark datasets demonstrate that the performance of our model outperforms previous work, mwa. besides the performance improvements, hlg introduces only half the additional parameters of mwa and its training/inference speed is 7x faster than mwa. at present, the experimental results of hlg are lagging behind sota, and we will try to migrate it to some of the latest plms. besides, hlg has the expansibility to introduce the representation layer of the cws model directly, or introduce some other information sources such as the knowledge graph, etc. we leave these further improvements to the future.