Revisiting and Advancing C hinese Natural Language Understanding with Accelerated Heterogeneous Knowledge Pre-training (2022)

Abstract: Recently, knowledge-enhanced pre-trained language models (KEPLMs) improve context-aware representations via learning from structured relations in knowledge bases, and/or linguistic knowledge from syntactic or dependency analysis. Unlike English, there is a lack of high-performing open-source Chinese KEPLMs in the natural language processing (NLP) community to support various language understanding applications. In this paper, we revisit and advance the development of Chinese natural language understanding with a series of novel Chinese KEPLMs released in various parameter sizes, namely CKBERT (Chinese knowledge-enhanced BERT). Specifically, both relational and linguistic knowledge is effectively injected into CKBERT based on two novel pre-training tasks, i.e., linguistic-aware masked language modeling and contrastive multi-hop relation modeling. Based on the above two pre-training paradigms and our in-house implemented TorchAccelerator, we have pre-trained base (110M), large (345M) and huge (1.3B) versions of CKBERT efficiently on GPU clusters. Experiments demonstrate that CKBERT consistently outperforms strong baselines for Chinese over various benchmark NLP tasks and in terms of different model sizes.

## introduction pre-trained language models (plms) such as bert @xcite are pre-trained by selfsupervised learning on large-scale text corpora to capture the rich semantic knowledge of words @xcite , improving various downstream nlp tasks significantly @xcite @xcite . although these plms have stored much internal knowledge @xcite , they can hardly understand external background knowledge from the world such as factual and linguistic knowledge @xcite @xcite . in the literature, most approaches of knowledge injection can be divided into two categories, including relational knowledge and linguistic knowledge. to overcome the above problems, we release a series of chinese keplms named ckbert (chinese knowledge-enhanced bert), with heterogeneous knowledge sources injected. we particularly focus on chinese as it is one of the most widely spoken languages other than english. the ckbert models are pre-trained by two well-designed pretraining tasks as follows: based on the above heterogeneous knowledge pre-training tasks, we produce various sizes of ck-bert models to meet the inference time and accuracy requirements of different real-world scenarios @xcite , including base (110m), large (345m) and huge (1.3b). the models are pre-trained using our inhouse implemented torchaccelerator that effectively transforms pytorch eager execution to graph execution on distributed gpu clusters, boosting the training speed by 40% per sample with our advanced compiler technique based on accelerated linear algebra (xla). in the experiments, we compare ckbert against strong baseline plms and keplms on various chinese general and knowledge-related nlp tasks. the results demonstrate the improvement of ckbert compared to sota models. ## related work we briefly summarize the related work on the following two aspects: plms and keplms. ## model in this section, we elaborate the techniques of the proposed ckbert model. the main architecture we all know that more practice is the only way to truly improve oral pronunciation. ## experiments we present comprehensive evaluation results of ckbert. due to space limitation, the details of data sources, baselines and hyper-parameter settings are shown in appendices a, b, c. 5 https://github.com/pytorch/xla