Unsupervised Boundary-Aware Language Model Pretraining for C hinese Sequence Labeling (2022)

Abstract: Boundary information is critical for various Chinese language processing tasks, such as word segmentation, part-of-speech tagging, and named entity recognition. Previous studies usually resorted to the use of a high-quality external lexicon, where lexicon items can offer explicit boundary information. However, to ensure the quality of the lexicon, great human effort is always necessary, which has been generally ignored. In this work, we suggest unsupervised statistical boundary information instead, and propose an architecture to encode the information directly into pre-trained language models, resulting in Boundary-Aware BERT (BABERT). We apply BABERT for feature induction of Chinese sequence labeling tasks. Experimental results on ten benchmarks of Chinese sequence labeling demonstrate that BABERT can provide consistent improvements on all datasets. In addition, our method can complement previous supervised lexicon exploration, where further improvements can be achieved when integrated with external lexicon information.

## introduction the representative sequence labeling tasks for the chinese language, such as word segmentation, partof-speech (pos) tagging and named entity recognition (ner) @xcite , have been inclined to be performed at the characterlevel in an end-to-end manner @xcite . the paradigm, naturally, is standard to chinese word segmentation (cws), while for chinese pos tagging and ner, it can better help reduce the error propagation @xcite @xcite compared with word-based counterparts by straightforward modeling. recently, all the above tasks have reached stateof-the-art performances with the help of bertlike pre-trained language models @xcite . the bert variants, such as bert-wwm @xcite , ernie @xcite , zen @xcite , nezha @xcite , etc., further improve the vanilla bert by either using external knowledge or larger-scale training corpus. the improvements can also benefit character-level chinese sequence labeling tasks. notably, since the output tags of all these character-level chinese sequence labeling tasks involve identifying chinese words or entities @xcite , prior boundary knowledge could be highly helpful for them. a number of studies propose the integration of an external lexicon to enhance their baseline models by feature representation learning @xcite @xcite . moreover, some works suggest injecting similar resources into the pre-trained bert weights. bert-wwm @xcite and ernie @xcite are the representatives, which leverage an external lexicon for masked word prediction in chinese bert. the lexicon-based methods have indeed achieved great success for boundary integration. however, there are two major drawbacks. first, the lexicon resources are always constructed manually @xcite @xcite , which is expensive and time-consuming. the quality of the lexicon is critical to our tasks. second, different tasks as well as different domains require different lexicons @xcite . a well-studied lexicon for word segmentation might be inappropriate for ner, and a lexicon for news ner might also be problematic for finance ner. the two drawbacks can be due to the supervised characteristic of these lexicon-based enhancements. thus, it is more desirable to offer boundary information in an unsupervised manner. in this paper, we propose an unsupervised boundary-aware bert (babert), which is achieved by fully exploring the potential of statisti-cal features mined from a large-scale raw corpus. we extract a set of n-grams (a predefined fixed n) no matter they are valid words or entities, and then calculate their corresponding unsupervised statistical features, which are mostly related to boundary information. we inject the boundary information into the internal layer of a pre-trained bert, so that our final babert model can approximate the boundary knowledge softly by using inside representations. the babert model has no difference from the original bert, so that we can use it in the same way as the standard bert exploration. we conduct experiments on three chinese sequence labeling tasks to demonstrate the effectiveness of our proposed method. experimental results show that our approach can significantly outperform other chinese pre-trained language models. in addition, compared with supervised lexicon-based methods, babert obtains competitive results on all tasks and achieves further improvements when integrated with external lexicon knowledge. we also conduct extensive analyses to understand our method comprehensively. the pre-trained model and code are publicly available at http://github.com/modelscope/ adaseq/examples/babert . our contributions in this paper include the following: 1) we design a method to encode unsupervised statistical boundary information into boundary-aware representation, 2) propose a new pre-trained language model called babert as a boundary-aware extension for bert, 3) verify babert on ten benchmark datasets of three chinese sequence labeling tasks. ## related work in the past decades, machine learning has achieved good performance on sequence labeling tasks with statistical information @xcite @xcite . recently, neural models have led to state-of-the-art results for chinese sequence labeling @xcite @xcite . in addition, the presence of language representation models such as bert @xcite has led to impressive improvements. in particular, many variants of bert are devoted to integrating boundary information into bert to improve chinese sequence labeling @xcite @xcite .