Lexicon Enhanced C hinese Sequence Labeling Using BERT Adapter (2021)

Abstract: Lexicon information and pre-trained models, such as BERT, have been combined to explore Chinese sequence labeling tasks due to their respective strengths. However, existing methods solely fuse lexicon features via a shallow and random initialized sequence layer and do not integrate them into the bottom layers of BERT. In this paper, we propose Lexicon Enhanced BERT (LEBERT) for Chinese sequence labeling, which integrates external lexicon knowledge into BERT layers directly by a Lexicon Adapter layer. Compared with existing methods, our model facilitates deep lexicon knowledge fusion at the lower layers of BERT. Experiments on ten Chinese datasets of three tasks including Named Entity Recognition, Word Segmentation, and Part-of-Speech Tagging, show that LEBERT achieves state-of-the-art results.

## introduction sequence labeling is a classic task in natural language processing (nlp), which is to assign a label to each unit in a sequence @xcite . many important language processing tasks can be converted into this problem, such as partof-speech (pos) tagging, named entity recognition (ner) and text chunking. the current state-ofthe-art results for sequence labelling have been achieved by neural network approaches @xcite @xcite . chinese sequence labelling is more challenging due to the lack of explicit word boundaries in chinese sentences. one way of performing chinese sequence labelling is to perform chinese word segmentation (cws) first, before applying word sequence labelling @xcite . however, it can suffer from the segmentation errors propagated from the cws system !"#@xmath0%&'"()" ## related work our work is related to existing neural methods using lexicon features and pre-trained models to improve chinese sequence labelling. lexicon-based. lexicon-based models aim to enhance character-based models with lexicon information. @xcite introduced a lat-tice lstm to encode both characters and words for chinese ner. it is further improved by following efforts in terms of training efficiency @xcite , model degradation @xcite , graph structure @xcite , and removing the dependency of the lexicon @xcite . lexicon information has also been shown helpful for chinese word segmentation (cws) and part-ofspeech (pos) tagging. yang et al. (2019) pre-trained model-based. transformer-based pretrained models, such as bert @xcite , have shown excellent performance for chinese sequence labelling. @xcite simply added a softmax on bert, achieving state-of-the-art performance on cws. @xcite ; @xcite showed that models using the character features from bert outperforms the static embedding-based approaches by a large margin for chinese ner and chinese pos tagging. hybrid model. recent work tries to integrate the lexicon and pre-trained models by utilizing their respective strengths. @xcite concatenated separate features, bert representation and lexicon information, and input them into a shallow fusion layer (lstm) for chinese ner. @xcite proposed a shallow flat-lattice transformer to handle the character-word graph, in which the fusion is still at model-level. similarly, character n-gram features and bert vectors are concatenated for joint training cws and pos tagging @xcite . our method is in line with the above approaches trying to combine lexicon information and bert. the difference is that we integrate lexicon into the bottom level, allowing in-depth knowledge interaction within bert. there is also work employing lexicon to guide pre-training. ernie @xcite exploited entity-level and word-level masking to integrate knowledge into bert in an implicit way. @xcite proposed entity enhanced bert, further pre-training bert using a domainspecific corpus and entity set with a carefully designed character-entity transformer. zen @xcite enhanced chinese bert with a multilayered n-gram encoder but is limited by the small size of the n-gram vocabulary. compared to the above pre-training methods, our model integrates lexicon information into bert using an adapter, which is more efficient and requires no raw texts or entity set. bert adapter. bert adapter @xcite aims to learn task-specific parameters for the downstream tasks. specifically, they add adapters between layers of a pre-trained model and tune only the parameters in the added adapters for a certain task. @xcite injected taskspecific adapter layers into pre-trained models for neural machine translation. mad-x @xcite is an adapter-based framework that enables high portability and parameter-efficient transfer to arbitrary tasks. @xcite proposed k-adapter to infuse knowledge into pre-trained models with further pre-training. similar to them, we use a lexicon adapter to integrate lexicon information into bert. the main difference is that our goal is to better fuse lexicon and bert at the bottom-level rather than efficient training. to achieve it, we fine-tune the original parameters of bert instead of fixing them, since directly injecting lexicon features into bert will affect the performance due to the difference between that two information. figure 3: character-words pair sequence of a truncated chinese sentence "美国人民 (american people)". there are four potential words, namely "美国 (america)", "美国人 (american)", "国人 (compatriot)", "人 民 (people)". "<@xmath1. ## method the main architecture of the proposed lexicon enhanced bert is shown in figure 2 . compared to bert, lebert has two main differences. first, lebert takes both character and lexicon features as the input given that the chinese sentence is converted to a character-words pair sequence. second, a lexicon adapter is attached between transformer layers, allowing lexicon knowledge integrated into bert effectively. in this section we describe: 1) char-words pair sequence (section 3.1), which incorporates words into a character sequence naturally; 2) lexicon adapter (section 3.2), by injecting external lexicon features into bert; 3) lexicon enhanced bert (section 3.3), by applying the lexicon adapter to bert. ## experiments we carry out an extensive set of experiments to investigate the effectiveness of lebert. in addition, we aim to empirically compare model-level and bert-level fusion in the same setting. standard f1-score (f1) are used as evaluation metrics.