D y L ex: Incorporating Dynamic Lexicons into BERT for Sequence Labeling (2021)

Abstract: Incorporating lexical knowledge into deep learning models has been proved to be very effective for sequence labeling tasks. However, previous works commonly have difficulty dealing with large-scale dynamic lexicons which often cause excessive matching noise and problems of frequent updates. In this paper, we propose DyLex, a plug-in lexicon incorporation approach for BERT based sequence labeling tasks. Instead of leveraging embeddings of words in the lexicon as in conventional methods, we adopt word-agnostic tag embeddings to avoid re-training the representation while updating the lexicon. Moreover, we employ an effective supervised lexical knowledge denoising method to smooth out matching noise. Finally, we introduce a col-wise attention based knowledge fusion mechanism to guarantee the pluggability of the proposed framework. Experiments on ten datasets of three tasks show that the proposed framework achieves new SOTA, even with very large scale lexicons.

## introduction sequence labeling is the task of assigning categorical labels to a text sequence. many conventional nlp tasks, such as named entity recognition (ner), chinese word segmentation (cws), and slot-filling based natural language understanding (nlu), can be formalized as the sequence labeling problem. the deep learning methods, especially the recently proposed bert and its variants, have achieved great success in such sequence labeling tasks. however, the bert-based methods are generally built based on word-piece or character embeddings. the word information (e.g., word boundary or type) is not fully exploited, which makes it difficult to accurately determine the entity boundary or correctly predict entity type. figure 1 : iron man can be a name of a smart device or a movie and the system would be unable to react properly upon "please play iron man" from a user. another case as "play just a little while longer now on iron man" requires the system to classify "play" between music and movie domains, and whether "now" should be combined with "just a little while longer" as a whole. as shown in figure 1 , it is infeasible to understand user's utterance correctly without using deterministic domain knowledge that "iron man" is the alias of a smart speaker or "just a little while longer" is a famous song. in commercial systems, the lexicon is widely used as an effective way to store various domain knowledge. in practice, the size of a lexicon can range from ten to a few million, and we usually need to update the contents of lexicons frequently, which dramatically increases the difficulty of incorporating lexicons into deep models. in this work, we will study how to effectively incorporate large-scale dynamic lexicons into bert-based sequence labeling models. recent works on incorporating lexicon knowledge @xcite @xcite ) can be summarized as follows. first, they match an input sentence with several lexicons to obtain all matched items. second, leveraging the matched item information through modifying the structure of the transformer layer or the feature representation layer. however, 1) current methods normally learn additional embeddings of the words in the lexicons, which bring us a challenge -if the lexicons get updated, the model must be re-trained; 2) they only use the words in the lexicon but ignore the category of words, which is important for many tasks. in this paper, we propose a general framework dylex for incorporating frequently updated lexicons into sequence labeling models. the matching results of the input are reconstructed as a wordagnostic tag sequence. then we design a supervised knowledge denoising module to smooth out noisy matches, and the remaining matches are further used as additional feature input for knowledge fusion. this step is based on a col-wise attention to seamlessly fuse word-piece embeddings of input sentence and the lexicon features. moreover, since we do not explicitly learn embeddings of the words in lexicons, there is no need to retrain the entire model when updating the lexicons. we conduct extensive experiments with the cws, ner, and nlu tasks on various datasets. the results show that our model consistently outperforms the strong baselines and achieves new state-of-the-art results. we summarize the contribution of this work as follows: ## approach in this section, we will present how to incorporate large-scale lexicons into bert. as illustrated in figure 2 , the proposed dylex framework contains two parts, namely the bert-based sequence tagger and lexicon knowledge extractor. the lexicon knowledge (lexkg) extractor has three submodules: matching, denoising and fusing. ## experiments we conduct experiments on several nlp tasks, including cws (chinese word segmentation), ner (named entity recognition), and nlu (natural language understanding). the experimental hyperparameter settings are listed in appendix f. ## related work with the advance of deep learning, sequence labelling tasks, such as segmentation and ner, have achieved excellent performance. more and more methods tend to be character-based @xcite @xcite , especially in languages, such as chinese, japanese, korean, etc., that require word segmentation. these languages do not have a natural segmentation delimiter as white space in latin languages. character-based input in these languages can avoid accumulation of word segmentation errors, then get better performances @xcite @xcite . however, the downside of the purely character-based method is that the word information is not fully exploited. to make full use of word information, incorporating a lexicon is an effective method. existing works on incorporating lexicon can be categorized as feature based, lattice based and graph based methods according to implementation complexity. feature based feature based method is a simpler way. some works directly use lexical information with simple matching features and the others use auxiliary tasks to leverage the lexical information. @xcite builds the template first and uses the template matching lexicon to build features, which help word segmentation tasks. @xcite uses a simple lexicon matching location information as features. @xcite and @xcite adopt word-level language modeling objective and multi-task to use word information implicitly. @xcite transfer crossdomain and cross-lingual knowledge via multi-task learning. lattice based lattice based method is to use lattice structure. @xcite proposes lattice-lstm for incorporating word lexicons into the character-based ner model. rather than heuristically choosing a word for the character when matching multiple words in the lexicon, they also introduce an elaborate modification to the sequence modeling layer of the lstm-crf model @xcite . considering that the short path in the lattice structure will cause the word-based structure to degenerate into a characterbased structure, @xcite propose a novel word character lstm(wc-lstm) model to add word information via four strategies. since the lattice structure is complex and dynamic, most existing lattice-based models cannot fully utilize gpus' parallel computation and usually have a low inference-speed. @xcite propose a transformer-based model for chinese ner, which converts the lattice structure into a flat structure. graph based graph based method uses a directed graph structure to fuse lexiconal information. @xcite uses a gnn-based method to explore multiple graph-based interactions among characters, potential words, and the whole-sentence semantics to effectively alleviate the word ambiguity. @xcite employ a collaborative graph network to assign both self-matched and the nearest contextual lexical terms. to automatically learn how to incorporate multiple gazetteers into a @xcite propose a novel approach based on graph neural networks with a multidigraph structure. the structure captures the information the gazetteers offer. ## conclusion and future work in this paper, we propose dylex, a framework incorporating dynamic lexicon to improve bert-like models' performance in sequence labeling tasks. to alleviate the problems caused by large-scale dynamic lexicons, we introduce word-agnostic tag embeddings and a knowledge denoising module. as a result, our framework outperforms the stateof-the-art works on many sequence labeling tasks. in future, how to extend it to text classification is a challenge, since denoising corpus cannot be automatically constructed at this time.