Entity Enhanced BERT Pre-training for C hinese NER (2020)

Abstract: Character-level BERT pre-trained in Chinese suffers a limitation of lacking lexicon information, which shows effectiveness for Chinese NER. To integrate the lexicon into pre-trained LMs for Chinese NER, we investigate a semi-supervised entity enhanced BERT pre-training method. In particular, we first extract an entity lexicon from the relevant raw text using a new-word discovery method. We then integrate the entity information into BERT using Char-Entity-Transformer, which augments the self-attention using a combination of character and entity representations. In addition, an entity classification task helps inject the entity information into model parameters in pre-training. The pre-trained models are used for NER fine-tuning. Experiments on a news dataset and two datasets annotated by ourselves for NER in long-text show that our method is highly effective and achieves the best results.

## introduction as a fundamental task in information extraction, named entity recognition (ner) is useful for nlp tasks such as relation extraction @xcite , event detection @xcite and machine translation @xcite . we investigate chinese ner @xcite , for which the state-of-the-art methods use a character-based neural encoder augmented with lexicon word information @xcite @xcite . ner has been a challenging task due to the flexibility of named entities. there can be a large number of oov named entities in the open domain, which poses challenges to supervised learning algorithms. in addition, named entities can be ambiguous. take figure 1 for example. the term "老 妇人(the old lady)" literally means "older woman". however, in the context of football news, it means the nickname of a football club juventus f.c.. thus entity lexicons that contain domain knowledge can be useful for the task @xcite . intuitively, such lexicons can be collected automatically from a set of documents that are relevant to the input text. for example, in the news domain, a set of news articles in the same domain and concurrent with the input text can contain highly relevant entities. in the finance domain, the financial report of a company over the years can serve as a context for collecting named entities when conducting ner for a current-year report. in the science domain, relevant articles can mention the same technological terms, which can facilitate recognition of the terms. in the literature domain, a fulllength novel itself can serve as a context for mining entities. there has been work exploiting lexicon knowledge for ner @xcite . however, little has been done integrating entity information into bert, which gives the stateof-the-art for chinese ner. we consider enriching bert @xcite with automatically extracted domain knowledge as mentioned above. in particular, we leverage the strength of newword discovery on large documents by calculating point-wise mutual information to identify entities in the documents. information over such entities is integrated into the bert model by replacing the original self-attention modules @xcite with a char-entity-self-attention mechanism, which captures the contextual similarities of characters and document-specific entities, and explicitly combines character hidden states with entity embeddings in each layer. the extended bert model is then used for both lm pre-training and ner fine-tuning. we investigate the effectiveness of this semisupervised framework on three ner datasets, including a news dataset and two annotated datasets (novels and financial reports) by ourselves, which aims to evaluate ner for long-text. we make comparisons with two groups of state-of-the-art chinese ner methods, including bert and ernie @xcite . for more reasonable comparison, we also complement both bert and ernie with our entity dictionary and further pre-train on the same raw text as ours. results on the three datasets show that our method outperforms these methods and achieves the best results, which demonstrates the effectiveness of the proposed char-entity-transformer structure for integrating entity information in lm pre-training for chinese ner. to our knowledge, we are the first to investigate how to make use of the scale of the input document text for enhancing ner. our code and ner datasets are released at https://github.com/ jiachenwestlake/entity_bert . ## related work chinese ner. previous work has shown that character-based approaches perform better for chinese ner than word-based approaches because of the freedom from chinese word segmentation errors @xcite @xcite . lexicon features have been applied so that the external word-level information enhances ner training @xcite @xcite . however, these methods are supervised models, which cannot deal with a dataset with relatively little labeled data. we address this problem by using a semisupervised method by using a pre-trained lm. pre-trained language models. pre-trained language models have been applied as an integral component in modern nlp systems for effectively improving downstream tasks @xcite @xcite @xcite . recently, there is an increasing interest to augment such contextualized representation with external knowledge @xcite @xcite . these methods focus on augmenting bert by integrating kg embeddings such as transe @xcite . different from the line of work, our model dynamically integrates document-specific entities without using any pre-trained entity embeddings. a more similar method is ernie @xcite , which enhances bert through knowledge integration. in particular, instead of masking individual subword tokens as bert does, ernie is trained by masking full entities. the entity-level masking trick for ernie pre-training can be seen as an implicit way to integrate entity information through error backpropagation. in contrast, our method uses an explicit way to encode the entities to the transformer structure. ## method as shown in figure 2 , the overall architecture of our method can be viewed as a transformer structure with multi-task learning. there are three output components, namely masked lm, entity classification and ner. with only the masked language model component, the model resembles bert without the next sentence prediction task, and the entity classification task is added to enhance pretraining. while only ner outputs are yielded, the model is a sequence labeler for ner. we integrate entity-level information by extending the standard transformer. ## experiments we empirically verify the effectiveness of entity enhanced bert pre-training on different ner datasets. in addition, we also investigate how different components in the model impact the performance of ner with different settings. ## conclusion we investigated an entity enhanced bert pretraining method for chinese ner. results on a news dataset and two long-text ner datasets show that it is highly effective to explicitly integrate the document-specific entities into bert pre-training with a char-entity-transformer structure, and our method outperforms the state-of-the-art methods for chinese ner.