C oarse-to- F ine P re-training for N amed E ntity R ecognition (2020)

Abstract: More recently, Named Entity Recognition hasachieved great advances aided by pre-trainingapproaches such as BERT. However, currentpre-training techniques focus on building lan-guage modeling objectives to learn a gen-eral representation, ignoring the named entity-related knowledge. To this end, we proposea NER-specific pre-training framework to in-ject coarse-to-fine automatically mined entityknowledge into pre-trained models. Specifi-cally, we first warm-up the model via an en-tity span identification task by training it withWikipedia anchors, which can be deemed asgeneral-typed entities. Then we leverage thegazetteer-based distant supervision strategy totrain the model extract coarse-grained typedentities. Finally, we devise a self-supervisedauxiliary task to mine the fine-grained namedentity knowledge via clustering.Empiricalstudies on three public NER datasets demon-strate that our framework achieves significantimprovements against several pre-trained base-lines, establishing the new state-of-the-art per-formance on three benchmarks. Besides, weshow that our framework gains promising re-sults without using human-labeled trainingdata, demonstrating its effectiveness in label-few and low-resource scenarios.

## introduction named entity recognition (ner) is the task of discovering information entities and identifying their corresponding categories, such as mentions of people, organizations, locations, temporal and numeric expressions @xcite . it is an essential component in many applications including machine translation @xcite , relation extraction @xcite , entity linking @xcite , and so on. recently, ner has seen remarkable advances with the help of pre-trained representation models, such as bert @xcite and xl-net @xcite . providing contextual representation, these pre-trained models could be easily applied to ner applications as an encoder by just fine-tuning it. despite refreshing the state-of-theart performance of ner, the current pre-training techniques are not directly optimized for ner. typically, these models build unsupervised training objectives to capture dependency between words and learn a general language representation @xcite , while rarely considering incorporating named entity information which can provide rich knowledge for ner. due to little knowledge connection between ner and general language modeling, how to adapt public pre-trained models to be ner-specific remains an open problem. to this end, injecting named entity knowledge during pre-training is a possible solution. however, this process of knowledge acquisition may be inefficient and expensive. in fact, there are extensive weakly labeled annotations that naturally exist on the web yet to be explored for ner model pre-training, which are relatively easier to obtain compared with labeled data @xcite . one can collect them from online resources, such as the wikipedia anchors and gazetteers (named entity dictionaries). although automatically derived corpora usually contain massive noisy data, it still contains some extend the valuable semantic information required for ner @xcite . in this paper, we propose a coarse-to-fine entity knowledge enhanced (cofee) pre-training framework for ner task, aiming to gather and utilize knowledge related to named entities. in particular, we first extract anchors from wikipedia and use them as training corpora for entity span identification. while anchors have no entity type information, the model could get general-typed entity knowledge from them and learn to distinguish entity words and non-entity words. in the second phase, we use gazetteers and anchors to generate weakly labeled data for specific entity types and use it to train the model for extracting entities with coarse-grained type. furthermore, another observation is that entities with the same coarse-grained type may belong to different fine-grained types. according to the cluster hypothesis @xcite , the features of entities with the same latent fine-grained label will cluster together in the semantic space. intuitively, mining these latent cluster structures provides auxiliary information about the coarse-grained entity type, which could be beneficial to improve the ner performance. based on such motivation, we finally devise a self-supervised method to exploit fine-grained type knowledge and tap the potential of weakly labeled data, which effectively train the ner model with clusteringgenerated pseudo labels. we conduct experiments on three realistic ner benchmarks in this paper. experimental results show that the proposed cofee pre-training framework significantly outperforms other competitive baselines, often by large margins. we also demonstrate that cofee pre-training can work well in more challenging, label-free and low-resource scenarios. further ablation studies show the impact of each pre-training task in achieving these strong performance. to the best of our knowledge, this is the first work that has tackled ner-specific representation during pre-training. for supervised ner task, some researchers utilize lattice structure to incorporate the lexical information into character-based ner and avoid the segmentation error propagation of word @xcite @xcite @xcite . additionally, gazetteers have long been regarded as a piece useful knowledge for ner, previous methods commonly incorporated gazetteers by either using them as handcraft features @xcite or auxiliary structural information @xcite . for weakly supervised ner, a typical line of methods centres around transfer learning to extract source knowledge for target, such as crossdomain @xcite @xcite or cross-lingual @xcite @xcite . there are also a lot of weak labels lying on the web or gazetteers, which have not been explored. consequently, a number of works focus on distantly supervised methods, using anchors or gazetteers to generate data by distant supervision @xcite @xcite . task specific pre-training. unsupervised language model pre-training and task-specific finetuning achieve sota results on many nlp tasks, including ner @xcite @xcite . recently, with the help of automatically minded knowledge lying in the web, researchers devoted them to the pre-training models for specific tasks, including word sense disambiguation @xcite , word-in-context tasks @xcite , entity-linking and relation classification @xcite , sentiment classification @xcite . ## background in this section, we give a brief introduction to mrc-ner @xcite , which achieves satisfying performance in ner and thus is chosen as the foundation of our work. given an input paragraph @xmath0, x 2 , • • • , x n } where x i denotes the i-th character, ner aims at discovering each entity x start,end in x and identify its corresponding type @xmath1, where y is the set of predefined tags(e.g., per, loc). x start,@xmath2, x start+1 , • • • , x end-1 , x end } is a substring of x satisfying @xmath3. specifically, mrc-ner formulates ner as a machine reading comprehension (mrc) problem. each entity type y is characterized by a natural language query q @xmath4, q y 2 , ..., q y m }, and entities are extracted by answering these queries given the contexts. for example, the task of assigning the per label to " @xcite was born into slavery on the farm" is formalized as answering the question "find person including fictional". this strategy naturally introduces the natural language query which encodes significant prior knowledge about the entity question: find entities paragraph: keflezighi and his family were refugees from asmara, eritrea. ## methodology in this section, we introduce the overall framework of our coarse-to-fine pre-training. ## experiments we evaluate the cofee framework under two settings: (i) supervised setting (ii) weakly supervised setting. in the supervised setting, the pre-trained model is fine-tuned on human-labeled datasets while in the weakly-supervised setting, the model pre-trained with cofee is directly applied to perform ner without fine-tuning. next, we describe these experiments in detail. ## conclusion we investigated coarse-to-fine entity knowledge enhanced pre-training for named entity recognition, which integrates three kinds of entity knowledge with different granularity levels. though conceptually simple, our framework is highly effective and easy to implement. on three popular ner benchmarks, we found consistent improvements over both state-of-the-art supervised and weakly-supervised methods. further analysis verifies the necessity of utilizing ner knowledge for pre-training models.