GEMNET : Effective Gated Gazetteer Representations for Recognizing Complex Entities in Low-context Input (2021)

Abstract: Named Entity Recognition (NER) remains difficult in real-world settings; current challenges include short texts (low context), emerging entities, and complex entities (e.g. movie names). Gazetteer features can help, but results have been mixed due to challenges with adding extra features, and a lack of realistic evaluation data. It has been shown that including gazetteer features can cause models to overuse or underuse them, leading to poor generalization. We propose GEMNET, a novel approach for gazetteer knowledge integration, including (1) a flexible Contextual Gazetteer Representation (CGR) encoder that can be fused with any word-level model; and (2) a Mixture-of- Experts gating network that overcomes the feature overuse issue by learning to conditionally combine the context and gazetteer features, instead of assigning them fixed weights. To comprehensively evaluate our approaches, we create 3 large NER datasets (24M tokens) reflecting current challenges. In an uncased setting, our methods show large gains (up to +49% F1) in recognizing difficult entities compared to existing baselines. On standard benchmarks, we achieve a new uncased SOTA on CoNLL03 and WNUT17.

## introduction identifying entities is a core nlp task. named entity recognition (ner) is the task of finding entities and recognizing their type (e.g., person or location). mention detection (md) is a simpler task of identifying entity spans, without the types. advances in neural ner have produced high scores on benchmark datasets like conll03 and ontonotes @xcite . however, a number of challenges remain. as noted by @xcite , these scores are driven by the use of well-formed news text, the presence of "easy" entities, and memorization effects due to entity overlap between train/test sets; these models perform significantly worse on unseen entities or noisy text. ## datasets we experiment using three standard benchmarks: conll03, ontonotes, and wnut17. however, these corpora do not capture the issues from ner taxonomy: we adopt the wnut 2017 @xcite taxonomy entity types: person (per for short, names of people), location (loc, locations/physical facilities), corporation (corp, corporations and businesses), groups (grp, all other groups), product (prod, consumer products), and creative-work (cw, movie/song/book/etc. titles). our datasets are described below. foot_0 all data are uncased, and we make them publicly available. 2 their statistics, listed in ## the gemnet model we propose gemnet, a generic gazetteer fusion approach that can be integrated with any word-level model, e.g., rnns and transformers. we experiment with both bilstm-crf and bert-crf models which produce (contextual) word representations, and complement these "word experts" with gazetteers. the overall architecture is shown in figure 1 , and the components are detailed below. ## conclusion we focused on integrating gazetteers into ner models. we proposed gemnet, a flexible architecture that includes a contextual gazetteer representation encoder, combined with a novel mixture-of-expert gating network to conditionally utilize this information alongside any word-level model. gemnet supports external gazetteers, allowing the model's knowledge to be updated without retraining. we also developed new datasets to represent the current challenges in ner. experimental results demonstrated that our method can alleviate the feature weight under-training issue, achieving significant improvements on our data and a standard benchmark, wnut17. the datasets we released can serve as benchmarks for evaluating the entity knowledge possessed by models in future work. future work involves investigating integration with different model architectures, partial gazetteer matching, and additional entity features. types to create slotted templates, e.g., "when did [[iphone]] come out" becomes "when @xmath0. the templates are then aggregated by frequency. this process results in 3, 445 unique question templates. while the ner system cannot correctly identify many entities, the most frequent templates are reliable. examples are listed in finally, we generate msq-ner by slotting the templates that have a frequency @xmath1. each template is slotted with the same number of times it appeared in ms-marco in order to maintain the same relative distribution as the original data. this results in 17, 868 questions, e.g., "when did [[xbox 360]] come out", which we use as a test set.