Less than One-shot: Named Entity Recognition via Extremely Weak Supervision (2023)

Abstract: We study the named entity recognition (NER) problem under the extremely weak supervision (XWS) setting, where only one example entity per type is given in a context-free way. While one can see that XWS is lighter than one-shot in terms of the amount of supervision, we propose a novel method X-NER that can outperform the state-of-the-art one-shot NER methods. We first mine entity spans that are similar to the example entities from an unlabelled training corpus. Instead of utilizing entity span representations from language models, we find it more effective to compare the context distributions before and after the span is replaced by the entity example. We then leverage the top-ranked spans as pseudo-labels to train an NER tagger. Extensive experiments and analyses on 4 NER datasets show the superior end-to-end NER performance of X-NER, outperforming the state-of-the-art few-shot methods with 1-shot supervision and ChatGPT annotations significantly. Finally, our X-NER possesses several notable properties, such as inheriting the cross-lingual abilities of the underlying language models.

## introduction named entity recognition (ner) with extremely weak supervision (xws) is the task of performing ner using only one entity example per entity type as the supervision. as shown in to develop effective ner models under the xws setting, the keystone is to find sufficiently large, high-quality pseudo labels from raw texts to train the ner model. a straightforward solution is to adapt few-shot ner methods (with uncertainty quantification) to select confident predictions ## preliminary and definitions named entity recognition is a sub-task of information extraction that seeks to locate and classify named entities in text into pre-defined categories. given a sentence @xmath0, • • • , x n ] where x i is the i-th token in the sentence, and n is the total number of tokens. the objective of the ner task is to identify entities in the sentence. for a specific entity type, each entity is represented by a pair of boundaries (i, j) where x i:@xmath1, • • • , x j ] signifies that the tokens from i-th to j-th, inclusive, form a named entity of that type. pretrained language models such as bert @xcite , gpt @xcite , and others @xcite learn to predict the distribution of a token x i based on its context x through their training objective, represented as f lm (x i |x). the context, contingent on the model's architecture, could encompass tokens around x i in bert-like masked language modeling (mlm) where x i is replaced by a special mask token during training, or just the preceding tokens in casual language modeling (clm) models such as gpt. the distribution f (x i |x) reflects the probable occurrence of tokens in the vocabulary at position i given the context and is an effective dense representation of the context information. we will use this distribution to tell the difference between two contexts x and x ′ . only one entity example per type as supervision. ## conclusions and future work in summary, our research offers a groundbreaking extremely weak supervision method for named entity recognition, which significantly enhances 1-shot ner performance. we generate high-quality pseudo annotations without requiring fine-tuning by capitalizing on pre-trained language models' inherent text comprehension. additionally, we address challenges related to span representation and context integration with a novel span test, leading to superior results on ner tasks compared to many established methods. future work will focus on improving the scalability of this approach, enhancing its efficacy across a broader array of ner scenarios, and refining the span test for more nuanced linguistic contexts.