NER etrieve: Dataset for Next Generation Named Entity Recognition and Retrieval (2023)

Abstract: Recognizing entities in texts is a central need in many information-seeking scenarios, and indeed, Named Entity Recognition (NER) is arguably one of the most successful examples of a widely adopted NLP task and corresponding NLP technology. Recent advances in large language models (LLMs) appear to provide effective solutions (also) for NER tasks that were traditionally handled with dedicated models, often matching or surpassing the abilities of the dedicated models. Should NER be considered a solved problem? We argue to the contrary: the capabilities provided by LLMs are not the end of NER research, but rather an exciting beginning. They allow taking NER to the next level, tackling increasingly more useful, and increasingly more challenging, variants. We present three variants of the NER task, together with a dataset to support them. The first is a move towards more fine-grained—and intersectional—entity types. The second is a move towards zero-shot recognition and extraction of these fine-grained types based on entity-type labels. The third, and most challenging, is the move from the recognition setup to a novel retrieval setup, where the query is a zero-shot entity type, and the expected result is all the sentences from a large, pre-indexed corpus that contain entities of these types, and their corresponding spans. We show that all of these are far from being solved. We provide a large, silver-annotated corpus of 4 million paragraphs covering 500 entity types, to facilitate research towards all of these three goals.

## introduction identification and extraction of typed entities in text (named entity recognition, "ner") is a central task in natural language understanding, which is both useful in and of itself and as a central component in other language understanding tasks. indeed, ## next-generation ner we now elaborate a sequence of increasingly more challenging criteria for an ideal ner engine, which we believe are now within reach, yet far from being achieved at present and require further research. while each intermediate ability is useful on its own, when combined they culminate in what we consider to be the holy grail: transforming ner from named entity recognition to named entity retrieval. ## performance of current models the neretrieve dataset allow us to assess the performance of current models on the challenging conditions we describe in section 3. models for the supervised case, we train (finetune) spacy foot_5 ner models over a pre-trained deberta-v3-large mlm. for the zero-shot scenarios, we prompt gpt-3.5-turbo and gpt4. ## conclusion we argued that ner remains a relevant and challenging task even in the age of llms: rather than solving ner, the introduction of llms allows us to take the ner task to the next level. we highlight three challenging directions in which the task can evolve, culminating in a move from named entity recognition to named entity retrieval, and created a unique dataset that allows future explorations of these directions.