N u NER : Entity Recognition Encoder Pre-training via LLM -Annotated Data (2024)

Abstract: Large Language Models (LLMs) have shown impressive abilities in data annotation, opening the way for new approaches to solve classic NLP problems. In this paper, we show how to use LLMs to create NuNER, a compact language representation model specialized in the Named Entity Recognition (NER) task. NuNER can be fine-tuned to solve downstream NER problems in a data-efficient way, outperforming similar-sized foundation models in the few-shot regime and competing with much larger LLMs. We find that the size and entity-type diversity of the pre-training dataset are key to achieving good performance. We view NuNER as a member of the broader family of task-specific foundation models, recently unlocked by LLMs. NuNER and NuNER’s dataset are open-sourced with MIT License.

## introduction named entity recognition (ner) -the generic task of extracting and classifying entities from text -is a core component of natural language processing, present in a variety of applications such as medical coding, financial news analysis, or legal documents parsing @xcite . such application typically involves solving a particular ner problem, for a particular set of entity types, thus requiring the creation of a custom model. for the last five years, the standard procedure for creating such custom model has consisted of using a transformer encoder @xcite pre-trained in a self-supervised way to satisfy a masked language modeling (mlm) objective, such as models of the bert family @xcite . this foundation model is then fine-tuned in a supervised way on human-annotated data, either using a simple token-classification, or a more advanced strategy @xcite . 1 nuner and nunerzero are available on huggingface figure 1 : nuner creation procedure. roberta is further pre-trained on a subset of c4 automatically annotated by gpt-3.5. the resulting model can be fine-tuned on various downstream ner problems. in the last few years, we have witnessed the emergence of generative large language models (llms) such as gpt-3 @xcite and, more recently, gpt-4 (openai, 2023), which typically have between 100 times and 10,000 times more parameters than bert. these massive autoregressive transformer models, trained via a nextword prediction objective, are language generators which can be prompted to perform a variety of tasks. for example, these models can be directly used through a well crafted prompt to tackle a particular ner problem with satisfying performance @xcite . the main issue with this approach is the high inference cost due to the size of llms. a simple solution to this inference-cost issue is to use a correctly-prompted llm to annotate data for the particular ner problem and then to train a smaller model on this data. llms have been shown to outperform crowd workers @xcite on some tasks for a fraction of the cost but this strategy has issues as well. first, crafting a good prompt -like delegating a task to someone else -is not easy; it requires multiple back-and-forth while validating the performance using human-annotated data. second, llms are not perfect annotators either @xcite . finally, the best llms are mainly hosted on private companies' opening the door to potential confidentiality and privacy leaks. we propose an alternative approach that leverages llms to reduce the amount of human annotations needed to create custom models. instead of using an llm to directly annotate a particular single-domain dataset for a particular ner problem, our idea is to use it to annotate a multi-domain dataset for variety of ner problems. we then further pre-train a small foundation model, such as bert, on this annotated dataset. the resulting pre-trained model can then be fine-tuned to any downstream ner problem, just like any other foundation model, as depicted in figure 1 . because the resulting pre-trained model is specialized to a generic task but is still meant to be fine-tuned to a particular problem, we refer to such a model as a task-specific foundation model. note that compact domain-specific foundation models like scibert @xcite or biobert @xcite are common, but task-specific foundation models of this kind are rare, mostly due to the lack of suitable datasets. generative llms are the key to building such models. in this paper, we apply the above idea to create nuner, a task-specific foundation model for the generic task of ner. in section 3, we describe both the dataset creation and the training procedures. in a nutshell, we use gpt-3.5 to annotate a subset of c4 @xcite , resulting in a 24.4m words dataset containing 4.38m annotations from 200k unique concepts. we then pre-train a base roberta on this dataset via a contrastive-learning approach @xcite to obtain nuner. in section 4, we analyze the transfer learning performance of nuner in an extended few-shot regime. we find that nuner largely outperforms both its base model and the same base model further pre-trained on ner-bert data @xcite , which is the largest and most diverse ner dataset we could find. these results demonstrate the validity of our approach. in section 5, we investigate the factors influencing nuner's abilities. we find that the diversity of the annotations and the size of the pre-training dataset are the most influential factors. surprisingly, the diversity of the text does not appear to be as influential. in section 6, for informational purposes, we compare fine-tuning nuner with using gpt-3.5 and gpt-4 via in-context learning. we find that nuner beats gpt-3.5 and competes with gpt-4 when more than a dozen entities of each type is seen during training. we also compare, via finetuning, nuner with universalner @xcite , a recent llm specialized in the ner task. we find that they exhibit similar transfer learning performance when fine-tuned, despite nuner being 56 times smaller. the contributions of our paper are as follows. ## related work early attempts to create ner-specific foundation models for low-resource ner problems focused on leveraging wikipedia anchors @xcite . in particular, @xcite (ner-bert) combined these anchors and dbpedia ontology to create ner dataset containing 3.64m entities from 315 entity types. bert was then pre-trained on this dataset, leading to improved few-shot performance. we demonstrate in section 4.1 that our llm-annotation procedure outperforms such an approach. subsequent work has focused on pre-training generative llms on a large number of existing human-annotated ner datasets to achieve strong zero-shot capabilities @xcite . these works differ from ours in terms of type of data used, model employed, and objective (zero-shot vs. few-shot). recently, @xcite proposed universal-ner, an llm with 7b and 13b parameters, also pre-trained on data annotated by gpt-3.5. our work primarily differ in the model and training procedure: we found a way to train an encoder model with only 125m parameters on such data, making it substantially cheaper to use. in section 6, we show that nuner and universalner have similar transfer-learning abilities when provided with more than a few training examples per entity types. even more recently, @xcite proposed gliner, which uses universalner's data to pre-train a small encoder model. our work was done concurrently and independently from this work. the principal distinction is in the architecture used: gliner merges the text and concept encoders, whereas in our approach, they remain independent (see figure 5 ). while gliner's integrated approach likely enhances performance, our decision to keep the encoders separate enables nuner to function as a concept-agnostic language representation model. this gives the possibility to pre-compute text embeddings for a variety of applications such as information retrieval. this also makes nuner a viable drop-in substitute for bert or roberta in standard ner methodologies. ## nuner the creation of nuner is a two-step process: dataset creation and model training. ## transfer learning performance nuner is designed to be fine-tuned to downstream ner problems in a data-efficient way. we are mostly interested in the transfer learning performance of nuner in an extended few-shot regime, typically from 1 to 100 annotations per entity types. we focus on this range because we believe it represents the level of annotation effort practitioners are willing to undertake without resorting to external annotation solutions, such as crowdsourcing. ## ablation studies we aim to understand which factors in the pretraining procedure most significantly affect the performance of nuner. to this end, we investigate the impact of text diversity, concept diversity, pretraining dataset size, and model size appendix a.1, using the same benchmark as in section 4.1. we also study nuner's multilingual performance in appendix a.2. ## comparison with llms although the comparisons are indicative, we finally provide for information a comparison of nuner with modern generative llms. we choose gpt-3.5 (gpt-3.5-turbo-16k-0613) for its popularity and longer context, gpt-4 (gpt-4-0613) for its high performance @xcite , and universalner (universalner-7b-type) for its specialization in the ner task. universalner has 56 times more parameters than nuner, and gpt-3.5 and gpt-4 are likely to have around 1,000 and 10,000 times more parameters than nuner, respectively. gpt-4 and gpt-3.5 are used via in-context learning using spacy's ner v3 prompt. this advanced prompt template allows to create a prompt for a particular ner problem by providing it the set of entity types and some training examples. universalner @xcite is trained to be used for zero-shot inferences, conducted through a conversation in which one sequentially prompt the model to identify each entity type. to adapt this model for a few-shot setting, we need to fine-tune it. we use the original training settings of uni-versalner but modify them to enhance few-shot learning performance, as detailed in appendix a.4. for nuner, we simply attach a two-layer fullyconnected network regularized via dropout, and fine-tune the entire network for 30 epochs, costing less than 5@xmath0, while nuner achieves similar performance while costing less than 0.0001$ per example, allowing for industrial use. the fact that nuner surpasses gpt-3.5 and possibly also gpt-4 is likely attributable to the limitations of in-context learning. the situation might differ with proper, albeit challenging, fewshot fine-tuning for these llms. more surprising to us is that nuner achieves performance comparable to universalner despite being 56 times smaller and trained on similar data. this could be due to an inherent advantage of encoders over generative models for this task. alternatively, it might be related to nuner's pre-training procedure, which encourages human concepts to emerge in the last layers of the network, being easily accessible during few-shot training. further experiments would be needed to explore this aspect. ## conclusion modern large language models are opening new possibilities for addressing traditional nlp tasks. we have introduced a procedure that uses these llms to create a compact, yet data-efficient, nerspecific foundation model. we foresee an increasing trend in the development of such task-specific foundation models, which will facilitate the creation of high-quality custom nlp models without requiring intensive human or computational resources.