Learning In-context Learning for Named Entity Recognition (2023)

Abstract: Named entity recognition in real-world applications suffers from the diversity of entity types, the emergence of new entity types, and the lack of high-quality annotations. To address the above problems, this paper proposes an in-context learning-based NER approach, which can effectively inject in-context NER ability into PLMs and recognize entities of novel types on-the-fly using only a few demonstrative instances. Specifically, we model PLMs as a meta-function Lambda_instruction, demonstrations, text.M, and a new entity extractor can be implicitly constructed by applying new instruction and demonstrations to PLMs, i.e., (Lambda . M) (instruction, demonstrations) ->F where F will be a new entity extractor F: text -> entities. To inject the above in-context NER ability into PLMs, we propose a meta-function pre-training algorithm, which pre-trains PLMs by comparing the (instruction, demonstration)-initialized extractor with a surrogate golden extractor. Experimental results on 4 few-shot NER datasets show that our method can effectively inject in-context NER ability into PLMs and significantly outperforms the PLMs+fine-tuning counterparts.

## introduction named entity recognition (ner) aims to detect and classify named entities in text, such as people, disease, and movie. traditional ner methods @xcite @xcite have achieved remarkable success ## in-context named entity recognition this section describes how to recognize entities through in-context ner. in in-context learning, the model will read the information of target entity types from both instruction and demonstrations, and then extract entities of target types within the text. in this way, new entity types can be extracted on-the-fly, without the need for model retraining. concretely, this paper formulates in-context ner as a sequence-to-sequence generation process. the input @xmath0; d; t ] includes instruction i, demonstrations d, and text t while the output is a list of extracted entities @xmath1, ..., e n ]. figure 2 shows an example, where an in-context ner model will identify that the target entity types are disease and virus, distill the knowledge about these types from demonstrations(e.g., the context patterns of a disease), and finally recognize "sars-cov-2" as virus and "covid-19" as disease using the above knowledge. the details are described as follows. instruction the instruction is a sequence of target entity types, guiding the model to extract what entity types @xcite . the instruction for target entity types {l 1 , . . . , l n } is @xmath2: l 1 ; . . . ; l n ". for example, in figure 2 the instruction is "target types: disease; virus". demonstrations demonstrations provide the intra-class knowledge of target entity types (e.g., entity semantics and context patterns) and illustrate the form of outputs. as shown in figure 2 , the demonstrations contain the illustrative instances for different target types, and each instance is "text: {text} entities: {extractions}", where {extractions} are entities presented in the {text}. ## meta-function pre-training for in this section, we will explain how to incorporate in-context named entity recognition (ner) capabilities into pre-trained language models (plms). although large-scale plms like gpt-3 have demonstrated the ability to learn in-context, this capability is not always controllable or predictable. additionally, unlike classification and question-answering tasks that align with the pre-training objective of language models (i.e., producing natural text output), ner requires more complex span extraction and type specification. as a result, gutiérrez et al. @xcite show that lms aren't well-suited for incontext ner tasks. in this paper, we propose metafunction pre-training, an algorithm that can inject in-context ner ability into plms in a controllable and predictable way. specifically, we model plms as a metafunction @xcite for ner λ instruction, demonstrations, text .m, and a new entity extractor can be implicitly constructed by applying new instructions and demonstrations to plms, i.e., (λ.m)(instructions, demonstractions) → f where f will be a new entity extractor f:@xmath3. based on the meta-function formulation, we further pre-train plms for in-context ner abilities by: ## experiments this section evaluates our method by conducting experiments on few-shot ner settings. ## conclusion in this paper, we propose an in-context learningbased ner approach and model plms as a metafunction, which can inject in-context ner ability into plms and recognize entities of new types onthe-fly using only a few demonstrative instances. experimental results show that our method is effective for in-context ner. for future work, we will extend our method to different nlp tasks like event extraction and relation extraction.