2 INER : Instructive and In-Context Learning on Few-Shot Named Entity Recognition (2023)

Abstract: Prompt-based learning has emerged as a powerful technique in natural language processing (NLP) due to its ability to leverage pre-training knowledge for downstream few-shot tasks. In this paper, we propose 2INER, a novel text-to-text framework for Few-Shot Named Entity Recognition (NER) tasks. Our approach employs instruction finetuning based on InstructionNER to enable the model to effectively comprehend and process task-specific instructions, including both main and auxiliary tasks. We also introduce a new auxiliary task, called Type Extracting, to enhance the model’s understanding of entity types in the overall semantic context of a sentence. To facilitate in-context learning, we concatenate examples to the input, enabling the model to learn from additional contextual information. Experimental results on four datasets demonstrate that our approach outperforms existing Few-Shot NER methods and remains competitive with state-of-the-art standard NER algorithms.

## introduction named entity recognition (ner) has been a fundamental task of natural language processing (nlp) and there are three types of sub-tasks in ner: flat ner @xcite , nested ner @xcite and discontinuous ner @xcite . all three sub-tasks aim to locate named entities, extract the entity spans, and classify each span into pre-defined label categories. in terms of the flat ner which is the main focus of this paper, it can be formulated as a sequence labeling paradigm by assigning labels to each token in the sentence through token-classification models. the dominant methods include combining pre-trained language models(plms) @xcite with label-specific classifier (lc) @xcite . however, the fixed shape of the output lc layer necessitates a consistent label set for both the training and testing data, which poses a challenge for knowledge transfer. therefore, these models need to be trained from scratch to adapt to a new domain with a different label set, highlighting the requirement for a large amount of data for these methods. due to the high cost of sequence labeling annotation in real-world scenarios, labeled data for ner is often limited. as a result, few-shot ner has gained significant attention due to its practical applications. meanwhile, applying prompt-base learning @xcite on plms is an effective way to solve few-shot problems @xcite . plms can learn a lot of knowledge regarding human languages by training on a large amount of self-supervised corpus. in order to explore the potential of plms, prompt-based learning reformulate the downstream tasks to text-totext framework with additional prompt indicating task descriptions (e.g. instruction fine-tuning @xcite @xcite ). through this approach, the model can effectively leverage the knowledge present in plms to enhance downstream skills without the need for additional large amounts of downstream data. this enables the model to achieve remarkable performance in few-shot settings. recently, many prompt-based ner methods have emerged to address the limitations of traditional few-shot ner approaches. templatener @xcite treats original sentence and predicted template filled by entity spans as source and target sequence, respectively, but all candidate spans must be enumerated during inference, leading to a high computational cost. bartner @xcite proposed a pointer mechanism to unified all ner sub-tasks into one sequenceto-sequence (seq2seq) framework. bartner utilizes the raw sentence as input and outputs pointer index and tag index which represent the location of the span and the corresponding label index in the category, respectively. to further adapt bart-ner for few-shot settings, lightner @xcite proposed a lightweight tuning approach for low-resource settings by adding a unified learnable verbalizer and incorporating learnable parameters into the self-attention layers. nonetheless, due to the fact that pointer mechanism only outputs the indexes of entities and labels, the model encounters challenges in effectively leveraging the capabilities of plms to directly comprehend the semantic meaning between entities and labels. thus instead of using a pointer mechanism, instruction-ner @xcite directly generates entity spans and types in the target sequence and applies instruction fine-tuning with two auxiliary tasks to further mining the capabilities of plms, which leads to significant few-shot improvements. in terms of the auxiliary tasks in instructionner, they propose two auxiliary tasks from two perspectives: span recognition (entity extraction) and entity labeling (entity typing). however, we argue that ner can be further divided into three parts: 1) understand the the relationship between the label and semantic meaning of the sentence. 2) extract the spans. 3) annotate the given spans. we believe that both span recognition and entity labeling can be benefit from having a deeper understanding of the label semantics. therefore, we proposed a new auxiliary task, called type extraction, to help the model to acquire this ability. meanwhile, none of the above methods take the additional external knowledge into account. current literature related to utilize external knowledge in ner involve @xcite and @xcite . sdnet @xcite proposes a self-describing mechanism to leverage external resources by self-describing both entity types and mentions, while @xcite ) uses a demonstration-based method by incorporating examples to the input but without a text-to-text framework. therefore, to the best of our knowledge, there is currently no existing literature that combines in-context external knowledge with instruction fine-tuning for few-shot ner. in this paper, we propose 2iner(instructive and in-context learning on few-shot ner). we build upon the work of instructionner by incorporating in-context examples and a novel auxiliary task. specifically, we first reformulate the ner tasks into a text-to-text framework and then employ t5 @xcite for natural language generation. in terms of the source sentence, we use instructions to distinguish between tasks by giving a comprehensive task description and include an alternative field to identify the entity type that requires detection. moreover, we suggest incorporating in-context demonstration examples into the source sentence to enable the model to learn from external knowledge. for the target sentence, we use natural language to represent entity spans and types instead of pointer mechanism. in addition to the two auxiliary tasks used in instructionner, we propose a new task called type extraction to further explore the potential of plms to understand label semantics. type extraction task requires the model to identify all the entity types presented in the original sentence and learn to understand the meaning of entity types at the overall semantic level of the sentence. our contributions can be summarized as follows: ## conclusion in this paper, we propose 2iner for few-shot ner using both instruction finetuning and in-context learning by converting ner into a text-to-text framework. based on instructionner, we create a template to concatenate task-specific instructions, input sentence and entity alternatives to make full use of the pre-training knowledge. besides, we decompose ner into three steps and introduce another auxiliary tasks, called type extraction, to help the model better understand the general semantic meaning of the entity types, which can improve both span recognition and entity labeling abilities. moreover, we apply the in-context examples to enable the model to learn from additional contextual information, enhancing few-shot abilities. multiple experiments on four ner datasets prove 2iner's effectiveness in few-shot ner scenario by consistently outperforming other baselines.