Rethinking Negative Instances for Generative Named Entity Recognition (2024)

Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities for generalizing in unseen tasks. In the Named Entity Recognition (NER) task, recent advancements have seen the remarkable improvement of LLMs in a broad range of entity domains via instruction tuning, by adopting entity-centric schema. In this work, we explore the potential enhancement of the existing methods by incorporating negative instances into training. Our experiments reveal that negative instances contribute to remarkable improvements by (1) introducing contextual information, and (2) clearly delineating label boundaries. Furthermore, we introduce an efficient longest common subsequence (LCS) matching algorithm, which is tailored to transform unstructured predictions into structured entities. By integrating these components, we present GNER, a Generative NER system that shows improved zero-shot performance across unseen entity domains. Our comprehensive evaluation illustrates our system’s superiority, surpassing state-of-the-art (SoTA) methods by 9 F_1 score in zero-shot evaluation.

## introduction named entity recognition (ner) is a critical and challenging task in the field of natural language processing (nlp). previous ner models are constrained by a pre-defined label set and require extensive human annotations, which limits their flexibility and adaptability to unseen entity domains. recent advantages in llms have enabled the models to be capable of generalizing to unseen tasks @xcite in an auto-regressive generation manner, making it possible to construct powerful ner systems. however, despite these advancements, recent studies @xcite show that average ai literature music 48 52 40 67 60 60 67 64 66 63 68 76 69 68 69 81 gpt-3.5 sota (7b) gner-llama (7b) gner-t5 (11b) politics science movie restaurant 69 67 5 33 61 61 63 43 69 70 69 48 75 77 63 51 figure 1 : zero-shot performance of our models. our models gner-llama and gner-t5 both outperform the sota @xcite in zero-shot settings. gpt results are from @xcite . the zero-shot performance of llms still falls behind the supervised training state-of-the-art (sota) methods, as llms train with limited ner data. to bridge this gap, recent works have fine-tuned open-sourced llms on diverse ner datasets, enhancing their domain adaptability for ner tasks. they utilize varied task schemas to handle ner tasks across multiple domains. specifically, instruc-tuie @xcite is fine-tuned on a wide range of ie datasets using a single-round conversation manner. meanwhile, universalner @xcite found that querying all entities at once is less effective than making multiple inquiries, with each inquiry focusing on one entity type at a time. additionally, gollie @xcite enhances zero-shot performance with well-crafted code-style guidelines. however, these approaches primarily adopt an entity-centric training strategy, focusing on recognizing entities while overlooking the non-entity text, which is crucial as negative instances. actually, negative instances play an important role in traditional classification models like bert tagging @xcite . for generative models, the role of negative instances in the training process has not yet been fully explored. to calibrate the potential enhancement of including negative instances in training, we first conduct a preliminary study. through experiments, we show that negative instances can significantly boost the model's performance by (1) incorporating the contextual information, and (2) enhancing the label boundary between entities and non-entities. the possible drawback of introducing the negative instances is the increase of the prediction length, leading to inaccurate predictions, reflected by the word addition, omission and substitution. to tackle the inaccuracy drawbacks, we aim to design a more accurate and efficient algorithm to convert unstructured text into structured entities. inspired by the above observations, we design an effective and efficient generative ner framework named gner. we first design a proper task schema integrating negative instances into the instruction tuning process. additionally, we design an lcs matching algorithm to tackle the issues in the structuring process efficiently. this innovation ensures accurate categorization and alignment of extracted entities. we also demonstrate that zero-shot performance can be enhanced with beam search through a self-correction mechanism. these strategic developments collectively advance the gner framework, setting a new standard for accuracy and efficiency in the field of ner. we conduct experiments on two representative generative models, flan-t5 and llama. the resulting models, gner-t5 and gner-llama, outperform sota by a large margin. as stated in fig. 1 , gner-llama-7b outperforms the gol-lie @xcite trained on code-llama-7b by 6 f 1 score. furthermore, compared to the similarly configured model universalner, gner-llama-7b shows an improvement of 12.7 f 1 score, with a 2.5× boost in inference speed. we also showcase the potential of smaller models with our 780m gner-t5-large model, which outperforms all baseline models in both zero-shot and supervised scenarios. figure 2: a simplified example of instructions in in-structuie @xcite , universalner @xcite and gollie @xcite . @xcite , utilizing the bio-tagging scheme. then, different methods are proposed to address more complex scenarios, i.e., nested and discontinuous ner. these methods regard ner as question answering @xcite , span classification @xcite , dependency parsing @xcite , word-level relation classification @xcite , and so on. in most of these approaches, negative instances have played a crucial role in the training process, either by integrating all negative instances or employing sampling methods to select part of them @xcite . however, the performance of the above-mentioned supervised models significantly decreases in zero-shot settings @xcite , especially when the data and domain distribution significantly diverge from those seen of training. zero-shot ner instruction tuning @xcite , also known as multitask fine-tuning, has emerged as a leading method to achieve generalization to unseen tasks by finetuning pre-trained llms on a diverse collection of tasks phrased as text-to-text problems @xcite . in ner, numerous works have explored the potential of llms across diverse domains. for instance, instructuie @xcite ) is fine-tuned on a wide range of ie datasets and achieves impressive results in both zero-shot and supervised settings. universalner @xcite explores the effectiveness of knowledge distillation and multi-round conversational training paradigms in enhancing model generalization, achieving superior results. gollie @xcite introduces an innovative strategy by integrating well-crafted code-style guidelines into instructions, which has been found to further improve the model's zero-shot performance. a simplified version of the task schema of the mentioned methods above is shown in fig. 2 . it can be concluded that the task schema plays an important role in determining the learning paradigm of models, significantly influencing their performance. we observe that these methods are entity-centric, meaning only the entity portions are involved in training and used for backpropagation to train the model. ## incorporating negative instances in this section, we start from the entity-centric schema and explore the possible improvements by incorporating negative instances. through experiments, we demonstrate the impact of negative instances in mitigating unlabeled errors (ue), noisy errors (ne), and boundary errors (be) by (1) introducing contextual information and (2) enhancing entity boundaries. ## method in this section, we present our gner framework. we start by describing our task schema, which integrates negative instances into the training process for better usage of contextual information (section 3.2) and sensitivity to the entity boundaries (section 3.3), followed by the correlated tuning strategies. we also propose an effective longest common subsequence (lcs) matching algorithm, to convert the model's unstructured text outputs into structured data efficiently, thereby enhancing the accuracy of our system. ## conclusion this paper explores the potential of a strong generative named entity recognition (ner) system based on pre-trained llms by integrating the negative instances into training. through experiments, we have demonstrated significant advancements. our approach, which combines the inclusion of contextual information and a clear definition of entity boundaries through negative instances, has proven to be highly effective in improving the model's performance, especially in zero-shot scenarios where prediction uncertainty is high. the introduction of an lcs matching algorithm further addresses the challenges of converting unstructured text into structured entities, ensuring accurate categorization and alignment. these findings highlight the crucial role of negative instances in ner tasks and the potential of generative models to revolutionize the field.