Self-Improving for Zero-Shot Named Entity Recognition with Large Language Models (2024)

Abstract: Exploring the application of powerful large language models (LLMs) on the named entity recognition (NER) task has drawn much attention recently. This work pushes the performance boundary of zero-shot NER with LLMs by proposing a training-free self-improving framework, which utilizes an unlabeled corpus to stimulate the self-learning ability of LLMs. First, we use the LLM to make predictions on the unlabeled corpus using self-consistency and obtain a self-annotated dataset. Second, we explore various strategies to select reliable annotations to form a reliable self-annotated dataset. Finally, for each test input, we retrieve demonstrations from the reliable self-annotated dataset and perform inference via in-context learning. Experiments on four benchmarks show substantial performance improvements achieved by our framework. Through comprehensive experimental analysis, we find that increasing the size of unlabeled corpus or iterations of self-improving does not guarantee further improvement, but the performance might be boosted via more advanced strategies for reliable annotation selection.

## introduction there have been many works exploring new possibilities of the named entity recognition (ner) task in the era of large language models (llms) @xcite @xcite recently. these studies include designing advanced prompting methods for zero-shot prediction or few-shot in-context learning (icl) @xcite @xcite , training task-specific llms for ner @xcite , and generating data with llms to train small specific models @xcite @xcite . in this work, we explore the possibility of pushing the performance boundary of zero-shot ner with llms via self-improving. we focus on the strict zero-shot scenarios where no annotated data is available but only an unlabeled corpus is accessible, and no training resource or auxiliary models are available. we propose a totally training-free self-improving framework for ner, which utilizes an unlabeled corpus to stimulate the self-learning ability of llms. the framework consists of the following three steps. (1) step 1: we use llms to self-annotate the unlabeled corpus using selfconsistency (sc, @xcite . each annotated entity is associated with a sc score, which is used as the measure of the reliability of this annotation. (2) step 2: we select reliable annotation to form a reliable self-annotated dataset, during which diverse annotation selection strategies are explored, including entity-level threshold filtering, sample-level threshold filtering and two-stage majority voting. (3) step 3: for each arrived test input, we perform inference via icl with demonstrations from the reliable self-annotated dataset. various strategies for demonstration retrieval are explored. our contributions include: (1) we proposed a training-free self-improving framework for zeroshot ner with llms. (2) this framework achieved significant performance improvements on four benchmarks. (3) we conduct comprehensive experimental analysis, finding that increasing the size of unlabeled corpus or iterations of self-annotating does not guarantee gains, but there might be room for improvements with more advanced strategies for reliable annotation selection. ## related work information extraction with llm. the research of information extraction (ie) with llms includes prompt designing @xcite @xcite , task-specific llms instruction-tuning @xcite and data augmentation @xcite @xcite . @xcite use llm to annotate data, which is used to fine-tune a specific ie model, then the fine-tuned model is used to help select the data to be annotated in the next iteration. unlike previous works, this work propose a training-free self-improving framework to push the zero-shot boundary of llm on ner. different from @xcite , no seed labeled data, expert small model nor training resources are required in our framework. in addition, our work is orthogonal to previous prompt designing works. they explored various advanced prompt formats to boost performance, and did not utilize unlabeled corpus. unlike them, this work improves zero-shot ner by using unlabeled corpus without designing any complex prompt format. demonstrations in icl. some works explored factors that have impacts on icl @xcite @xcite . @xcite investigate the impact of randomly assigning labels to demonstrations in icl. however, this random labeling method is not suitable for tasks like ner, which requires label information on the token-level instead of sentence-level. different from them, we first use llm to make predictions on the unlabeled corpus, then select reliable selfannotated data as demonstrations. ## conclusion we propose a training-free self-improving framework for zero-shot ner with llms, which achieves significant performance improvements on four benchmarks. comprehensive experimental analysis shows that, simply increasing the size of unlabeled corpus or the iterations of self-annotation do not guarantee further improvement, but there might still be room for improvement with more advanced strategies for reliable annotation selection.