Double-Checker: Large Language Model as a Checker for Few-shot Named Entity Recognition (2024)

Abstract: Recently, few-shot Named Entity Recognition (NER) has attracted significant attention due to the high cost of obtaining high-quality labeled data. Decomposition-based methods have demonstrated remarkable performance on this task, which initially train a type-independent span detector and subsequently classify the detected spans based on their types. However, this framework has an evident drawback as a domain-agnostic detector cannot ensure the identification of only those entity spans that are specific to the target domain. To address this issue, we propose Double-Checker, which leverages collaboration between Large Language Models (LLMs) and small models. Specifically, we employ LLMs to verify candidate spans predicted by the small model and eliminate any spans that fall outside the scope of the target domain. Extensive experiments validate the effectiveness of our method, consistently yielding improvements over two baseline approaches. Our code is available at https://github.com/fanshu6hao/Double-Checker.

## introduction in recent years, few-shot named entity recognition (ner) has attracted significant attention due to the high cost of obtaining high-quality labeled data @xcite . this task mainly focuses on enabling the model to learn from a resource-rich source domain dataset, and further requires the model to predict unseen entity types in a resource-scarce target domain based on a small amount of data, i.e., the support data @xcite . to solve the above problem, a common approach is to decompose the task into entity span detection and entity type classification @xcite . specifically, a type-independent entity span detector is first trained, and then the * corresponding authors type classification is performed according to the detection spans. since the span detector trained in the first stage does not need to focus on specific entity types, it can effectively reduce the distribution gap between the source domain and the target domain, and has excellent performance @xcite . however, this paradigm has an obvious drawback: a domain-agnostic detector cannot guarantee that the entity span identified is specific to the target domain, and it will obviously identify many non-target domain candidates 1 . fortunately, large language models (llms) have shown remarkable performance on various natural language processing tasks @xcite , such as semantic understanding @xcite , knowledge reasoning @xcite , and recommender systems @xcite @xcite . however, some recent studies point out that llms are not ideal for ner directly @xcite , and often need to decompose the task into multiple steps or continue to fine-tune on large-scale data @xcite @xcite . these methods will undoubtedly consume a lot of resources. therefore, in this paper, we propose to leverage the collaboration of small language models (slms) and llms to exploit their respective advantages: low resource consumption of slms and the extensive knowledge base of llms. we aim to address the non-target domain entity span problem inherent in slms while mitigating the high resource consumption of llms. along these lines, we propose double-checker, a framework where the llm functions as a checker. instead of re-identifying entities, the llm rechecks the candidates identified by the small model, ensuring more accurate and domain-specific entity recognition. specifically, we first obtain the candidates predicted by the slm target sentence: "the traditional view among chilean historians and historians of the inca empire is that maule river {location-bodiesofwater} was the frontier. " candidate possible type probability maule river location-bodiesofwater 8.299 inca empire location-gpe 4.385 chilean location-other 3.172 [maule river] correct step 1 [inca empire] recheck [chilean] recheck first-stage check given type definition, sentence, types, and candidate, answer the question. type definition: xxx refers to ..., none refers an entity that does not belong to the above types, or not an entity. sentence: the traditional view among chilean historians ... types: xxx, none. candidate: ... question: please select the most relevant type (from types) for candidate in the sentence. answer: the most relevant type of inca empire is none. answer: the most relevant type of chilean is location-other. second-stage check question: consider the possible type xxx, whether the candidate in the sentence is an entity or not and explain why. ## methodology in this section, we introduce double-checker, an efficient framework specifically designed for eliminating non-target domain candidates by rechecking the predictions made by small models. the framework consists of two main steps: firstly, we obtain the candidates predicted by the small model and select the ones to be rechecked; subsequently, a two-stage check utilizing llm is conducted. an overview of the framework is shown in figure 1 . ## conclusion in this paper, we presented double-checker, a framework that effectively combines llm and slm for few-shot ner task. specifically, we initially employed a type-adaptive selector to choose candidates predicted by the small model. subsequently, the llm is utilized to conduct a two-stage check process on these selected candidates, removing entity spans and non-entities that are not relevant to the target domain. extensive experiments conducted using two different small models consistently demonstrated significant improvements, thereby showcasing the efficacy of our approach.