Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples! (2023)

Abstract: Large Language Models (LLMs) have made remarkable strides in various tasks. Whether LLMs are competitive few-shot solvers for information extraction (IE) tasks, however, remains an open problem. In this work, we aim to provide a thorough answer to this question. Through extensive experiments on nine datasets across four IE tasks, we demonstrate that current advanced LLMs consistently exhibit inferior performance, higher latency, and increased budget requirements compared to fine-tuned SLMs under most settings. Therefore, we conclude that LLMs are not effective few-shot information extractors in general. Nonetheless, we illustrate that with appropriate prompting strategies, LLMs can effectively complement SLMs and tackle challenging samples that SLMs struggle with. And moreover, we propose an adaptive filter-then-rerank paradigm to combine the strengths of LLMs and SLMs. In this paradigm, SLMs serve as filters and LLMs serve as rerankers. By prompting LLMs to rerank a small portion of difficult samples identified by SLMs, our preliminary system consistently achieves promising improvements (2.4% F1-gain on average) on various IE tasks, with an acceptable time and cost investment.

## introduction large language models @xcite @xcite ) have shown remarkable abilities on various nlp applications such as factual question answering @xcite , arithmetic reasoning @xcite and logical reasoning @xcite . given the reasoning, memorization, instruction-following and few-shot adaption capabilities emerging from llms, it prompts a compelling question: can llms be used to boost performance in few-shot information extraction (ie) tasks? to answer this question, we conduct an extensive empirical study to compare the performance between llms using in-context learning foot_1 (icl) and fine-tuned small language models (slms). we fairly evaluate slms-based and llms-based methods across nine datasets spanning four common ie tasks: (1) named entity recognition, (2) relation extraction, (3) event detection and (4) event argument extraction. for each dataset, we explored four to six settings to encompass typical low-resource extents, from 1-shot to 20-shot or even more. given the potential sensitivity of llms' performance to the prompt context, we meticulously considered variations in instruction, demonstration number and selection strategy, prompt format, etc. our study reveals that llms excel over slms only when annotations are extremely limited, i.e., both label types foot_2 and the samples foot_3 per label are extremely scarce. with more (e.g., hundreds of) samples, slms significantly outperform llms. furthermore, llms incur greater inference latency and costs than fine-tuned slms. hence, we claim that current llms are not good few-shot information extractors in general. we further investigate whether llms and slms exhibit different abilities to handle various types of samples. we categorize samples according to their difficulty measured by slms' confidence scores, and compare llms' and slms' results within each group. we find that llms are good at hard samples, though bad at easy samples. we posit that the knowledge and reasoning abilities in llms enable them to handle hard samples (which are sim-ply beyond slms' capabilities) well. nevertheless, llms demonstrate strong predisposition to falsepositive predictions on negative samples. since most negative samples are easy samples (which could be solved readily by slms), the performance of llms on easy samples sometimes collapses and are usually much worse than fine-tuned slms. leveraging these findings, we pursue an approach to incorporate llms and slms within a single system and combine their merits. to this end, we propose a novel filter-then-rerank framework. the basic idea is that slms serve as a filter and llms as a reranker. specifically, slms initially predict and determine the difficulty of each sample. if the sample is a hard one, we further pass the top-n most-likely candidate labels from slms to llms for reranking. otherwise we view the prediction from slms as the final decision. by providing easy/hard samples with different solution strategies, our system utilizes each model's strengths to complement each other. also, it reranks only a small subset of samples and minimizes the extra latency and budgets for calling llms. with a modest cost increase, our framework yields a consistent f1 improvement, averaging 2.4% higher than previous methods on various few-shot ie tasks. to the best of our knowledge, this is the first successful attempt to use llms to enhance few-shot ie tasks. ## adaptive filter-then-rerank paradigm above findings can be summarized as: (1) slms generally outperform llms, especially with more training samples and fine-grained labels. (2) slms are much more time-and cost-efficient. (3) llms serve as powerful rerankers on hard samples that challenge slms. based on them, we propose a simple, efficient, and effective adaptive reranker that combines the strengths of slms and llms. ## conclusion through an extensive empirical study on nine datasets spanning four ie tasks, we find that llms, despite their superiority in extreme low-resource scenarios, are not effective few-shot information extractors in general. they struggle with ie-related prompts, have limited demonstration capacity, and incur high inference costs. however, llms significantly improve the performance on hard samples when combined with slm. building on these insights, we propose an adaptive filter-then-rerank paradigm to leverage the strengths of slms and llms and mitigate their limitations. this approach consistently achieves promising results, with an average 2.4% f1 gain across multiple few-shot ie tasks, while minimizing latency and budget costs.