C - ICL : Contrastive In-context Learning for Information Extraction (2024)

Abstract: There has been increasing interest in exploring the capabilities of advanced large language models (LLMs) in the field of information extraction (IE), specifically focusing on tasks related to named entity recognition (NER) and relation extraction (RE). Although researchers are exploring the use of few-shot information extraction through in-context learning with LLMs, they tend to focus only on using correct or positive examples for demonstration, neglecting the potential value of incorporating incorrect or negative examples into the learning process. In this paper, we present C-ICL, a novel few-shot technique that leverages both correct and incorrect sample constructions to create in-context learning demonstrations. This approach enhances the ability of LLMs to extract entities and relations by utilizing prompts that incorporate not only the positive samples but also the reasoning behind them. This method allows for the identification and correction of potential interface errors. Specifically, our proposed method taps into the inherent contextual information and valuable information in hard negative samples and the nearest positive neighbors to the test and then applies the in-context learning demonstrations based on LLMs. Our experiments on various datasets indicate that C-ICL outperforms previous few-shot in-context learning methods, delivering substantial enhancements in performance across a broad spectrum of related tasks. These improvements are noteworthy, showcasing the versatility of our approach in miscellaneous scenarios.

## introduction information extraction (ie) is an important task in natural language processing, which aims to obtain structured knowledge from plain text. it can be applied to different domains, such as knowledge graph construction @xcite and question answering @xcite . with the rise of large language models @xcite @xcite , information extraction has made significant progress @xcite . recent advancements in few-shot ie have shifted focus from traditional supervised fine-tuning approaches towards in-context learning (icl) demonstrations with large language models (llms) @xcite . prior works @xcite @xcite have explored the use of natural language prompts or icl demonstrations to guide llms in labeling test data under fewshot settings, sometimes requiring additional pretraining or fine-tuning steps. to align more closely with the structured nature of information extraction tasks, recent methods @xcite @xcite adopt code-like or structured prompts to improve the consistency between pre-training and inference. however, these methods can not fully unleash the potential of llms, partly due to the reliance of models on limited positive data and their inability to learn from their own errors. to address this, we propose a contrastive incontext learning approach that utilizes both correct (positive) and incorrect (negative) examples to enhance the learning process of llms by exposing them to a broader spectrum of scenarios, including typical mistakes as shown in figure 1 . this method aims to exploit the often-overlooked value in negative data, thus enabling more comprehensive and robust information extraction capabilities. assume that the model has learned its own tasks and problem-solving modes from the positive ie data set, but the prediction is still wrong. should it think about the reasons for the errors, summarize the types of reasons, and try to avoid the above problems in the subsequent inference? then, adding negative sample-related information can help solve this problem. inspired by this, we integrate right/positive and wrong/negative examples as icl demonstrations to enhance the performance of in-context learning ie. specifically, we first use a large model to generate the label of annotated data to select hard negative samples, then select positive samples semantically similar to the current test data from training data, and then design the most in-context demonstrations using different models (nl-llms or code-llms). in the module that selects wrong/negative samples that contain more knowledge, we use semantic similarity-aware self-consistency to conduct ranking. to demonstrate the superiority of our method, we conduct experiments on three ner and four re benchmarks and carefully analyze the benefits of our approach. our main contributions are summarized as follows: ## task formulation given a sentence x with l tokens x 1 , x 2 , • • • , x l , ie tasks are to predict structured target y ( ner or re in this paper) from x. the target y in ner is entity spans e((e, t)|x i , . . . , x j ) with entity types. the e is entity in the sequence, and t is the entity type in the predetermined entity types t (such as loc, per, org). in the re task, the target y is a set of relations within entities, usually expressed in the form of a triple (e 1 , r, e 2 ). we not only predict the @xmath0, but also the entity types t 1 , t 2 of e 1 , e 2 . r denotes the relation types (e.g., work for, live in, located in). types of e 1 , e 2 also should be predicted. @xmath1. we treat ie as a text generation task that completes the text to be predicted targets via llms. we define the task name as type instruction i to prompt the llms for ner or re. when in nl-llms, its representation is a text sequence. in code-style prompts, it is treated as comments inspired by @xcite . we transform few-shot sample demonstrations into three parts: the first part is the sentence text, the second is the targets, and the third is a flag denoting whether the sample is positive or not. in nl-llms, the output is a list of tuples [(e 1 , t 1 ), . . . , (e j , t j )]. while in code generation, it is expressed as the operation of adding tuples to the dictionary like " en-tity_dict["person"].append["steve"] ". the given test sentence has the first two parts, like the fewshot sample demonstrations. in our work, we pay more attention to code-style large model information extraction beacause of its structure. ## conclusion in this work, we introduce c-icl, contrastive in-context learning for few-shot information extraction, including right/positive and wrong/negative demonstrations. in addition through type instruction demonstrations prompt mention tags in the ie task. from the contrastive samples, the llms could obtain effective information and indirect but positive, valuable additional knowledge for ie tasks. besides, our method adopts semantic similarity retrieval strategies and self-consistency votes to retrieve in-context examples better suited for the current sentence and task, significantly improving ie performance. extensive experiments prove the effectiveness of c-icl on various benchmarks.