LLM as a metric critic for low resource relation identification (2024)

Abstract: In extremely low resource relation identification scenario, small language models (SLMs) incline to overfit, which significantly diminishes their accuracy. Recently, large language models (LLMs) are gradually applied to classification tasks with converting original objective into the generation task via in-context learning. However, abundance of the classifier categories poses challenges in selecting demonstrations. Moreover, the mapping between category labels and textual descriptions requires expensive expert knowledge, thereby constraining the efficacy of in-context learning for LLMs. We uphold that SLM is optimal for handling classification tasks, and its shortcomings in the low resource setting can be mitigated by leveraging LLM. Hence, we propose a co-evolution strategy on SLM & LLM for relation identification. Specifically, LLM provides essential background knowledge to assist training process of the SLM classifier, while evaluation metrics from the classifier, in turn, offer valuable insights to refine the generation prompts of the LLM. We conduct experiments on several datasets which demonstrates preponderance of the proposed model.

## introduction relation identification aims to identify target relationship between a specify entity pair mentioned in a text. pre-trained language models (plms) @xcite are capable of absorbing and representing a wealth of knowledge from extensive data. fine-tuning plms has been demonstrated an effective approach for relation identification. however, the gap between the objectives of pre-training and fine-tuning often lead to performance decay in low resource scenarios. after that, prompt tuning is proposed to bridge this gap @xcite @xcite . concretely, a masked language modeling (mlm) prompt is constructed and concatenated with the given text as input (e.g., "he has a sister lisa. the relation between @xcite ."). the relation identification task can be transferred into a mlm problem by filling the @xcite token in the input. recently, llm has shown remarkable abilities. there are works using llm for relation identification via generation @xcite @xcite and comparing its performance with slm @xcite . the results show that llm succeeds slm only when the annotation type is extremely limited, i.e., both relation label and samples for each category are extremely scarce. when the number of samples increases slightly, slm significantly outperforms llm. furthermore, due to the constraint of prompt length, llm is unable to deal with lengthy texts and a large number of relations. despite these limitations, the question remains: how can the extensive knowledge encapsulated within llm be leveraged to enhance relation identification tasks? driven by this inquiry, we propose to harness the llm as a reservoir of knowledge, offering a com-prehensive context that can augment the training and performance of slm. given the imperfections of knowledge produced by llm, there is a risk yielding counterproductive outcomes. consequently, maintaining the accuracy of llm responses is essential for the effective collaboration between slm and llm. various strategies have been developed to improve it, including series of thoughts (chain of thoughts @xcite , tree of thoughts @xcite , graph of thoughts @xcite ) and post-processing optimization in interactive scenarios (reflexion @xcite , self-refine @xcite , self-contrast @xcite ). @xcite to use policylevel reflection and optimization to iteratively update prompt instructions, empowering the agent to progressively evolve. tran @xcite generates rules by observing mistakes from unsatisfactory generated content to guide llm for better performance. inspired by these during-interaction llm evolving approaches, we explore the possibility of collaboration between two different optimization modality, prompt optimization of llm and parameter optimization of slm. overall, we propose a co-evolution framework for low resource relation identification with combining adavantages of both slm and llm. as is shown in figure 1 , slm for relation identification and llm for knowledge enhancement alternately learn from each other during the training process. our contributions are summarized as follows: ## related work addressing the challenge of relation identification with few shot samples, prevailing strategies can be broadly categorized into two schools of thought. the first approach enhances traditional language models through prompt tuning. at its core, the method involves inserting textual snippets or tem-plates, into the input and recasting classification task as the mlm problem, which enhances model performance by integrating textual information or label information into the training process. know-prompt @xcite incorporates knowledge between relation labels into the prompt tuning phase of identification. specifically, it infuses the potential knowledge embedded within labels into the creation of prompts that include learnable virtual type tokens and answer terms. subsequently, their representations are jointly optimized under structured constraints. bayesprompt @xcite leverages a known distribution to approximate the debiased factual distribution on the target domain. it subsequently performs uniform sampling of certain representative features, thereby generating the final prompts. retrievalre @xcite regards relation identification as an openbook exam, proposing a retrieval-augmented semiparametric extraction prompt tuning paradigm. it establishes an open-book data repository, where instance representations based on prompts and their corresponding relation labels serve as key-value pairs for retrieval. during inference, the model infers relations by linearly interpolating the base outputs with a non-parametric nearest neighbor distribution retrieved from the data repository. the second strategy advocates for the synergy between traditional models and large-scale models, enabling them to capitalize on their respective strengths in low-data scenarios. this collaborative effort harnesses the precision and specialized knowledge of slm alongside the broader contextual understanding and generative capabilities of llm, resulting in a more robust and adaptable system even when data is scarce. ma et al. propose a novel approach named llm-ie where slm act as a filter and llm serves as a reranker. by prompting llm to rerank a small subset of difficult instances identified by slm, it achieves notable improvements across various information extraction (ie) tasks. gpt-re @xcite employs simcse @xcite to compute sentence similarities, extracts relation representations from a bert-based @xcite fine-tuning method for retrieving demonstration examples, and takes @xcite with generating the reasoning logic process for each example under the corresponding factual relation label. this collaborative effort between models compensates for the shortcomings within the gpt-3 framework, thereby enhancing its performance. relation identification (slm) context generator (llm) ce loss triplet loss mlm prompt data metric ℳ > 0 ℳ < 0 please give me the reasons to explain {compare result}. (output:{rule}) given {rule}. generate a short background text to help identify the relationship in {data}. (output:{context}) : in graphic arts and architecture, 'lines' refer to the fundamental... : an artist is a person skilled in the creation of art... auxiliary mlm [s] lines [mask] artist [/s] there are guide [e1] lines [/e1] under the paint that [e2] artist [/e2] used... ## proposed method this part demonstrates our co-evolution framework comprehensively. we perceive slm as the bedrock for relation identification task and introduce triplet constraint as the training auxiliary (see in section 3.1). with llm, we supply entity context to its representation in auxiliary task (section 3.2). we update the generation prompt on llm according to current training metrics, which procures a more robust context afterwards (section 3.3). ## experiments in this section, we conduct our experiments on several popular relation identification datasets, being in comparison with the sota baselines (section 4.1). we make an ablation study on the coevolution structure to verify its effectiveness (section 4.2). moreover, a case study is demonstrated to embody the dynamic improvement of both context and the rule (section 4.3). on a couple of gpus (tesla v100s-pcie-32gb), with one for slm training and another for llm generation. we set the learning rate of the slm part as 5e-5, and the training batch size as 8. coefficients of d m and α in equations 4 & 5 are 0.2 and 0.02 separately. for a stable training process, we execute three epochs for each training stage. correspondingly, the variable h in equations 10 and 12 updates its value every three epochs. evaluation datasets: semeval (semeval 2010 task 8 @xcite ), tacred series 1 (tacred @xcite , tacred-revisit @xcite and re-tacred @xcite ). these datasets are public for research utilization. baselines: slm series (knowprompt, re-trievalre, bayesprompt), llm series (llm-ie, gpt-re). ## conclusions we propose a co-evolution framework operated on slm & llm for relation identification task in the extremely low resource scenario. in the model, slm is the classifier which treats the task as prompt tuning, and llm serves as a knowledge base to provide entity contexts. we introduce an auxiliary task with triplet information as a bond between them. moreover, a three-stage training process is established which conveys training states of slm to llm for its generation reflection, hence requited contexts with improvement produced by llm will further foster the training of slm. we conduct experiments on four datasets with comparison to both slm and llm related methods which displays advances of our model.