Grasping the Essentials: Tailoring Large Language Models for Zero-Shot Relation Extraction (2024)

Abstract: Relation extraction (RE) aims to identify semantic relationships between entities within text. Despite considerable advancements, existing models predominantly require extensive annotated training data, which is both costly and labor-intensive to collect. Moreover, these models often struggle to adapt to new or unseen relations. Few-shot learning, aiming to lessen annotation demands, typically provides incomplete and biased supervision for target relations, leading to degraded and unstable performance. To accurately and explicitly describe relation semantics while minimizing annotation demands, we explore the definition only zero-shot RE setting where only relation definitions expressed in natural language are used to train a RE model. We introduce REPaL, comprising three stages: (1) We leverage large language models (LLMs) to generate initial seed instances from relation definitions and an unlabeled corpus. (2) We fine-tune a bidirectional Small Language Model (SLM) with initial seeds to learn relations for the target domain. (3) We expand pattern coverage and mitigate bias from initial seeds by integrating feedback from the SLM’s predictions on the unlabeled corpus and the synthesis history. To accomplish this, we leverage the multi-turn conversation ability of LLMs to generate new instances in follow-up dialogues, informed by both the feedback and synthesis history. Studies reveal that definition-oriented seed synthesis enhances pattern coverage whereas indiscriminately increasing seed quantity leads to performance saturation. Experiments on two datasets show REPaL significantly improved cost-effective zero-shot performance by large margins.

## introduction relation extraction is a pivotal task in information extraction (ie) that involves identifying and classifying semantic relationships between entities within texts. it has wide applications in various downstream tasks including knowledge graph construction @xcite , question answering (qa) @xcite and event mining @xcite . ## methodology our proposed repal is an iterative framework which consists of three major components: @xcite definition-based seed construction with llm and the unlabeled corpus ( sec. 3.1). (2 ) pattern learning with re-specialized slm (sec. 3.2) which trains a slm extractor with initial seeds for inference efficiency and performance. (3) feedback-driven instance improvement and bias rectification (sec. 3.3) by providing llms with independent feedback and leveraging llms' multi-turn conversations for followup positive and negative instance synthesis. the framework overview is shown in fig. 4 . ## experiments 4.1 evaluation dataset construction we first construct datasets for model evaluation on the definition only zero-shot relation extraction task. our constructed datasets are based on fewrel @xcite and wiki-zsl (chen and li, 2021) respectively 4 . the public portion of fewrel contains 80 relations, each comes with 700 instances annotated by crowd workers. wiki-zsl consists of 113 relations with total 93,383 instances obtained by distant supervision. as their relations are from wikidata's 5 properties, we query the property definitions and slightly modify them to become complete sentences with entity placeholders [ent0] and [ent1] (corresponding to e 0 and e 1 as in definition notation d(e 0 , e 1 )). for evaluation, we sample 5 groups of 14 fewrel relations and 3 groups of 15 wiki-zsl relations as test sets. to construct unlabeled corpus, for each group of the test set, we down-sample 10,000 instances from the unlabeled corpora used by gao et al. for fewshot relation learning which contains 744 distantly supervised relations (labels are not used in this work) and in total 899,996 instances (2020). the final test sets with the corpus are denoted as defon-fewrel and defon-wiki-zsl respectively. 4 an ideal test set for our setting should follow: (1) annotations should follow the officially documented relation definitions and the sentences should be sufficient to deduce the target relations without external knowledge; (2) relations are better semantically disjoint without ambiguity so that we do not need to manually adjust binary test labels for overlapped relations in each test iteration. these principles also apply to existing multi-way classification re datasets. however, we leave these for future dataset related work. 5 wikidata main page ## related work zero-shot relation extraction our work is related to zero-shot re @xcite . majority zero-shot re approaches mainly leverage clustering, label-verbalization, or siamese-based architectures @xcite @xcite which seek for the instance-instance similarity or the similarity between the relation instances and the unseen relations' information. chen and li (2021) utilize relation descriptions for zero-shot re but their approach still relies on seen data to align relation descriptions with instances in a supervised manner. @xcite adopt the relation descriptions but only for verifying synthesized data with the instance-level seeds. llm-based re works focus on designing prompting strategies or llm alignment to tackle zero-shot re @xcite @xcite . our work is distinguished from pre-llm zero-shot re works as they heavily rely on the supervision from massive seen data and the complete negative relation space. and the majority do not focus on relation definitions. our work is different from llm-based zero-shot re works as we emphasize both the rich relation definitions for data synthesis and synergy between slm and llm. definition-driven text mining bertnet @xcite applies definitions for distilling entities from lm parametric knowledge. label definitions/descriptions have also been proven to be powerful in text classification @xcite . in zero-shot re, several plm-based works have utilized relation definitions @xcite but they mainly focus on computing instance-definition similarities. in our work, llm is used to distill patterns and extend or rectify the learning of slm based on definitions. ## conclusion in this work, we have introduced a new zero-shot re task where only relation definitions instead of seen-unseen relation instances are provided. correspondingly, we have proposed repal which leverages llms and unlabeled corpora to generate relation instances and iteratively self-improves the generation pattern coverage while rectifying the bias by automatically acquiring and reflecting over sampled feedback from multiple sources. quantitative experiments and qualitative analysis on our two modified datasets show the effectiveness and robustness of our framework as well as our largemargin advance over most baselines. exploratory experiments show that generating more data in a single-turn conversation does not yield proportionally larger pattern coverage. we also proposed a derive-definition-then-generate approach which achieves much better performance than just utilizing few-shot instances. this gives insights into low-resource re works to capture the complete relation semantics to avoid partial coverage by fewshot instances.