Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors (2023)

Abstract: Recent work has shown that fine-tuning large language models (LLMs) on large-scale instruction-following datasets substantially improves their performance on a wide range of NLP tasks, especially in the zero-shot setting. However, even advanced instruction-tuned LLMs still fail to outperform small LMs on relation extraction (RE), a fundamental information extraction task. We hypothesize that instruction-tuning has been unable to elicit strong RE capabilities in LLMs due to RE’s low incidence in instruction-tuning datasets, making up less than 1% of all tasks (Wang et al. 2022). To address this limitation, we propose QA4RE, a framework that aligns RE with question answering (QA), a predominant task in instruction-tuning datasets. Comprehensive zero-shot RE experiments over four datasets with two series of instruction-tuned LLMs (six LLMs in total) demonstrate that our QA4RE framework consistently improves LLM performance, strongly verifying our hypothesis and enabling LLMs to outperform strong zero-shot baselines by a large margin. Additionally, we provide thorough experiments and discussions to show the robustness, few-shot effectiveness, and strong transferability of our QA4RE framework. This work illustrates a promising way of adapting LLMs to challenging and underrepresented tasks by aligning these tasks with more common instruction-tuning tasks like QA.

## introduction large language models (llms) @xcite @xcite have been shown to achieve impressive performance on many nlp tasks. using the in-context learning paradigm, without any parameter updating, llms are able to achieve comparable performance with small language models (lms) fine-tuned on thousands of examples (liu et al., 2022; min et al., 1 code and data are available at https://github.com/osunlp-group/qa4re . 2022a; @xcite . 2 more recently, finetuning llms on datasets containing thousands of downstream tasks transformed into an instruction following format (i.e., instruction-tuning) has been shown to improve llms considerably across the board, especially in zero-shot setting @xcite @xcite . we examine the capability of llms in identifying the relationship between entities in a sentence, i.e., relation extraction (re), a fundamental task in information extraction. recent work @xcite has found that llms underperform fine-tuned small lms for re in the biomedical domain. our results on general domain re in fig. 1 reveal that even two of the most advanced instruction-tuned llms, flan-t5 xxl @xcite ) and text-davinci-003 @xcite , fail to outperform the stateof-the-art (sota) zero-shot re method based on small lms @xcite . we hypothesize that the limited relation extraction capability of instruction-tuned llms could be a byproduct of the low incidence of re tasks in instruction-tuning datasets @xcite @xcite . 3 to address the low incidence issue, we propose the qa4re framework, which aligns re with multiple-choice question answering (qa), a task that appears much more frequently in most instruction-tuning datasets-around 12-15% of all the tasks in both @xcite and @xcite . specifically, by casting the input sentence as a question and possible relation types as multiple-choice options (fig. 2 ), llms are able to perform re by selecting the option representing the correct relation type. thorough evaluations on four real-world relation extraction datasets and six instruction-tuned models from two different series (openai @xcite ) show that qa4re brings significant gains over the standard re formulation on, validating its effectiveness and our hypothesis concerning the low incidence of re. more specifically, our framework enables textdavinci-003 and flan-t5-xxlarge to achieve an average of 8.2% and 8.6% absolute improvements in f1, respectively. for the first time, an llm is able to outperform prior small-lm-based sota in the zero-shot setting by a large margin. in-depth analyses further demonstrate the robustness and few-shot effectiveness of qa4re. more importantly, our framework has been proven to be effectively transferable on instruction-tuned models with various sizes, ranging from 80m to 175b. our contributions are summarized as follows: ## methodology in this section, we formally define the relation extraction problem and describe our multi-choice qa approach for the problem in detail. ## conclusions and future work in this work, we first show that even the most recent instruction-tuned llms underperform fine-tuned small lms on the relation extraction (re) task. to address this limitation, we reformulate re into multiple-choice question answering (qa) with the purpose of leveraging a task that is widely cov-ered in instruction-tuning datasets like qa, instead of re, which is barely present in these datasets. comprehensive experiments demonstrate that our qa4re framework unlocks the power of llms as zero-shot relation extractors, especially for two recent llms (text-davinci-003 and flan-t5 xxl). we also conduct thorough experiments to explore the robustness and few-shot effectiveness of our method as well as study in what llm training scenarios it is most effective. in future work, we hope to explore additional underrepresented tasks in instruction-tuning that might be challenging for llms and could be successfully aligned with more widely adopted instruction-tuning tasks like qa. additionally, we plan to continue exploring this line of work by leveraging our qa4re framework for other llms such as the opt-series @xcite and @xcite , which are not included in this work due to the limited computational resources and/or access.