Revisiting Relation Extraction in the era of Large Language Models (2023)

Abstract: Relation extraction (RE) is the core NLP task of inferring semantic relationships between entities from text. Standard supervised RE techniques entail training modules to tag tokens comprising entity spans and then predict the relationship between them. Recent work has instead treated the problem as a sequence-to-sequence task, linearizing relations between entities as target strings to be generated conditioned on the input. Here we push the limits of this approach, using larger language models (GPT-3 and Flan-T5 large) than considered in prior work and evaluating their performance on standard RE tasks under varying levels of supervision. We address issues inherent to evaluating generative approaches to RE by doing human evaluations, in lieu of relying on exact matching. Under this refined evaluation, we find that: (1) Few-shot prompting with GPT-3 achieves near SOTA performance, i.e., roughly equivalent to existing fully supervised models; (2) Flan-T5 is not as capable in the few-shot setting, but supervising and fine-tuning it with Chain-of-Thought (CoT) style explanations (generated via GPT-3) yields SOTA results. We release this model as a new baseline for RE tasks.

## introduction relation extraction (re) is the task of identifying entities and their semantic relationships from texts. standard supervised approaches @xcite to re learn to tag entity spans and then classify relationships (if any) between these. more recent work has shown that conditional language models can capably perform this task-achieving sota or near-sota results-when trained to output linearized strings encoding entity pairs and their relations @xcite @xcite . however, to date such work has considered only moderately sized 1 https://sominw.com/acl23llms pre-trained models for re such as bart @xcite . in this work we investigate the use of very large language models--including gpt-3 @xcite -for end-to-end relation extraction via generation. our contributions are as follows. our results indicate that, in general, llms should be the default approach to re, especially given that one can train flan-t5-which is dramatically smaller than gpt-3, and publicly available-to achieve sota performance (figure 1 ). ## re via text generation we treat re as a conditional text generation task. concretely, for a dataset of size n , we model the probability of generating a linearized string y of a relation triplet (entity_1, relation_type, entity_2) conditioned on a context string c. specifically, c includes a chain of n linearized examples (x i , y i ), with @xmath0. formally: we provide examples of context strings in the appendix. we conduct experiments over four standard re datasets comprising varying numbers of entities and relation types, namely ade @xcite , conll @xcite , nyt @xcite , and docred @xcite ; details in for inputs with multiple relation types (as in conll04 and nyt), we form triplets comprising a subject, relation, and object (along with their corresponding types), in the order of appearance of the subject entity: challenges inherent to evaluating generative large language models for re the expressivity of language models coupled with the openendedness of re makes evaluation difficult. this has led to inconsistent approaches to evaluation @xcite . past work, especially that pre-dating llms for the task, has tended to perform "strict" evaluation, requiring exact matches between generated linearized relation tuples and references. this may be appropriate when is evaluating smaller conditional generation models (such as bart) for re, which have been fine-tuned on large training sets, because after training such models consistently generate standardized outputs. by contrast, however, models like gpt-3 (or other large language models capable of zero-or few-shot application) can produce a wide variety of output formats which convey similar content. for example, given an input from ade and prompted to list all drugs and associated adverse events, a large language model might yield aspirin: stomach pain, chest pain. or it may instead output: side effects of aspirin include cramping and stomach pain, and pain in the chest. there are countless possible variants which may all communicate the correct answer; we provide additional real examples in the appendix d. the flexibility of language means that parsing out the structured result to compare it to a reference (to calculate standard metrics like precision, recall, and f-1) is a non-trivial problem. this is in stark contrast to traditional approaches to tasks like ner and re where models effectively classify input tokens instead of generating new ones from a vast vocabulary. training models, either via traditional supervised learning or in-context few-shot learning, encourages models to comport with the structure of training instances. we therefore focus our analysis on such supervised settings in this work, starting with an evaluation of few-shot learning with gpt-3 for re. nonetheless, even when supervised, llms used for re are prone to generating outputs which may be accurate but nonetheless differ from the target. to address this, we enlist human annotators to judge whether the model outputs convey the same information as the reference targets. ## in-context few-shot learning with gpt-for re in this section we first describe our few-shot prompting strategy for gpt-3, and report the results realized by this approach across a set of re corpora. we adopt forms of instructional in-context few-shot prompting to gpt-3. 2 motivated by the preceding discussion regarding evaluation challenges, we collect human annotations judging the model's generations against the gold references. finally, using these annotations we report results achieved using gpt-3 with few-shot prompting for re ( all references to gpt-3 in this work refer to the "text-davinci-002" variant. ## sota re performance with flan-t5 we use flan-t5 (large), an llm trained on a large number of tasks with instructional prompts. we first evaluate this in a few-shot setting (section 4.1), shortening prompts in light of t5's smaller size, compared to gpt-3. we then consider fine-tuned variants, including a novel approach in which we train flan-t5 using chain-of-thought (cot) style explanations for re elicited from gpt-3. the latter strategy yields sota results across all datasets considered. ## related work standard nlp methods for identifying relations in free text have included conditional random fields @xcite , structured svms @xcite , and more recently, training large deep learning models with a joint objective @xcite @xcite to identify entities and relations simultaneously. more recently, the rise of massive language models @xcite @xcite has also motivated research into prompt-based learning methods for structured prediction @xcite . ## conclusions and future directions we have evaluated the capabilities of modern large language models (llms)-specifically gpt-3 and flan t5 (large)-on the task of relation extraction (re). we found that, when evaluated carefully, gpt-3 performs comparably to fully supervised state-of-the-art (sota) models, given only 10s of examples. we then proposed a distillation technique in which we augmented target re labels with chain of thought (cot) style explanations elicited from gpt-3 and used this to fine-tune flan-t5; this yielded sota performance across all datasets considered, often by wide margins (5-10 points in f1). our results suggest that where feasible, llms should be a standard baseline for re.