G en RES : Rethinking Evaluation for Generative Relation Extraction in the Era of Large Language Models (2024)

Abstract: The field of relation extraction (RE) is experiencing a notable shift towards generative relation extraction (GRE), leveraging the capabilities of large language models (LLMs). However, we discovered that traditional relation extraction (RE) metrics like precision and recall fall short in evaluating GRE methods. This shortfall arises because these metrics rely on exact matching with human-annotated reference relations, while GRE methods often produce diverse and semantically accurate relations that differ from the references. To fill this gap, we introduce GenRES for a multi-dimensional assessment in terms of the topic similarity, uniqueness, granularity, factualness, and completeness of the GRE results. With GenRES, we empirically identified that (1) precision/recall fails to justify the performance of GRE methods; (2) human-annotated referential relations can be incomplete; (3) prompting LLMs with a fixed set of relations or entities can cause hallucinations. Next, we conducted a human evaluation of GRE methods that shows GenRES is consistent with human preferences for RE quality. Last, we made a comprehensive evaluation of fourteen leading LLMs using GenRES across document, bag, and sentence level RE datasets, respectively, to set the benchmark for future research in GRE

## introduction relation extraction (re) is one of the most critical tasks in natural language processing @xcite . in essence, re transforms unstructured text into structured, actionable knowledge (e.g., knowledge graphs). however, the traditional re methods only mine the predefined patterns referring to the predefined sets of relations and entities, thus often struggling to capture the complexity of natural language. recently, large language models (llms) 1 source code and guidelines are available at https:// github.com/pat-jj/genres like @xcite , promise a transition to generative relation extraction (gre). llm-based gre methods are capable of comprehending the input texts and then identifying complex relationships without the constraints of predefined patterns in a zero-shot manner. this is particularly advantageous when there is a scarcity of training data, and the input texts are varied. existing applications of llms in gre are either performing binary classification tasks @xcite given entity pairs and a set of predefined relation types, or given restricted entity types @xcite , which overlook extensive novel relations and entities beneath the text. notably, to unlock the full power of llms in gre, we advocate a transformation from "defining a set of relation types" → "finding matches between entities" to "exploring as many relations and entities as possible without limitation" → "refinement" @xcite . this strategy elicits llms' implicit knowledge to discover a wider array of relationships with minimal predefined constraints @xcite ), which we define as "open gre" that can be applied to knowledge graph construction for various downstream tasks @xcite @xcite @xcite . we illustrate the difference of gre strategies in figure 1 . the versatility of gre, however, poses significant challenges in evaluation @xcite . specifically, we identified that traditional relation extraction (re) metrics like precision and recall only capture the exact matching with humanannotated reference relations, while gre methods often produce diverse and semantically accurate relations that differ from the references. as such, we argue that precision in gre should be verified against the source text, and recall should be based on soft matching to accommodate the output flexibility of generative models. furthermore, a proficient model should not only cover crucial information in the text but also avoid redundant results, ensuring the extracted knowledge is both comprehensive and atomistic. to navigate these new dimensions, we introduce genres (generative relation extraction scoring), a multi-dimensional framework tailored for evaluating gre. our key contributions are as follows. ## preliminaries definition 1 (source document) a source document d is a piece of free-text, which can be a sentence, a passage, or a document. definition 2 (extracted triples) a triple τ = ⟨s|r|o⟩ is a structure formatting a piece of free text into a subject s, a relation r, and an object o. example: for a sentence "alice lives in champaign.", "alice" is the subject, "live in" is the relation, and "champaign" is the object. together, they form a triple ⟨alice|live_in|champaign⟩. we define t @xmath0, τ 2 , ...] as a list of triples extracted from the source document d. ## genres evidenced by previous work conducting semi-open gre @xcite , traditional metrics for re like hard matching precision/recall/f1 are inadequate to evaluate gre tasks as the llm generations are flexible. to fill in this gap, we introduce genres, an automated multi-aspect evaluation framework for gre. genres are composed of a series of sub-scores defined as follows. ## conclusions in this paper, we introduced genres, a framework for evaluating generative relation extraction using large language models, marking a significant shift in the nlp field. which is crucial for reliable relationship extraction. the provided demonstrations span a variety of contexts and exemplify the format in which the relationships should be presented. the clear and topicoriented examples aim to fine-tune the model's performance, ensuring it can navigate the complexities of relation extraction with precision across both biomedical and general domains.