Fine-Tuning Large Language Models for Relation Extraction within a Retrieval-Augmented Generation Framework (2025)

Abstract: Information Extraction (IE) plays a pivotal role in transforming unstructured data into structured formats, such as Knowledge Graphs. One of the main tasks within IE is Relation Extraction (RE), which identifies relations between entities in text data. This process enriches the semantic understanding of documents, enabling more precise information retrieval and query answering. Recent works leveraging pre-trained language models have demonstrated significant performance improvements in RE. In the current era of Large Language Models (LLMs), fine-tuning these LLMs can mitigate the limitations of zero-shot RE methods, particularly in overcoming the domain adaptation challenges inherent in RE. This work explores not only the effectiveness of fine-tuned LLMs but also their integration into a Retrieval-Augmented Generation (RAG)-based RE approach to address domain adaptation challenges when general-purpose LLMs serve as generators within the RAG framework. Empirical evaluations on the TACRED, TACRED-Revisited (TACREV), and Re-TACRED datasets reveal substantial performance improvements with fine-tuned LLMs, such as Llama2-7B, Mistral-7B, and Flan-T5 Large and surpass previous methods on these datasets.

## introduction information extraction (ie) converts unstructured data into structured formats, such as knowledge graphs (kgs). a key ie task is relation extraction (re), which identifies relationships between entities in text at sentence (see figure 1 ) or document levels @xcite . re methods include supervised, unsupervised, and rule-based approaches @xcite . supervised re methods generally yield strong performance but require extensive labeled data. however, recent studies show that re methods using pre-trained language models (plms) can surpass traditional supervised approaches @xcite @xcite . in the era of large language models (llms), retrieval-augmented generation (rag) @xcite using zero-shot prompting settings, in-context learning @xcite , or simple vanilla prompting @xcite , have been utilized for re without the need for additional model training. italian red cross worker eugenio vagni was freed in the restive southern philippine province of sulu early sunday, almost three months after he and his two colleagues were abducted by local militants, the military said. the rag-based prompting approach performs well when entity relations are easily derived from sentence tokens but struggles when relation types are not introduced into llms @xcite . general-purpose llms, like mistral @xcite , llama2 @xcite , and flan-t5 @xcite , also show shortcomings in re tasks due to insufficient domain-specific relation knowledge @xcite @xcite . incorporating these relation types into llms could enhance re through zero-shot prompting @xcite . to tackle this issue, we fine-tune language models on small sets of re prompt datasets to enhance their ability to identify relations between entities at the sentence level. to evaluate the performance of fine-tuned llms, we conduct experiments using llama2-7b foot_0 @xcite , mistral-7b-instruct-v0.2 2 (jiang et al., 2023), and flan t5 large @xcite across three re benchmark datasets: ta-cred @xcite , tacred-revisited (tacrev) @xcite and re-tacred @xcite . in this work, fine-tuning is used to overcome the limitations of zero-shot llm prompting settings, such as rag4re @xcite , in identifying relations between entities across tacred and its variants. the contributions of our approach are as follows: the rest of this paper first summarizes re approaches using the language models in section 2 and then introduces our proposed approach foot_2 in section 3. afterwards, the experimental setup and results are presented in section 4 and discussed in section 5. lastly, all concluding remarks and future works are summarized in section 6. ## related works relation extraction (re), as a core task of information extraction (ie), plays a significant role in natural language processing. re aims to identify or classify the relations between (head and tail) entities in a given text. in this work, we primarily focus on sentence-level re approaches. re can be achieved through various methods: supervised, unsupervised, distant supervision, weak supervision, and rule-based @xcite . supervised methods require costly, annotated data @xcite ; distant supervision reduces data needs but risks noise @xcite ; weak supervision may lead to semantic drift @xcite ; and rule-based methods are limited by predefined rules @xcite . in addition to the fundamental approaches, leading re methods with fine-tuned llms include cohen et al.'s span prediction for broader entity relations (cohen et al., 2020), deepstruct's structural enhancements, zhou et al.'s entity-aware self-attention @xcite , and li et al.'s label graph for top-k prediction analysis @xcite . furthermore, @xcite used multiplechoice prompts, improving re predictions with added context, though it does not surpass prior rule-based methods. chen et al. @xcite introduced context-aware prompt tuning, while rag4re (efeoglu and paschke, 2024) utilized retrieval-augmented prompting, all tested on ta-cred and similar benchmarks. in this work, we aim to fine-tune llms on re prompt datasets to improve domain adaptation and evaluate their performance on benchmark datasets. ## methodology this work addresses the challenge of sentencelevel re using general-purpose llms with zeroshot prompting. general-purpose llms struggle with domain-specific relation types, so we fine-tune them on a small re prompt dataset to improve their ability to identify entity relations. we detail the fine-tuning process in section 3.1 and describe the integration of fine-tuned llms into the rag4re approach in section 3.2. ## evaluation we evaluate our approach using three benchmark datasets and language models. in section 4.1, we detail the datasets, metrics, and experimental settings, including the fine-tuning of language models and the use of retrieval-augmented generation with these fine-tuned models. then, we present and analyze the experimental results, comparing them with those of previous high-performing re methods in section 4.2. ## discussion our findings demonstrate significant improvements over the original rag4re @xcite results on the tacred, tacrev, and re-tacred datasets, as shown in ## conclusion we address domain adaptation challenges in zeroshot relation extraction (re) with general-purpose llms by fine-tuning flan-t5 large, mistral-7b-instruct-v0.2, and llama2-7b on tacred, tacrev, and re-tacred datasets. our finetuned models outperformed previous methods, including rag4re @xcite . integrating these fine-tuned llms into rag4re significantly enhanced its performance, especially with flan-t5 large. however, llama2-7b and mistral-7b showed inconsistent f1 scores, likely due to single-task fine-tuning issues. future work will explore multi-task fine-tuning for re and entity recognition to mitigate catastrophic forgetting @xcite @xcite .