The contribution of LLM s to relation extraction in the economic field (2025)

Abstract: Relation Extraction (RE) is a fundamental task in natural language processing, aimed at deducing semantic relationships between entities in a text. Traditional supervised extraction methods relation extraction methods involve training models to annotate tokens representing entity mentions, followed by predicting the relationship between these entities. However, recent advancements have transformed this task into a sequence-to-sequence problem. This involves converting relationships between entities into target string, which are then generated from the input text. Thus, language models now appear as a solution to this task and have already been used in numerous studies, with various levels of refinement, across different domains. The objective of the present study is to evaluate the contribution of large language models (LLM) to the task of relation extraction in a specific domain (in this case, the economic domain), compared to smaller language models. To do this, we considered as a baseline a model based on the BERT architecture, trained in this domain, and four LLM, namely FinGPT specific to the financial domain, XLNet, ChatGLM, and Llama3, which are generalists. All these models were evaluated on the same extraction task, with zero-shot for the general-purpose LLM, as well as refinements through few-shot learning and fine-tuning. The experiments showedthat the best performance in terms of F-score was achieved with fine-tuned LLM, with Llama3 achieving the highest performance.

## introduction the goal of relation extraction (re) task is to identify and classify relationships between entities in unstructured texts. in domain-specific fields like economic foot_0 , this task is particularly challenging due to the complexity and diversity of linguistic expressions, as well as the presence of domain-specific terminology. extracting meaningful domain relations from documents requires models that can handle the inherent ambiguities and varied structures present in texts. over the past decade, deep learning has led to significant advancements in re tasks. pretrained models like @xcite and t5 @xcite have been extensively applied to general relation extraction, showing impressive results. in more specialized domains, models like @xcite leverage ope-nai's models within an in-context learning (icl) framework and use retrieval mechanisms to extract domain relations. although these models exhibit great potential, directly using them for domainspecific tasks can lead to suboptimal performance. this is primarily due to their limited ability to fully perceive internal relationships, especially when entity mentions are ambiguous or when the sentence structures are highly complex, which is the case in many specific domains. the arrival of llm represented a further step forward for nlp, and consequently for the task of extracting relations @xcite . however, research has shown that using llm does not result in significant performance gains compared with small models, particularly in the task of extracting relationships, which is similar to a classification problem @xcite . a way to improve llm performances for the re task on specific domains is to refine them. two techniques at least have proved their worth: fewshot learning and fine-tuning. the first one needs a simple set of prompts, while the second one, which is more costly, requires an annotated dataset and important computational resources. the key research questions we aim to address in this paper are the following: to answer these questions, we led several experiments, each of them involving a language model processed on the same corpus core @xcite which is a high-quality resource specifically designed for extracting economic relations. in this domain, preserving data confidentiality is a critical concern for organizations, particularly when dealing with sensitive economic information. sharing data with third-party servers via apis, which is often required for using proprietary llms, poses significant risks to privacy and security. as a result, organizations are increasingly prioritizing models that can be fully deployed, trained, and fine-tuned locally, ensuring that data never leaves their infrastructure. this approach not only addresses confidentiality concerns but also provides greater control over the training process, enabling the customization of models for specific tasks and datasets. these constraints strongly influenced our choice to focus on open-source models that could be installed and operated entirely on our servers, eliminating the need for external dependencies and ensuring compliance with strict data protection policies. the different tested models are a language model based on a bert architecture, a economic specific llm fingpt @xcite and three general llm: chatglm2 @xcite , xlnet @xcite and llama3 @xcite . these three models have been refined thanks to few-shot learning and fine-tuning techniques alternately. we report these experiments in the following. the rest of the paper is organized as follows. section 2 presents related work for re, limited to the sentence level, in specific domains and when using llm. section 3 outlines the problem and presents our methodology. specific-domain resources used for our experiments are described in section 4, and section 5 gives and comments the obtained results. we pro-pose in section 6 a discussion, before concluding and giving perspectives to this work. ## resources from the economic field in this section, we present the resources we used for relation extraction in the economic field, including the dataset and models tested throughout our experiments. ## experiments and results this section presents the experimental setup, results, and their analysis. we aim to address the following key research questions: ## discussion and conclusion the results of our experiments demonstrate that fine-tuning llm is a highly effective strategy for improving performance on domain-specific tasks, such as economic relation extraction. across our trials, models like llama3 consistently outperformed smaller bert-based models and exhibited significant performance gains when fine-tuned with domain-specific data. this study supports the hypothesis that while large models may not always show substantial improvement in general tasks, their adaptation to specialized domains is crucial for realizing their full potential. one key observation from the experiments is that fine-tuning even on a fraction of the available data (30-50%) yielded substantial improvements. however, further increases in data usage led to diminishing returns, indicating that optimal performance can be achieved without needing the full dataset. this underscores the importance of efficient resource allocation in training, as fine-tuning large models can be computationally expensive. moreover, fine-tuning open-source llm locally offers a compelling alternative to propriety solutions, especially in privacy-sensitive domains like economic, where data confidentiality is a critical factor. in conclusion, this study demonstrates that finetuning llm for domain-specific relation extraction not only improves performance but also offers a cost-effective and scalable solution. for future work, we aim to focus on the extraction of multiple triplets from paragraphs, where several relationships need to be identified within the same text. additionally, we plan to investigate the extraction of n-ary relations, extending the traditional binary relations extraction approach to handle more complex relationships involving multiple entities.