From Syntax to Semantics: Evaluating the Impact of Linguistic Structures on LLM -Based Information Extraction (2025)

Abstract: Large Language Models (LLMs) have brought significant breakthroughs across all areas of Natural Language Processing (NLP), including Information Extraction (IE). However, knowledge gaps remain regarding their effectiveness in extracting entity-relation triplets, i.e. Joint Relation Extraction (JRE). JRE has been a key operation in creating knowledge bases that can be used to enhance Retrieval Augmented Generation (RAG) systems. Prior work highlights low-quality triplets generated by LLMs. Thus, this work investigates the impact of incorporating linguistic structures, such as constituency and dependency trees and semantic role labeling, to enhance the quality of the extracted triplets. The findings suggest that incorporating specific structural information enhances the uniqueness and topical relevance of the triplets, particularly in scenarios where multiple relationships are present.

## introduction ie is a crucial nlp task that extracts structured knowledge from unstructured data. named entity recognition (ner) and relation extraction (re) are two essential sub-processes that facilitate ie. they play an integral role in the population of knowledge bases (kbs), where entities serve as nodes and relationships as connecting links. most recently, ner and re have been employed to create knowledge graphs for graphrag applications @xcite . a popular paradigm, jre, unifies ner and re by identifying entities and relationships in a single task from the text sample. jre garnered significant attention from the nlp community before llms emerged. however, research on this topic has become scarce with the advent of llms, leaving a gap in understanding their impact on jre. the limited studies exploring llms for jre have highlighted issues such as redundancy in extracted triplets and low topical similarity to the target sentence @xcite . these issues are further exacerbated when there is a possibility of multiple relationships. these problems, coupled with the challenges in evaluation caused by the open-ended nature of the llms @xcite , have limited their usage for this task. however, llms possess vast knowledge and strong instruction-following capabilities, suggesting their potential to serve as effective joint extractors. linguistic structures have been employed as an aid for ie systems through the advancements in nlp. on the one hand, dependency trees (dt) have imparted fine-grained knowledge about the connections between the words in a sentence to neural networks @xcite @xcite . on the other hand, semantic role labeling (srl) has shown close connections with openie triplets @xcite . additionally, constituency trees (ct) have provided a structured representation of input text to the models @xcite . this work investigates the potential of using linguistic structures ct, dt, and srl as additional knowledge to the llms to enhance their performance on sentence-level jre. the goal is to improve their fine-grained semantic understanding from the comparatively shorter context present at the sentence level. our findings indicate that structural information improves the quality of extracted triplets, especially with smaller models. additionally, these linguistic structures help llms better handle text with multiple relationships, ensuring more accurate and contextually relevant triplets. ## results this study investigates the influence of linguistic structures on llm-based jre models. as discussed above, both traditional and soft metrics were used to quantify the influence of these elements. figure 2 depicts the performance of the selected llms across datasets and prompting strategies. it can be observed that adding structural information helped enhance the quality of triplets, specifically increasing their uniqueness and topical similarity. however, it had a detrimental effect on the completeness with respect to the ground truths. finegrained dataset-specific scores can be found in the appendix in ## discussion the results highlight the redundancy reduction in the extracted triplets, specifically when dependency-based structural information was added to the llm's prompt. previous research has highlighted that triplet redundancy increases when the source text contains multiple relationships. to analyze this effect, the us scores were examined across sentence categories with varying numbers of ground truth relations. the methodology for this experiment can be found in the appendix in section a.1.5. figure 3 indicates that linguistic structures help maintain the uniqueness of extracted triplets compared to baseline llms. while the us score for baseline experiments decreases as the number of relations increases, the us scores for llms with structural information remain relatively consistent. additionally, statistically significant gain was observed between the "baseline" and "base+dt" experiments in the n2-n5 categories. thus, it can be inferred that linguistic structures help llms differentiate between distinct relationships, thereby helping them extract only the most relevant triplets. some examples of how llms can filter out similar meaning triplets by enhancing their semantic reasoning capabilities can be found in appendix a.2.2. @xcite . very few studies have been conducted for the jre objective, likely due to the difficulties in evaluation. these studies investigate the potential of using llms as zero-shot and fewshot extractors by experimenting with various icl and prompting strategies @xcite @xcite . recently, the genres benchmark @xcite was proposed to qualitatively evaluate llm-based jre extractors. ## conclusion this study investigates how incorporating linguistic structures influences llms' entity and relation extraction capabilities. the results highlight the improved quality of extracted triplets when structural information is incorporated into the llms' prompts. this enhancement in quality was achieved through the reduction of redundant triplets (especially in the presence of multiple relationships) and increased similarity to the source text-both critical in real-world applications. for instance, in a kb construction task from finance data where multiple relationships are common, redundant triplets can create unnecessary paths, reducing the efficiency of the kb. in contrast, topical relevance is critical when extracting knowledge from user-facing systems such as chatbots, making extracting the most relevant entities and relationships essential. further, the study highlights a potential drawback: the inclusion of structural information for quality enhancement comes at the cost of misalignment from the ground truth labels, which are oftentimes very restrictive. this finding suggests a trade-off between completeness and uniqueness, which should be carefully considered based on the application's requirements. finally, there is a need to re-evaluate sota datasets in ie, as many contain highly specific and constrained labels. developing datasets with more generalized label spaces would provide a more comprehensive evaluation framework for ie systems in the llm era.