Injecting Structured Knowledge into LLM s via Graph Neural Networks (2025)

Abstract: Large language models (LLMs) have achieved remarkable success in natural language processing (NLP), but they often struggle to capture explicit linguistic structures and world knowledge. To address this limitation, we propose a hybrid model that integrates LLMs with graph neural networks (GNNs) to inject structured knowledge into NLP tasks. Our approach leverages the strengths of both components: LLMs provide rich contextual representations, while GNNs encode explicit structural priors from sources such as dependency trees, Abstract Meaning Representations (AMRs), and knowledge graphs. We evaluate the hybrid model on a diverse set of tasks, including semantic parsing, multi-hop question answering, text summarization, commonsense reasoning, and dependency parsing. Experimental results demonstrate consistent improvements over both standalone baselines and state-of-the-art methods, with relative gains of up to 2.3% in Exact Match scores for multi-hop QA and 1.7% in accuracy for commonsense reasoning. Ablation studies and sensitivity analyses further highlight the importance of balancing contextual and structural information. By bridging the gap between unstructured textual data and structured knowledge, our work advances the state of the art in NLP and paves the way for more interpretable and robust language models.

## introduction models like gpt-3 @xcite , bert @xcite , and t5 @xcite have demonstrated remarkable capabilities in understanding and generating human-like text. however, despite their successes, llms often struggle to capture explicit linguistic structures, such as syntactic dependencies or semantic relationships, which are critical for tasks requiring structured reasoning @xcite . this limitation raises an important question: can we enhance llms by integrating structured knowledge into their architectures? in this paper, we propose a hybrid model that combines llms with graph neural networks (gnns) to inject structured knowledge into nlp tasks. our approach leverages the strengths of both components: llms provide rich contextual representations, while gnns encode explicit structural priors from sources such as dependency trees, abstract meaning representations (amrs), and knowledge graphs. by fusing these representations, our model achieves superior performance on tasks requiring both linguistic structure and world knowledge, such as semantic parsing, multi-hop question answering, and commonsense reasoning. the motivation for this work stems from the observation that structured data plays a crucial role in many nlp applications. for example, amrs have been shown to improve semantic parsing @xcite , while knowledge graphs like con-ceptnet enhance commonsense reasoning @xcite . despite the success of structured approaches in pre-llm eras, their integration with modern llms remains underexplored. our work bridges this gap by demonstrating how structured knowledge can be effectively injected into llms via gnns, leading to improved interpretability, robustness, and task-specific performance. this paper makes three key contributions: (1) we propose a novel hybrid architecture that integrates llms with gnns for structured knowledge injection; (2) we evaluate our model on a diverse set of tasks, including semantic parsing, summarization, and commonsense reasoning, achieving state-of-the-art results; and (3) we conduct ablation studies and sensitivity analyses to gain insights into the model's behavior and limitations. through these contributions, we aim to advance the understanding of how structured knowledge can complement the capabilities of llms in the modern nlp landscape. ## literature review the integration of structured knowledge into nlp systems has long been a cornerstone of research in computational linguistics. early efforts focused on rule-based methods and statistical models, which relied heavily on handcrafted features and annotated datasets @xcite . with the advent of deep learning, attention-based architectures like transformers @xcite enabled end-to-end learning of contextual representations, reducing the reliance on explicit structural annotations. however, recent studies have highlighted the limitations of purely surface-level approaches, particularly in tasks requiring structured reasoning @xcite . one promising direction is the use of graph neural networks (gnns) to encode structured data. gnns have achieved significant success in domains such as social network analysis @xcite , molecular property prediction @xcite , and nlp tasks involving graphs, such as dependency parsing @xcite and amr generation @xcite . for example, @xcite demonstrated that gnns could effectively capture hierarchical relationships in text, improving performance on tasks like relation extraction and event detection. similarly, @xcite proposed a gnn-based framework for encoding discourse graphs, achieving state-of-the-art results on narrative understanding tasks. we have also studied similar approaches in @xcite @xcite . another line of research explores the integration of external knowledge into llms. knowledge graphs like conceptnet @xcite and wikidata @xcite have been widely used to augment nlp models with factual and commonsense information. recent work has focused on combining knowledge graphs with llms through techniques such as retrievalaugmented generation @xcite and knowledge-aware fine-tuning @xcite . for instance, @xcite introduced a method for injecting knowledge graph embeddings into transformer layers, achieving significant improvements in question answering and fact verification tasks. we have also studied similar work like @xcite . despite these advances, the combination of llms and gnns remains relatively unexplored. a notable exception is the work of @xcite , who proposed a hybrid model for incorporating dependency parse trees into llms using gnns. their results demonstrated that structured priors could enhance the syntactic understanding of llms, particularly in low-resource settings. similarly, @xcite explored the use of gnns to encode amrs for semantic parsing, achieving state-of-the-art performance on the amr bank dataset. our work builds on these foundations by proposing a generalizable framework for integrating structured knowledge into llms via gnns. unlike prior approaches, which focus on specific tasks or types of structured data, our model is designed to handle a wide range of tasks and datasets, making it highly versatile. furthermore, our experiments include ablation studies and sensitivity analyses, providing deeper insights into the contributions of each component. ## methodology our proposed hybrid model combines the strengths of llms and graph neural networks (gnns) to inject structured knowledge into nlp tasks. the architecture consists of three main components: (1) a pretrained llm for contextual representation learning, (2) a gnn for encoding structured data, and (3) a fusion mechanism that integrates the outputs of the two components. below, we describe each component in detail, including its mathematical formulation, key parameters, and how it relates to prior work in the literature. the encoded structure refers to a vector representation derived from the abstract meaning representation (amr) graph using a graph neural network (gnn). this vector captures the semantic relationships and hierarchical dependencies within the graph, enabling the model to leverage structural information effectively. ## conclusion in this paper, we introduced a novel hybrid model that combines large language models with graph neural networks to inject structured knowledge into nlp tasks. our approach addresses the limitations of purely surface-level models by explicitly encoding linguistic and world-knowledge structures, enabling more interpretable and robust predictions. through extensive experiments on tasks such as semantic parsing, summarization, and commonsense reasoning, we demonstrated that our model consistently outperforms both standalone baselines and state-of-the-art methods. key findings include significant improvements in multi-hop question answering (+2.3% em) and commonsense reasoning (+1.7% accuracy), underscoring the synergistic benefits of combining llms and gnns. ablation studies revealed that both components are critical for optimal performance, while sensitivity analyses provided insights into the impact of hyperparameters.