Exploring the Potential of Large Language Models for Heterophilic Graphs (2025)

Abstract: Large language models (LLMs) have presented significant opportunities to enhance various machine learning applications, including graph neural networks (GNNs). By leveraging the vast open-world knowledge within LLMs, we can more effectively interpret and utilize textual data to better characterize heterophilic graphs, where neighboring nodes often have different labels. However, existing approaches for heterophilic graphs overlook the rich textual data associated with nodes, which could unlock deeper insights into their heterophilic contexts. In this work, we explore the potential of LLMs for modeling heterophilic graphs and propose a novel two-stage framework: LLM-enhanced edge discriminator and LLM-guided edge reweighting. In the first stage, we fine-tune the LLM to better identify homophilic and heterophilic edges based on the textual content of their nodes. In the second stage, we adaptively manage message propagation in GNNs for different edge types based on node features, structures, and heterophilic or homophilic characteristics. To cope with the computational demands when deploying LLMs in practical scenarios, we further explore model distillation techniques to fine-tune smaller, more efficient models that maintain competitive performance. Extensive experiments validate the effectiveness of our framework, demonstrating the feasibility of using LLMs to enhance node classification on heterophilic graphs.

## introduction large language models (llms) have demonstrated remarkable capabilities across a wide range of applications, from natural language processing @xcite to computer vision @xcite , leveraging the extensive open-world knowledge llms encode. inspired by these successes, recent efforts have extended the application of llms to the graph domain, particularly * co-first authors with equal contribution. in text-attributed graphs where node attributes are composed of textual information @xcite . in the context of heterophilic graphs, where connected nodes often exhibit contrasting features or class labels @xcite @xcite , llms offer a unique opportunity to enhance the understanding of complex semantic relationships between these connected nodes, which remains largely unexplored. existing approaches to addressing heterophily in gnns typically involve extracting shallow embeddings from textual information, using them as initial node features without fully exploiting their rich semantic content. they can be broadly categorized into two main strategies: non-local neighbor extension and architectural refinement @xcite . the former extends the node's receptive field to include distant, highorder neighbors @xcite or potential connections @xcite @xcite , thereby enhancing node representations through a broader scope of information integration. the latter modifies the core functions of gnns, such as the message aggregation and updating functions, to better suit heterophilic contexts @xcite . in summary, current methods for heterophilic graphs largely overlook the rich textual content associated with the nodes in real-world graphs, which can provide deeper insights into heterophilic contexts. for instance, textual content on hyperlinked webpages can enrich the understanding and prediction of heterophilic links. traditionally, gnns employ bag-of-words or shallow embeddings to incorporate textual attributes, which are inadequate for capturing complex semantics. while llms @xcite have been used to empower gnns for text-attributed graphs @xcite @xcite , existing efforts focus on homophilic graphs, leaving heterophilic graphs largely unexplored. in this work, we delve into the potential of llms for heterophilic graphs. to the best of our knowledge, this is the first investigation into exploiting llms for heterophilic graphs. we aim to bridge the gap between the general capabilities of llms and the unique characteristics of heterophilic graphs. specifically, we aim to address the following research questions. first, can llms be effectively adapted to characterize and identify heterophilic contexts? as llms encompass general open-world knowledge, they can be utilized for the semantic understanding of nodes' textual content. however, it is not sufficient to merely extract features from the textual content. unlike homophilic graphs, a key distinction of heterophilic graphs is that edges frequently form between dissimilar nodes. therefore, distinguishing heterophilic edges from homophilic ones is crucial for subsequent aggregation on graphs. to address this, we leverage llms (or any pre-trained language model in general) to identify heterophilic edges. specifically, we propose llm-enhanced edge discrimination, where we fine-tune an llm using low-rank adaptation (lora) @xcite to discriminate heterophilic and homophilic edges based on a limited amount of ground truth labels. this module focuses on adapting the general semantic capabilities of llms to the specific task of predicting heterophily between nodes. the fine-tuned llm is subsequently used to infer heterophilic edges on the graph to facilitate the integration of heterophilic contexts in the next stage. second, can llms effectively guide the finegrained integration of heterophilic contexts into graph models? with respect to a target node, nodes with a potential heterophilic (or homophilic) edge provide valuable heterophilic (or homophilic) contexts for the target node. given the diverging characteristics of homophilic and heterophilic contexts, it is important to differentiate them when integrating into a gnn. specifically, heterophilic contexts of a target node can be identified by our llmenhanced edge discrimination. building on this, we further propose llm-guided edge reweighting to further aggregate heterophilic and homophilic contexts through gnns. in this module, we aim to learn adaptive weights for both heterophilic and homophilic edges. these weights are adapted to each edge based on its features, structure, and heterophilic or homophilic characteristics, thereby guiding the fine-grained, edge-sensitive aggregation in gnns. additionally, llms often incur high computational costs even for inference, limiting their practical deployment for real-world applications. to this end, we further explore model distillation techniques @xcite to condense the knowledge from fine-tuned llms into small language models (slms) @xcite @xcite to speed up the inference stage required for edge discrimination and reweighting. specifically, we utilize the llm well-tuned for edge discrimination to generate high-quality pseudo-labels for heterophilic and homophilic edges. these pseudo-labels supplement the limited ground truth labels, forming an expanded label set that further enables the fine-tuning of slms. the fine-tuned slm then replaces the llm for conducting inference for the edge discrimination and reweighting, maintaining effectiveness while significantly reducing inference time. to summarize, we propose a two-stage framework that leverages llms for heterophilic graph modeling (llm4heg). our main contributions are as follows. (1) to the best of our knowledge, this is the first study to explore llms specifically for modeling heterophilic graphs. this exploration not only opens new research avenues but also provides valuable insights into the capabilities of llms in addressing unique graph characteristics such as heterophily. (2) we introduce llm4heg, a novel two-stage framework that fine-tunes llms to enhance gnns for heterophilic graphs. the two stages, llm-enhanced edge discrimination and llm-guided edge reweighting, accurately identify heterophilic edges and adaptively integrate heterophilic contexts into gnns, respectively. (3) we further investigate the distillation of llms fine-tuned for heterophilic edge discrimination into slms, achieving faster inference time with minimal performance degradation. (4) finally, we conduct extensive experiments on five real-world datasets and demonstrate the effectiveness and efficiency of our work. ## related work we review the literature on llm-based and heterophilic graph learning and highlight the key distinctions of our work from existing studies. existing research on llms for graph learning includes llm-based methods that adopt llm as the backbone and gnn+llm-based methods that integrate the advantages of both gnns and llms (liu et al. @xcite . the former works focus on aligning graph data with natural language via graph-to-token and graph-to-text approaches @xcite . the graph-to-token approach involves tokenizing graph data to align it with natural language, enabling joint understanding with data from other modalities @xcite . graph-to-text focuses on describing graph information using natural language @xcite @xcite . the latter harnesses the strengths of both language understanding from llms and structural analysis from gnns by using gnn-centric methods utilizing llms to extract node features from raw data and make predictions using gnns @xcite or llm-centric methods utilizing gnns to enhance the performance of llm @xcite . existing heterophilic graph learning approaches generally fall into two main strategies: non-local neighbor extension and architectural refinement @xcite . non-local neighbor extension approaches aim to extend the neighbors to include non-local nodes in the graph that may share similar labels or features. these methods often involve high-order neighbor mixing @xcite @xcite or discovering potential neighbors based on various distance measurements, such as feature-based distance @xcite , structure-based distance @xcite , or hybrid approaches @xcite @xcite @xcite . architectural refinement approaches enhance the gnn architecture by employing identifiable message aggregation to discriminate and amplify messages from similar neighbors while minimizing the influence of dissimilar ones @xcite @xcite , or by leveraging inter-layer combinations to capture information from different neighbor ranges @xcite @xcite , thereby improving the model's representation power in heterophilic graphs. the key distinctions of our method lie in two aspects. first, we explore llms to enhance textattributed heterophilic graphs' modeling specifically. while prior works leverage llms for textattributed graphs, including edge reweighting @xcite , they do not explicitly target heterophilic graphs. additionally, existing heterophilic graph methods often overlook rich textual node attributes, which provide essential semantic contexts. second, our two-stage framework employs an llm-enhanced edge discriminator to predict edge types, followed by adaptive message propagation in gnns using a comprehensive suite of information, including node semantics, structural contexts, and llm-inferred edge characteristics. while gbk-gnn @xcite follows a similar two-stage approach, it does not leverage the power of llms. ## proposed model: llm4heg in this section, we first introduce some preliminaries on the problem formulation and classic gnns. then we introduce the overview of the proposed model followed by details of different components. ## experiment in this section, we present an empirical study to demonstrate the feasibility of leveraging llms for node classification on heterophilic graphs. ## conclusion in this study, we explored the potential of llms to enhance the performance of gnns for node classification on heterophilic graphs. we introduced a novel two-stage framework llm4heg, integrating llms into the gnn learning process through an llm-enhanced edge discriminator and an llmguided edge reweighting module. llm4heg allows more precise identification of heterophilic edges and finer-grained context aggregation, leveraging the rich semantics in nodes' textual data. additionally, to address the computational challenges of deploying llms, we implemented model distillation techniques to create smaller models that achieve much faster inference while maintaining competitive performance. our extensive experiments demonstrate that llm4heg significantly improves node classification on heterophilic graphs, underscoring the potential of llms for advancing complex graph learning.