Each graph is a new language: Graph Learning with LLM s (2025)

Abstract: Natural language has been extensively used for modeling text-attributed graphs with LLMs. Natural language is used to describe the graph for LLMs to understand or serve as component of the graph, e.g., textual attributes for embedding generation. However, natural language is inherently redundant and unstructured, making it unsuitable for modeling high-order neighbors with LLMs. Specifically, (i) graph descriptions become verbose, overwhelming LLMs, and (ii) only relying on attribute embeddings limits LLM’s ability to capture the adequate graph structural information. These limitations make it difficult to model graphs both concisely and adequately using sole natural language with LLMs.Inspired by the observation that LLMs pre-trained on one language can achieve exceptional performance on another with minimal additional training, we propose Graph-Defined Language for Large Language Model (GDL4LLM). This novel framework enables LLMs to transfer their powerful language understanding capabilities to graph-structured data. GDL4LLM translates the graph into a graph language corpus instead of graph descriptions and pre-trains LLMs on this corpus to adequately understand the graph. This corpus represents the subgraph centered around target nodes concisely with only a few tokens during fine-tuning on downstream tasks. By treating the graph as a new language, GDL4LLM enables LLMs to model text-attributed graph adequately and concisely. Extensive experiments on five datasets demonstrate that GDL4LLM outperforms description-based and embedding-based baselines by efficiently modeling different orders of neighbors.

## methodology in this section, we introduce gdl4llm, a framework for modeling text-attributed graphs through graph language for downstream tasks. as illustrated in figure 1 , gdl4llm operates in two primary stages. in the pre-training stage, we construct a graph language corpus and pre-train llms to learn the underlying language patterns. the finetuning stage leverages this learned graph language to represent the subgraph centered around the target nodes, adapting the pre-trained llms for specific downstream tasks. ## experiments we conduct experiments across three datasets to evaluate gdl4llm's performance and analyze its behavior. we aim to address following four research questions: rq1: how does gdl4llm ## conclusion in this paper, we identify two key limitations of using natural language alone to model text-attributed graphs with llms: (i) graph descriptions become excessively verbose, particularly when modeling high-order neighbors, and (ii) textual attributes often lack sufficient structural information, limiting llms' ability to capture high-order graph structures. we address two main challenges: (i) the absence of a built-in mechanism in llms, akin to message passing in gnns, and (ii) the inter-order dependencies between high-order neighbors and target nodes. to overcome these challenges, we propose the gdl4llm framework, which enables llms to model text-attributed graphs as if learning a new language. gdl4llm consists of two main stages: (i) collecting a graph language corpus and pre-training llms on this corpus to adequately understand the graph, and (ii) fine-tuning llms with this corpus to concisely represent subgraph centered around target nodes. this approach allows llms to effectively capture graph structure of various orders. extensive experiments demonstrate the efficiency and effectiveness of gdl4llm in modeling text-attributed graphs for downstream tasks. our future work will explore more efficient graph learning methods with graph language.