Language is All a Graph Needs (2024)

Abstract: The emergence of large-scale pre-trained language models has revolutionized various AI research domains. Transformers-based Large Language Models (LLMs) have gradually replaced CNNs and RNNs to unify fields of computer vision and natural language processing. Compared with independent data like images, videos or texts, graphs usually contain rich structural and relational information. Meanwhile, languages, especially natural language, being one of the most expressive mediums, excels in describing complex structures. However, existing work on incorporating graph problems into the generative language modeling framework remains very limited. Considering the rising prominence of LLMs, it becomes essential to explore whether LLMs can also replace GNNs as the foundation model for graphs. In this paper, we propose InstructGLM (Instruction-finetuned Graph Language Model) with highly scalable prompts based on natural language instructions. We use natural language to describe multi-scale geometric structure of the graph and then instruction finetune an LLM to perform graph tasks, which enables Generative Graph Learning. Our method surpasses all GNN baselines on ogbn-arxiv, Cora and PubMed datasets, underscoring its effectiveness and sheds light on generative LLMs as new foundation model for graph machine learning. Our code is available at https://github.com/agiresearch/InstructGLM.

## introduction prior to the advent of transformers @xcite , various artificial intelligence domains with different inductive biases had diverse foundational model architectures. for instance, cnns @xcite were designed with considerations for spatial invariance in images, leading to superior performance in computer vision tasks @xcite . memoryenhanced models like rnns @xcite and lstm @xcite were widely used for handling sequential data such as natural language @xcite and audio @xcite . graph neural networks (gnns) have long been the preferred choice in graph learning due to their proficiency in capturing topological information through message passing and aggregation mechanisms @xcite @xcite . in recent years, the ai community has witnessed the emergence of numerous powerful pre-trained large language models (llms) @xcite @xcite @xcite , which are driving huge advancements and lead to the pursuit of artificial general intelligence (agi) @xcite . under this background, there is a trend towards unification in model architectures across different domains. specifically, pretrained transformers have demonstrated remarkable performance on various modalities, such as images @xcite and videos @xcite in computer vision, text in natural language processing @xcite , structured data in graph machine learning @xcite , personalized data in recommender systems @xcite , decision sequences in reinforcement learning @xcite , and visual-text pairs in multimodal tasks @xcite . there has even been transformers capable of handling twelve modalities @xcite . alongside advancements in model architectures, there is also a noteworthy trend towards the adoption of unified processing techniques for multimodal data. t5 @xcite established a text-to-text framework, unifying all nlp tasks as a sequence generation problem. moreover, models like clip @xcite utilize image-text pairs for multimodal tasks with the images captioned by natural language. in the realm of reinforcement learning, di @xcite improves the agent by employing natural language to describe environmental states. p5 @xcite @xcite and its variants @xcite @xcite , further contributes to this trend by reformulating all personalized recommendation tasks as language modeling tasks via prompts. the aforementioned works collectively demonstrate that employing natural language for multimodal data representation has emerged as a prominent and promising trend. however, in graph machine learning, such an exploration still remains limited. existing methods that utilize llms for graph can be roughly categorized into two types: 1) combining llms and gnns, where the llm acts as a feature extractor or data augmentation module to enhance the downstream gnns @xcite @xcite . these methods often require training multiple models, incurring significant computational overhead and tend to easily inherit drawbacks of gnns such as over-smoothing @xcite . 2) purely relying on transformers but necessitating novel designs of token embedding for nodes and edges @xcite or creating complex graph attention modules to learn structural information @xcite . this type of method demands local attention calculation on every node during each optimization step, leading to considerable computation costs and thus limiting each node's scope to only 1-hop neighbors. additionally, the complex pipeline with special attention mechanisms or token representations prevents the model from directly observing and learning structural information like gnns, thus restricting further improvement on performance. to address the issues of llm-based graph learning and bridge the gap between languages and graphs, we propose instructglm (instructionfinetuned graph language model). given that llms have succeeded in many ai domains, we aim to answer the question: besides cnns and rnns, can llms also replace gnns as the foundation model for graph machine learning? intuitively, as one of the most expressive medium, natural language is adept at describing complex structures such that instructglm owns the following advantages over gnns: 1) flexibility. a natural language sentence is capable of effectively describing the connectivity at any desired hop level and intermediate paths without iterative message passing and aggregation. even multimodal features of the nodes and edges can be directly integrated into natural language prompts, making natu-ral language a very flexible medium to convey both structure and content on the graph. in this paper, we focus on node classification and link prediction-two of the most fundamental tasks for graph learning. besides, self-supervised link prediction can augment and enhance the node classification performance. we design a series of graph prompts for generative llms. specifically, we systematically employ natural language to describe the graphs' topological structures according to our prompts, making the graph structure clearly and intuitively provided to llm without complex pipelines tailored to graphs. therefore, we can handle graph tasks efficiently and succinctly by the vanilla transformer architecture @xcite and language modeling objective @xcite in a generative manner. overall, our contributions can be summarized as: • structural information is the most fundamental information for graphs, and our research shows that this fundamental information can be effectively described by languages. to the best of our knowledge, we are the first to propose purely using natural language for graph structure representation and conduct instruction tuning on generative llms to solve graph problems. we eliminate the requirement of designing specific complex attention mechanisms tailored for graphs. instead, we offer a concise and efficient natural language processing interface for graph learning, which exhibits high scalability to a unified multimodal and multitask framework, aligning with the current trend across other ai domains. the results demonstrate our instructglm outperforms previous competitive gnn baselines and transformers-based methods across all three datasets, achieving the top-ranked performance. llm envisions a technical paradigm where "everything is tokenized". benefiting from llm's powerful expressive capability in representing raw data of various modality into text or non-text tokens, all types of node or edge features can essentially be transformed into llm-compatible tokens, thereby reshaping both the graph structure and the graph attribute information into language tokens, showing the general applicability of our approach. our experimental results validate the effectiveness of instructglm under general graph problem settings and emphasize the trend of utilizing generative llms as the new foundational model for graph machine learning. ## instructglm in this section, we introduce instructglm, a framework utilizing natural language to describe both graph structure and meta features of node and edge for generative llms and further addressing graph-related tasks by instruction-tuning. we start with notation setup, followed by outlining the principles behind the design of instruction prompts, and then present a detailed illustration of the pipeline. ## conclusions and future work to the best of our knowledge, this work is the first attempt to represent graph structure via natural language description and then further perform instruction-tuning on generative llms for graph learning tasks, demonstrating the huge potential of llms as the new foundation model for graph ml. our instructglm outperforms all single-model gnns and graph transformers on ogbn-arxiv, cora and pubmed datasets. moreover, benefiting from our highly scalable instruction prompts and unified generative pipeline applicable to multi-modality data, instructglm can be readily extended to valuable future works along four directions: 1) leveraging llms to generate improved features like tape, simteg @xcite and instruction prompts @xcite for instructglm; 2) enhancing instructglm with knowledge distillation @xcite and iterative training @xcite frameworks; 3) deploying instructglm on more graph tasks such as question answering on knowledge graphs @xcite ; 4) extending instructglm to other languages beyond natural language under the premise that "everything is tokenized," to include visual tokens, acoustic tokens, other multi-modality tokens, or even domain specific languages or tokens @xcite such as chemical languages. detailed future works are summarized in appendix section d. overall, our instructglm provides a powerful nlp interface for graph machine learning, with generative llms and natural language as the driving force, it further contributes to the trend of unifying foundational model architecture and pipeline across multiple ai domains for the agi pursuit.