Let’s Ask GNN : Empowering Large Language Model for Graph In-Context Learning (2024)

Abstract: Textual Attributed Graphs (TAGs) are crucial for modeling complex real-world systems, yet leveraging large language models (LLMs) for TAGs presents unique challenges due to the gap between sequential text processing and graph-structured data. We introduce AskGNN, a novel approach that bridges this gap by leveraging In-Context Learning (ICL) to integrate graph data and task-specific information into LLMs. AskGNN employs a Graph Neural Network (GNN)-powered structure-enhanced retriever to select labeled nodes across graphs, incorporating complex graph structures and their supervision signals. Our learning-to-retrieve algorithm optimizes the retriever to select example nodes that maximize LLM performance on graph. Experiments across three tasks and seven LLMs demonstrate AskGNN’s superior effectiveness in graph task performance, opening new avenues for applying LLMs to graph-structured data without extensive fine-tuning.

## introduction textual attributed graphs (tags) @xcite are pivotal in modeling complex real-world systems, from social networks to recommendation engines and information retrieval systems. in tags, nodes represent text documents, while edges depict relationships between them like shown in figure 1 . the interconnected nature of tags encapsulates rich knowledge and context, offering a more comprehensive understanding of the underlying data structures than isolated text analysis alone. recent advancements in large language models (llms) @xcite @xcite have demonstrated remarkable zeroshot and few-shot capabilities across a wide range of tasks. these llms excel in areas such as data is cs.lg; is cs.ai . to bridge this gap and empower llms for dataefficient graph learning @xcite , researchers have developed two primary approaches to integrate structural information into llm understanding. one line of research, exemplified by studies such as @xcite and @xcite , employs text-based serialization methods to encode k-hop neighbors as the context around the query node for llms (as illustrated in figure 1 (a) ). however, this method often overlooks nodes beyond the k-hop neighborhood, missing critical signals and introducing noise, leading to underperformance compared to graph neural networks (gnns) in tasks like node classification @xcite . another line of research, explored by @xcite ; @xcite ; @xcite , employs gnns to encode tags into compact graph tokens (fig. 1 (b) ). while this approach preserves structural information through the message-passing mechanism, it raises the modality misalignment issue between graph structure and text spaces, impeding effective integration with llms. these limitations underscore the need for a new solution that can better elicit the power of llms on tags. in this paper, we propose askgnn, a novel framework that transforms graph structural information and task supervision signals into a limited number of document node-label pairs. this approach benefits from llms' exceptional power in in-context learning (icl) @xcite @xcite , leveraging both graph structural information and textual information without extensive fine-tuning. at the core of askgnn is a gnn-based structureenhanced retriever, designed to select the most relevant document node-label pairs as icl examples for llm few-shot prompting. this strategy harnesses gnns' prowess in extracting structural information while circumventing the semantic gaps between text and non-text tokens @xcite @xcite . to optimize the retriever, we further introduce a learning-to-retrieve algorithm that fine-tunes the structure-enhanced gnn retriever, ensuring the selection of the most pertinent icl node examples. by integrating these components, askgnn offers a new solution that addresses the limitations of previous approaches, enabling llms to effectively process and utilize the rich structural and supervisory information inherent in tags, while leveraging the adaptability of icl to diverse tasks with limited supervision, as depicted in figure 1(c ). to summarize, our main contributions are as follows: ## related work llms for graph learning. applying llms to text-attributed graph tasks provides significant opportunities for advancing data-efficient graph learning @xcite @xcite @xcite . these tasks require llm to process and understand the complex graph structural information, a challenge that has only recently gained attention @xcite @xcite . some approaches enable llms to learn structural information by tuning their parameters @xcite or through graph instruction tuning @xcite . others transform graphs into hidden embeddings to aid in understanding relational data @xcite @xcite . however, these methods either cannot fully capture informative graph structural information or align the graph and text tokens. in-context learning. icl @xcite allows pretrained llms to make predictions for diverse downstream tasks by directly prompting them with a few examples of the task or textual instructions @xcite without finetuning. jiang et al. (2023) improves language models by iteratively retrieving relevant information during text generation, boosting performance in long-form tasks. asai et al. (2024)enhances language models through adaptive retrieval and self-reflection, outperforming stateof-the-art models in various tasks. rubin et al. (2022); li et al. (2023a); wu et al. ( 2023 ) train a separate retriever that selects relevant examples using a unified ranking framework, outperforming task-specific methods. this capability significantly reduces the adaptation effort compared to traditional fine-tuning approaches and has shown robust performance across a variety of models and tasks @xcite @xcite . unlike models such as bert @xcite , which require extensive fine-tuning for new tasks, llms leverage their pretrained knowledge effectively through in-context prompts @xcite . you are an ai trained to categorize arxiv computer science papers into specific categories based on their abstracts. your task is to analyze the paper description provided and identify the most relevant category. give me the category of this content. respond only with the category key (e.g., 'cs.ai', 'cs.sy'), without any additional text or explanation. input: neural networks have proven effective at solving difficult problems but designing their architectures can be challenging, even for image classification problems alone. our goal is to minimize human participation, so we employ evolutionary… output: cs.ne in this paper, we propose efficient progressive neural architecture search, a neural architecture search that efficiently handles large search space through a novel progressive ... your answer: [query node] [icl examples] [system prompt] figure 2: example of node classification task, illustrating how retrieved icl examples are integrated with the query node with the prompt. ## problem definition in this work, we define a tag as @xmath0, x ), where @xmath1, 1} n ×n represents the adjacency matrix, with a @xmath2, n is the total number of nodes and @xmath3. each node i is associated with a text document represented as a sequence of tokens among the nodes in tag, only a subset are labeled, denoted as @xmath4, y i )} m @xmath5, where m ≪ n and our objective is to encode both the graph structural information and task supervision signals by retrieving a subset of k labeled examples, d @xmath6, y i )} @xmath7. these examples serve as context for prompting llms to address graphspecific tasks for a query node q. this retrieval process, constrained by both context length and computational costs, selects far fewer examples than the number of labeled nodes, with k ≪ n . the retrieval and prompting process is formalized as follows: ## proposed approach -askgnn we present askgnn, a structure-enhanced framework designed to optimize llms for graph icl. the whole framework is illustrated in figure 3 . this section details the key components: the gnnbased structure-enhanced retriever (section 4.1), llm feedback collection (section 4.2), retriever optimization (section 4.3), and the utilization of the optimized retriever for icl example selection (sectio 4.4). ## limitations we also recognize the following limitations of this work: first, data quality issues, such as lowquality product descriptions, can hinder larger models more than smaller ones, affecting performance on complex datasets. second, the limited input window of current open-source llms capped the number of icl examples at 45, restricting the exploration of larger example sets that might improve performance. last, our approach relies heavily on the learned gnn retriever, meaning inaccurate structure information could reduce overall model effectiveness. ## acknowledgement this research was supported in part by nih r01lm01372201. a baseline details.