O pen G raph: Towards Open Graph Foundation Models (2024)

Abstract: Graph learning has become essential in various domains, including recommendation systems and social network analysis. Graph Neural Networks (GNNs) have emerged as promising techniques for encoding structural information and improving performance in tasks like link prediction and node classification. However, a key challenge remains: the difficulty of generalizing to unseen graph data with different properties. In this work, we propose a novel graph foundation model, called OpenGraph, to address this challenge. Our approach tackles several technical obstacles. Firstly, we enhance data augmentation using a large language model (LLM) to overcome data scarcity in real-world scenarios. Secondly, we introduce a unified graph tokenizer that enables the model to generalize effectively to diverse graph data, even when encountering unseen properties during training. Thirdly, our developed scalable graph transformer captures node-wise dependencies within the global topological context. Extensive experiments validate the effectiveness of our framework. By adapting OpenGraph to new graph characteristics and comprehending diverse graphs, our approach achieves remarkable zero-shot graph learning performance across various settings. We release the model implementation at https://github.com/HKUDS/OpenGraph.

## introduction graph learning is a crucial methodology in various fields, such as recommender systems @xcite , social network analysis @xcite , citation networks @xcite , and transportation networks @xcite . by utilizing graph neural networks (gnns) and recursive message passing, we capture the complex structures of graphs effectively. gnns leverage interdependencies among nodes to incorporate high-* chao huang is the corresponding author. a primary challenge in current end-to-end graph neural networks is their heavy reliance on scarce and low-quality labeled data @xcite . to overcome this, self-supervised learning (ssl) has emerged as a solution by leveraging augmented self-supervision signals. contrastive ssl, exemplified by dgi @xcite and graphcl @xcite , incorporates contrastive objectives as self-supervised alignment loss. recent advancements like joao @xcite and gca @xcite automate contrastive learning through adaptive augmentation. by integrating ssl techniques, we enhance graph neural networks with limited labeled data. graph pre-training excels at capturing intrinsic graph properties but struggle to effectively generalize to diverse downstream domains, particularly when faced with distribution shifts @xcite @xcite . for example, in recommender systems, handling previously unseen user interaction graphs in cold-start recommendation scenarios is crucial @xcite . transferring knowledge from pre-trained graph domains to other downstream domains is desirable @xcite . however, applying these models to unseen graphs results in significant performance deterioration due to variations in node sets and relation semantics across different scenarios. recent research explores prompt-tuning as a task-specific alternative to fine-tuning, bridging the gap between pre-training and downstream objectives @xcite @xcite . these approaches align the pre-trained model's understanding with specific task requirements. however, practical scenarios involve variations in node sets and feature semantics across diverse downstream graph domains. further exploration is needed to enhance graph models' generalization and adaptability to real-world graphs. this work aims to develop a scalable graph model that enables zero-shot learning, effectively making accurate predictions on unseen graphs. building such a model poses significant challenges. nodes in large-scale graphs have complex dependencies. understanding local and global interdependencies among all nodes is crucial for accurate prediction. efficient node-wise dependency encoding is vital to enhance the performance and scalability of graph models. present work. to overcome the challenges, we introduce a graph model for zero-shot learning that captures universal and transferable structural patterns across multiple domains. to address c1, we propose combining large language models (llms) with data augmentation techniques for synthetic graph generation. by generating augmented graphs resembling real-world instances, we enhance the pre-training process of opengraph and gain a deeper understanding of downstream task contexts. this is achieved through the integration of tree-ofprompt regularization with gibbs sampling. to address c2, we propose a topology-aware graph tokenizer that generates universal graph tokens from arbitrary graphs. for c3, we develop a scalable graph transformer with efficient selfattention using anchor sampling. our approach ensures computational efficiency through a twostage self-attention process and optimizes training by leveraging token sequence sampling, reducing sequence length while preserving crucial context. we conducted extensive experiments on diverse datasets, demonstrating the remarkable generalization abilities of our model across various settings. ## preliminaries and node attributes @xmath0. graph learning aims to produce node representations that encode both structural and attribute information. these embeddings are used for tasks such as link prediction and node classification, involving the prediction of node connections and categories, respectively. the corresponding losses to be minimized are: this objective is to develop a universal graph modeling architecture f (•). its parameters θ f are learned by minimizng graph learning losses l on the training graphs {g s }. notably, the training graphs and test graphs g t have no common nodes, edges, or node features. this presents a unique challenge for the graph model to handle the significant distribution shift that occurs across different graph domains with entirely distinct datasets. ## methodology this section presents the design details of the proposed opengraph framework. figure 1 gives an overall illustration for the model. in appendix, we ## conclusion this research aims to develop an adaptable framework for capturing complex topological patterns in diverse graph structures. our model demonstrates exceptional generalization capabilities in zero-shot graph learning tasks across various applications. we utilize a scalable graph transformer architecture and llm-enhanced data augmentation for efficiency and robustness. extensive experiments on benchmark datasets validate our model's performance. future plans include incorporating counterfactual learning to discover noisy connections and influential structures while learning universal and transferable structural patterns in diverse graphs.