GRENADE : Graph-Centric Language Model for Self-Supervised Representation Learning on Text-Attributed Graphs (2023)

Abstract: Self-supervised representation learning on text-attributed graphs, which aims to create expressive and generalizable representations for various downstream tasks, has received increasing research attention lately. However, existing methods either struggle to capture the full extent of structural context information or rely on task-specific training labels, which largely hampers their effectiveness and generalizability in practice. To solve the problem of self-supervised representation learning on text-attributed graphs, we develop a novel Graph-Centric Language model – GRENADE. Specifically, GRENADE harnesses the synergy of both pre-trained language model and graph neural network by optimizing with two specialized self-supervised learning algorithms: graph-centric contrastive learning and graph-centric knowledge alignment. The proposed graph-centric self-supervised learning algorithms effectively help GRENADE to capture informative textual semantics as well as structural context information on text-attributed graphs. Through extensive experiments, GRENADE shows its superiority over state-of-the-art methods.

## introduction text-attributed graph (tag) @xcite (a.k.a., textual graph) has been widely used for modeling a variety of real-world applications, such as information retrieval @xcite , product recommendation @xcite and many more. in tag, each node represents a text document, while the relationships among these text nodes are depicted by the edges. for instance, in citation networks, text nodes represent academic papers, and edges are the citation relationship between different papers. to conduct different analytics tasks on tag, the key is to learn expressive node representations for the text nodes. recent research has demonstrated that selfsupervised learning (ssl) can substantially improve the effectiveness of representation learning on text data @xcite @xcite without using human supervision. those methods are commonly learned under the assumption that text documents are independently and identically distributed (i.i.d.), which neglects the structural interdependencies among text nodes on tag. however, the interdependencies between different text documents can provide valuable insights for understanding their semantic relationships. take citation networks as an example, those academic papers (text nodes) that have citation relationships often share similar topics. hence, it is necessary for ssl models to account for not only textual semantics but also structural context information. in fact, self-supervised representation learning on tag remains in its infancy: (i) though recent research efforts @xcite @xcite try to empower pre-trained language models (plm) with structural context information, most of them still stay superficial by designing local structuredependent ssl objectives. for example, both giant @xcite and specter @xcite ) train the language model by inferring the local neighborhood based on representations of text nodes. however, simply relying on those ssl objectives cannot help the plm fully understand complex graph structures, especially compared to models like graph neural networks (gnn) @xcite @xcite ; (ii) meanwhile, another line of research @xcite try to combine the advantages of both plm and gnn by distilling the knowledge from one to the other @xcite and have shown promising results. nonetheless, one major issue is that those methods are task-specific (e.g., semi-supervised node classification) and require human-annotated labels to enable knowledge distillation. such an inherent limitation jeopardizes the versatility of their models for handling different and even unseen downstream tasks, which runs counter to the goal of ssl. to go beyond the existing learning paradigms and capture informative textual semantic and graph structure information, we develop a new model for self-supervised learning on tag, namely grenade (graph-centric language model). grenade is built with a plm encoder along with an adjuvant gnn encoder that provides complementary knowledge for it. more importantly, grenade is learned through two new self-supervised learning algorithms: graph-centric contrastive learning (gc-cl), a structure-aware and augmentation-free contrastive learning algorithm that improves the representation expressiveness by leveraging the inherent graph neighborhood information; and graph-centric knowledge alignment (gc-ka), which enables the plm and gnn modules to reinforce each other by aligning their learned knowledge encoded in the text node representations. specifically, gc-cl enforces neighboring nodes to share similar semantics in the latent space by considering them as positive pairs. even without using data augmentation, gc-cl performs node-wise contrastive learning to elicit the structural context information from tag. in the meantime, gc-ka bridges the knowledge gap between plm and gnn by performing dual-level knowledge alignment on the computed representations: at the node level, we minimize the distance between the representations learned from two encoders that focus on different modalities. at the neighborhood level, we minimize the discrepancy between two neighborhood similarity distributions computed from plm and gnn. by virtue of the two proposed graph-centric selfsupervised learning algorithms, we are able to learn grenade that can generate expressive and generalizable representations for various downstream tasks without using any human supervision. in summary, our work has the following contributions: ## problem definition notations. we utilize bold lowercase letters such as d to represent vectors, bold capital letters like w to denote matrices and calligraphic capital letters like w to represent sets. let @xmath0, d) denote a text-attributed graph with adjacency matrix @xmath1, 1} |d|×|d| and text set d. the a @xmath2. each node i represents a text document which consists of a sequence of tokens note that the whole learning process is performed solely on the input graph g without the utilization of human-annotated labels. ## proposed approach: graph-centric language model (grenade) to learn expressive representations from tag in a self-supervised learning manner, we propose our graph-centric language model grenade, which bridges the knowledge gap between pre-trained language model (plm) and graph neural network (gnn). by optimizing two distinct encoders with a set of novel self-supervised learning algorithms, the plm encoder and gnn encoder mutually reinforce each other, and we can finally derive our graph-centric language model (grenade). gnn. the overall framework is shown in fig. 1 . ## experiment to evaluate the effectiveness of our approach grenade, we conduct comprehensive experiments on different datasets and various downstream tasks. ## conclusion in this paper, we introduce a self-supervised graphcentric language model: grenade, for learning expressive and generalized representation from textual attributed graphs (tag). grenade is learned through two self-supervised learning algorithms: