Unleashing the Power of Language Models in Text-Attributed Graph (2023)

Abstract: Representation learning on graph has been demonstrated to be a powerful tool for solving real-world problems. Text-attributed graph carries both semantic and structural information among different types of graphs. Existing works have paved the way for knowledge extraction of this type of data by leveraging language models or graph neural networks or combination of them. However, these works suffer from issues like underutilization of relationships between nodes or words or unaffordable memory cost. In this paper, we propose a Node Representation Update Pre-training Architecture based on Co-modeling Text and Graph (NRUP). In NRUP, we construct a hierarchical text-attributed graph that incorporates both original nodes and word nodes. Meanwhile, we apply four self-supervised tasks for different level of constructed graph. We further design the pre-training framework to update the features of nodes during training epochs. We conduct the experiment on the benchmark dataset ogbn-arxiv. Our method achieves outperformance compared to baselines, fully demonstrating its validity and generalization.

## introduction text-attributed graphs, characterized by the association of nodes with text attributes @xcite , are prevalent in diverse real-world contexts. for instance, in paper citation networks, each paper is accompanied by textual content, while in social networks, each user can be described through a text description. the investigation of learning techniques on text-attributed graphs has garnered considerable attention in domains such as graph learning, information retrieval, and natural language processing, reflecting the growing importance of understanding and analyzing textual information within the context of graph-based structures. existing research on learning from textattributed graph mainly falls into three lines: (1)language models(lms) only, in these works, lms @xcite are applied to leverage the local textual information of individual nodes and generate representation for them @xcite . ## method in this section, we present the entire training framework to learn paper and word representations simultaneously without supervision based on r-gnns, including the modeling method of text-attributed graph, self-supervised tasks and pre-training architecture, see overall framework in figure 2 . ## experiment setup in this section, we apply our entire pre-training method to a real-world citation network ogbnarxiv @xcite and report performance on the downstream tasks. to demonstrate the generalization of our method, we considered two settings on the same dataset: transductive learning and inductive learning. meanwhile, we select several baseline models to prove the validity of our method. ## conclusion in this work, we propose to learn representations of papers and words simultaneously via co-modeling the raw text and graph based on hierarchical textattributed graph. we design a pre-training framework and corresponding self-supervised tasks for this scenario. sufficient experiments conducted on the benchmark dataset ogbn-arxiv demonstrate the effectiveness and generality of our method.