Patton: Language Model Pretraining on Text-Rich Networks (2023)

Abstract: A real-world text corpus sometimes comprises not only text documents, but also semantic links between them (e.g., academic papers in a bibliographic network are linked by citations and co-authorships).Text documents and semantic connections form a text-rich network, which empowers a wide range of downstream tasks such as classification and retrieval. However, pretraining methods for such structures are still lacking, making it difficult to build one generic model that can be adapted to various tasks on text-rich networks. Current pretraining objectives, such as masked language modeling, purely model texts and do not take inter-document structure information into consideration. To this end, we propose our PretrAining on TexT-Rich NetwOrk framework Patton.Patton includes two pretraining strategies: network-contextualized masked language modeling and masked node prediction, to capture the inherent dependency between textual attributes and network structure. We conduct experiments on four downstream tasks in five datasets from both academic and e-commerce domains, where Patton outperforms baselines significantly and consistently.

## introduction texts in the real world are often interconnected through links that can indicate their semantic relationships. for example, papers connected through citation links tend to be of similar topics; e-commerce items connected through co-viewed links usually have related functions. the texts and links together form a type of network called a textrich network, where documents are represented as nodes, and the edges reflect the links among documents. given a text-rich network, people are usually interested in various downstream tasks (e.g., document/node classification, document retrieval, and link prediction) (zhang et al., 2019; wang et al., 1 code is available at https://github.com/ petergriffinjin/patton 2019; @xcite . for example, given a computer science academic network as context, it is intuitively appealing to automatically classify each paper @xcite , find the authors of a new paper @xcite , and provide paper recommendations @xcite . in such cases, pretraining a language model on a given text-rich network which can benefit a great number of downstream tasks inside this given network is highly demanded @xcite . while there have been abundant studies on building generic pretrained language models @xcite @xcite , they are mostly designed for modeling texts exclusively, and do not consider inter-document structures. along another line of research, various network-based pretraining strategies are proposed in the graph learning domain to take into account structure information @xcite . yet, they focus on pretraining graph neural networks rather than language models and cannot easily model the rich textual semantic information in the networks. to empower language model pretraining with network signals, linkbert @xcite is a pioneering study that puts two linked text segments together during pretraining so that they can serve as the context of each other. however, it simplifies the complex network structure into node pairs and does not model higher-order signals @xcite . overall, both existing language model pretraining methods and graph pretraining methods fail to capture the rich contextualized textual semantic information hidden inside the complex network structure. to effectively extract the contextualized semantics information, we propose to view the knowledge encoded inside the complex network structure from two perspectives: token-level and document-level. at the token level, neighboring documents can help facilitate the understanding of tokens. for example, in figure 1 , based on the text information of neigh- experience the ultimate in smoothness and creaminess with dove ferrero rocher collection, fine hazelnut milk chocolates … … … soap bar? chocolate? … h hershey's assorted chocolate candy mix olay moisture coconut oasis body wash l'oreal paris elvive repairing shampoo dove's smooth formula will leave you feeling refreshed figure 1 : an illustration of a text-rich network (a product item co-viewed network). at the token level, from network neighbors, we can know that the "dove" at the top is a personal care brand and the "dove" at the bottom is a chocolate brand. at the document level, referring to the edge in the middle, we can learn that the chocolate from "hershey's" should have some similarity with the chocolate from "ferrero". absorbing such two-level hints in pretraining can help language models produce more effective representations which can be generalized to various downstream tasks. to this end, we propose patton, a method to continuously pretrain language models on a given text-rich network. the key idea of patton is to leverage both textual information and network structure information to consolidate the pretrained language model's ability to understand tokens and documents. building on this idea, we propose two pretraining strategies: 1) network-contextualized masked language modeling: we randomly mask several tokens within each node and train the language model to predict those masked tokens based on both in-node tokens and network neighbors' tokens. 2) masked node prediction: we randomly mask some nodes inside the network and train the language model to correctly identify the masked nodes based on the neighbors' textual information. we evaluate patton on both academic domain networks and e-commerce domain networks. to comprehensively understand how the proposed pretraining strategies can influence different downstream tasks, we conduct experiments on classification, retrieval, reranking, and link prediction. in summary, our contributions are as follows: ## preliminaries definition 2.1. text-rich networks @xcite . a text-rich network can be denoted as @xmath0, e, d), where v, e and d are node set, edge set, and text set, respectively. each v @xmath1. for example, in an academic citation network, @xmath2, @xmath3, and @xmath4. in this paper, we mainly focus on networks where the edges can provide semantic correlation between texts (nodes). for example, in a citation network, connected papers (cited papers) are likely to be semantically similar. ## attention map study we conduct a case study by showing some attention maps of patton and the model without pretraining on four downstream tasks on sports. we randomly pick a token from a random sample and plot the self-attention probability of how different tokens (xaxis), including neighbor virtual token @xcite ) and the first eight original text tokens ([tk_x]), will contribute to the encoding of this random token in different layers (y-axis). the result is shown in figure 4 . from the result, we can find that the neighbor virtual token is more deactivated for the model without pretraining, which means that the information from neighbors is not fully utilized during encoding. however, the neighbor virtual token becomes more activated after pretraining, bringing more useful information from neighbors to enhance center node text encoding. ## conclusions in this work, we introduce patton, a method to pretrain language models on text-rich networks. patton consists of two objectives: (1) a networkcontextualized mlm pretraining objective and (2) a masked node prediction objective, to capture the rich semantics information hidden inside the complex network structure. we conduct experiment on four downstream tasks and five datasets from two different domains, where patton outperforms baselines significantly and consistently.