M eta F ill: Text Infilling for Meta-Path Generation on Heterogeneous Information Networks (2022)

Abstract: Heterogeneous information network (HIN) is essential to study complicated networks containing multiple edge types and node types. Meta-path, a sequence of node types and edge types, is the core technique to embed HINs. Since manually curating meta-paths is time-consuming, there is a pressing need to develop automated meta-path generation approaches. Existing meta-path generation approaches cannot fully exploit the rich textual information in HINs, such as node names and edge type names. To address this problem, we propose MetaFill, a text-infilling-based approach for meta-path generation. The key idea of MetaFill is to formulate meta-path identification problem as a word sequence infilling problem, which can be advanced by pretrained language models (PLMs). We observed the superior performance of MetaFill against existing meta-path generation methods and graph embedding methods that do not leverage meta-paths in both link prediction and node classification on two real-world HIN datasets. We further demonstrated how MetaFill can accurately classify edges in the zero-shot setting, where existing approaches cannot generate any meta-paths. MetaFill exploits PLMs to generate meta-paths for graph embedding, opening up new avenues for language model applications in graph analysis.

## introduction heterogeneous information network (hin) is an effective framework to model complicated realworld network data @xcite @xcite @xcite @xcite . in contrast to a conventional network @xcite @xcite @xcite , hin supports multiple node types and edge types, thus facilitating the integrative analysis of multiple datasets @xcite @xcite . one of the most important applications on hin is to discover interactions between different node types by framing it as a link prediction problem @xcite @xcite . link prediction is particularly challenging on heterogeneous and long-distance node pairs, which often do not share any neighbors, thus presenting substantial false-negative predictions by conventional networkbased approaches @xcite @xcite . hin exploits meta-paths to address link prediction, especially for heterogeneous and longdistance node pairs. a meta-path is a sequence of node types and edge types. a good meta-path often consists of paths that frequently appear in a hin, thus guiding the hin to focus on these important paths in a large network. since manually curating meta-paths requires domain expertise, automated meta-path generation approaches have become essential for link prediction on hins @xcite @xcite @xcite @xcite @xcite . despite their sophisticated design to leverage the network topological features, existing approaches largely overlook the rich textual information on nodes and edges. in fact, real-world hins contain rich textual information @xcite @xcite , such as node name, node type name and edge type name, which are often the key evidence for human experts to curate meta-paths. in this paper, we propose to identify meta-paths using pretrained language models (plms) @xcite @xcite @xcite in order to explicitly utilize the rich textual information in hins. the key idea of our method metafill is to form the meta-path identification problem as a text infilling problem @xcite . in effect, this converts a graph-based approach to an nlp problem, enabling us to enjoy a variety of new techniques developed along with plms. specifically, metafill samples many paths from the hin according to the plm-based probability for a word sequence consisting of node names and edge type names on that path, then aggregates these paths into meta-paths by a novel context-aware node type classifier. we evaluated our method on two large-scale hins and observed substantial improvement on link predictions compared to existing meta-path generation approaches under two meta-path-based link prediction frameworks. furthermore, we found that the improvement of our method is larger with the decreasing of training data size, indicating the ability of compensating data sparsity using textual information. in addition to link prediction, our method also achieved prominent results on node classification. collectively, we demonstrate how language model can be used to accurately generate meta-paths on hins, opening up new venues for heterogeneous graph analysis using language models. ## preliminaries heterogeneous information network (hin) is a network that contains multiple node types and edge types @xcite . let @xmath0, e) be a hin, where @xmath1. each node @xmath2. each edge @xmath3. there are three kinds of textual information in most hins. 1) node name: the textual description of a node v (e.g., breast cancer). 2) node type name: the textual description of a node type a (e.g., disease). 3) edge type name: the textual description of an edge type r (e.g., treated by). some hins might also have edge names. while we do not consider edge names in this paper, they can be easily incorporated into our framework. most of the conventional hin embedding approaches do not fully exploit this rich textual information. we aim to use language models to incorporate textual information into hin modeling. meta-path is one of the most effective techniques for embedding hins through explicitly modeling heterogeneous and long-distance semantic similarity @xcite @xcite ). an l-hop meta-path ω is defined as a sequence a then p is a path instance of ω, if a i is the node type of v i and r i is the edge type of e i . hin embedding is often performed through repeatedly sampling a path in the graph and then optimizing the embedding of each node along this path. meta-paths could be used to record prior knowledge and then encourage the sampling process to focus on the path that is an instance of curated meta-paths. ## hypothesis validation our hypothesis is that plms can be used to identify important meta-paths based on the textual information along each path. since plms have shown to contain a great amount of real-world knowledge @xcite , they might also be used to extract meta-paths similar to expert curations. we sought to validate this hypothesis using a widely-used hin dataset nell @xcite . in particular, we randomly sampled 1000 2-hop paths in nell. for each path, we calculated a plm-based similarity score, a name-based similarity score and a connectivity-based similarity score. the plm-based similarity score concatenated node names and edge type names along a given path as a word sequence and then obtained a probability for this sequence using gpt-2. the name-based similarity score calculated the textual similarity between the node names of the starting node and the end node using gpt-2 embeddings. the connectivity-based similarity score is 1 if two nodes are connected and 0 otherwise. we first compared plm-based similarity score and name-based similarity score and observed a substantial agreement of spearman correlation 0.31 (fig. 1 a ). this indicates that plm is able to find nodes with similar node names. next, we found that plm-based similarity score is also highly predictive of the connectivity-based similarity score (fig. 1 b ), demonstrating the possibility to predict missing links using a plm. collectively, the language model probability of a path is predictive of nodes similarity and connectivity, suggesting the opportunity to find meta-paths using plms. ## methodology the key idea of our approach is to form meta-path identification as a text infilling problem @xcite and then exploit plms to fill in the node names and edge type names that best reflect the graph information. these word sequences then form paths on the hin and are then aggregated into meta-paths using a context-aware node type classifier (fig. 2 ). ## conclusion and future work in this paper, we have proposed a novel textinfilling-based meta-path generation method. the key idea of our method is to form meta-path generation as a text infilling problem and sample important paths from a hin by using plms to generate sequences of node names and edge type names. we have evaluated our method on two datasets under two meta-path-based hin embedding frameworks, and obtained the best performance on node classification and link prediction. the improvement of our method is also consistent in low-resource settings. in addition, we found that fine-tuning the plm using a hin can further improve word embeddings, again indicating how our method creates an synergistic effort between hins and plms. to the best of our knowledge, our method is the first attempt to apply plms to meta-path generation. since plms are constructed on large-scale real-word text corpus, they often contain rich realworld knowledge. we envision that our method will motivate future research in investigating how plms can be used to advance other graph analysis problems.