Are Message Passing Neural Networks Really Helpful for Knowledge Graph Completion? (2023)

Abstract: Knowledge graphs (KGs) facilitate a wide variety of applications. Despite great efforts in creation and maintenance, even the largest KGs are far from complete. Hence, KG completion (KGC) has become one of the most crucial tasks for KG research. Recently, considerable literature in this space has centered around the use of Message Passing (Graph) Neural Networks (MPNNs), to learn powerful embeddings. The success of these methods is naturally attributed to the use of MPNNs over simpler multi-layer perceptron (MLP) models, given their additional message passing (MP) component. In this work, we find that surprisingly, simple MLP models are able to achieve comparable performance to MPNNs, suggesting that MP may not be as crucial as previously believed. With further exploration, we show careful scoring function and loss function design has a much stronger influence on KGC model performance. This suggests a conflation of scoring function design, loss function design, and MP in prior work, with promising insights regarding the scalability of state-of-the-art KGC methods today, as well as careful attention to more suitable MP designs for KGC tasks tomorrow.

## introduction knowledge graphs (kgs) @xcite are a type of knowledge base, which store multi-relational factual knowledge in the form of triplets. each triplet specifies the relation between a head and a tail entity. kgs conveniently capture rich structured knowledge about many types of entities (e.g. objects, events, concepts) and thus facilitate numerous applications such as information retrieval @xcite , recommender systems @xcite , and * corresponding author question answering @xcite . to this end, the adopted kgs are expected to be as comprehensive as possible to provide all kinds of required knowledge. however, existing large-scale kgs are known to be far from complete with large portions of triplets missing @xcite . imputing these missing triplets is of great importance. furthermore, new knowledge (triplets) is constantly emerging even between existing entities, which also calls for dedicated efforts to predict these new triplets @xcite . therefore, knowledge graph completion (kgc) is a problem of paramount importance @xcite . a crucial step towards better kgc performance is to learn low-dimensional continuous embeddings for both entities and relations @xcite . recently, due to the intrinsic graph-structure of kgs, graph neural networks (gnns) have been adopted to learning more powerful embeddings for their entities and relations, and thus facilitate the kgc. there are mainly two types of gnn-based kgc methods: message passing neural networks (mpnns) @xcite and path-based methods @xcite @xcite . in this work, we focus on mpnn-based models, which update node features through a message passing (mp) process over the graph where each node collects and transforms features from its neighbors. when adopting mpnns for kgs, dedicated efforts are often devoted to developing more sophisticated mp processes that are customized for better capturing multi-relational information @xcite @xcite . the improvement brought by mpnn-based models is thus naturally attributed to these enhanced mp processes. therefore, current research on developing better mpnns for kgs is still largely focused on advancing mp processes. present work. in this work, we find that, sur-prisingly, the mp in the mpnn-based models is not the most critical reason for reported performance improvements for kgc. specifically, we replaced mp in several state-of-the-art kgc-focused mpnn models such as rgcn @xcite , compgcn @xcite and kb-gat @xcite with simple multiple layer perceptrons (mlps) and achieved comparable performance to their corresponding mpnnbased models, across a variety of datasets and implementations. we carefully scrutinized these mpnn-based models and discovered they also differ from each other in other key components such as scoring functions and loss functions. to better study how these components contribute to the model, we conducted comprehensive experiments to demonstrate the effectiveness of each component. our results indicate that the scoring and loss functions have stronger influence while mp makes almost no contributions. based on our findings, we develop ensemble models built upon mlps, which are able to achieve better performance than mpnnbased models; these implications are powerful in practice, given scalability advantages of mlps over mpnns @xcite . ## preliminaries before moving to main content, we first introduce kgc-related preliminaries, five datasets and three mpnn-based models we adopt for investigations. ## kgc without message passing it is well known that mp is the key bottleneck for scaling mpnns to large graphs @xcite @xcite . observation 1 suggests that the mp may be not helpful for kgc. thus, in this section, we investigate if we can develop mlp-based methods (without mp) for kgc that can achieve comparable or even better performance than existing mpnn methods. compared with the mpnn models, mlp-based methods enjoy the advantage of being more efficient during training and inference, as they do not involve expensive mp operations. we present the time complexity in appendix h. the scoring and loss functions play an important role in mpnnbased methods. likewise, we next study the impact of the scoring and loss functions on mlp-based methods. ## related work there are mainly two types of gnn-based kgc models: mpnn-based models and path-based models. when adopting mpnns for kg, recent efforts have been made to deal with the multirelational edges in kgs by designing mp operations. rgcn @xcite introduces the relation-specific transformation matrices. compgcn @xcite integrates neighboring information based on entity-relation composition operations. kbgat @xcite learns attention coefficients to distinguish the role of entity in various relations. path-based models learn pair-wise representations by aggregating the path information between the two nodes. nbfnet @xcite integrates the information from all paths between the two nodes. red-gnn @xcite makes use of the dynamic programming and a * net @xcite prunes paths by prioritizing important nodes and edges. in this paper, we focus on investigating how the mp component in the mpnns affects their performance in the kgc task. hence, we do not include these path-based models into the comparison. a concurrent work @xcite has similar observations as ours. however, they majorly focus on exploring how the mp component affects the performance. our work provides a more thorough analysis on the major contributors for mpnn-based kgc models and proposes a strong ensemble model based upon the analysis. ## conclusion in this paper, we surprisingly find that the mlpbased models are able to achieve competitive performance compared with three mpnn-based models (i.e., compgcn, rgcn and kbgat) across a variety of datasets. it suggests that the message passing operation in these models is not the key component to achieve strong performance. to explore which components potentially contribute to the model performance, we conduct extensive experiments on other key components such as scoring function and loss function. we found both of them play crucial roles, and their impact varies significantly across datasets. based on these findings, we further propose ensemble methods built upon mlp-based models, which are able to achieve even better performance than mpnn-based models.