I NDUCTIVE R ELATION P REDICTION U SING A NALOGY S UBGRAPH E MBEDDINGS
A BSTRACT
Prevailing methods for relation prediction in heterogeneous graphs including knowledge graphs aim at learning the latent representations (i.e., embeddings) of observed nodes and relations, and are thus limited to the transductive setting where the relation types must be known during training.
In this paper, we propose AN alogy Sub G raph E mbedding L earning (GraphANGEL), a novel relation prediction framework that predicts relations between each node pair by checking whether the subgraphs containing the pair are similar to other subgraphs containing the considered relation.
Each graph pattern explicitly represents a speciﬁc logical rule, which contributes to an inductive bias that facilitates generalization to unseen relation types and leads to more explainable predictive models.
Our model consistently outperforms existing models in terms of heterogeneous graph based recommendation as well as knowledge graph completion.
We also empirically demonstrate the capability of our model in generalizing to new relation types while producing explainable heat maps of attention scores across the discovered logics.
I NTRODUCTION
Relation modeling aims to learn the relations between nodes, leading to advances in a wide range of applications, e.g., recommender systems (Koren et al., 2009), knowledge graphs (Bordes et al., 2013), and biology (Yasunaga et al., 2021).
As most relational data in the real world is heterogeneous, a principal way is to organize it into a heterogeneous graph.
The dominant paradigms for relation prediction can be categorized into matrix factorization techniques (Nickel et al., 2011; 2012), statistical relational learning approaches (Richardson & Domingos, 2006; Singla & Domingos, 2005), and neural-embedding-based methods (Bordes et al., 2013; Dettmers et al., 2018).
Among these, neural-embedding-based methods which learn to encode relational information using lowdimensional representations of nodes and relations, have shown good scalability (Bordes et al., 2013) and inductive learning capability (Battaglia et al., 2018) in terms of validating unseen nodes.
Results of such methods show that graph neural networks (GNNs) are able to condense the neighborhood connectivity pattern of each node into a node-speciﬁc low-dimensional embedding and successfully exploit such local connectivity patterns and homophily.
Recent advances further reveal the logical expressiveness of GNNs (Barceló et al., 2019) and support their inductive ability to generalize to unseen nodes (Teru et al., 2020; Zhang & Chen, 2019).
In contrast, a limited study has been conducted (Yang et al., 2014) on the inductive learning capability for unseen relation types.
Such inductive ability, if successfully exploited, can directly improve logical expressiveness of GNNs and enable them to effectively capture the underlying logical semantics (e.g., logical rules).
Note that this is more challenging than unseen nodes, since it is usually hard to deﬁne the “neighborhood” of a relation type topologically.
In this paper, we propose AN alogy Sub G raph E mbedding L earning (GraphANGEL), a new relation prediction paradigm that holds a strong inductive bias to generalize to unseen relation types.
Given a pair of nodes to predict the existence of a speciﬁc relation between them, the core idea is to extract some analogy subgraphs containing the pair, and compare them against other subgraphs sharing similar shapes.
We call these shapes graph patterns .
Taking Figure 1 as an example, the task is to predict whether Person E lives in London .
We construct three patterns involving the relation live .
The ﬁrst is target pattern containing the source and target nodes, i.e., Person E and London , for which we are to predict the existence of the relation live .
The second is supporting pattern that includes live as evidence supporting the existence.
The third is refuting pattern that does not include live as a baseline for comparison.
If an edge does exist between the two nodes, the subgraphs matching the target pattern should be more similar to those matching supporting patterns than the refuting patterns.
Following the above intuition, we ﬁnd a set of subgraphs that match each of the patterns.
Then we compare the ﬁrst set against the set matching supporting patterns, and also against the set matching refuting patterns, using a neural network.
As shown in the bottom part of the ﬁgure, the subgraphs in the second set share the higher similarity with the one in the ﬁrst set, so the prediction result in this case is that there exists a live .
Given a triplet (cid:104) s, r, t (cid:105) , GraphANGEL consists of the following stages: (1) determining target patterns from s, t as well as supporting and refuting patterns from r , (2) retrieving subgraphs matching each pattern, (3) computing the representations of each set and then the similarity between the subgraph set matching target pattern and the set matching supporting/refuting patterns, and the ﬁnal prediction based on the similarities.
For the ﬁrst stage, our architecture design only involves the graph patterns in pair, 3-cycle, and 4-cycle shapes, which is efﬁcient to match and already shows good results.
For the second stage, we introduce efﬁcient searching and sampling techniques to ﬁnd the subgraphs matching the patterns.
We use a GNN with attention in the third stage to combine the sets of subgraphs as well as their node features.
The attention module can simultaneously produce an explainable heat map across the discovered patterns.
Notably, none of the above stages requires explicitly learning a representation of r , the relation we are predicting, thus, GraphANGEL naturally generalizes to modeling unseen relation types.
We benchmark GraphANGEL on heterogeneous graph based recommendation and knowledge graph completion tasks with the state-of-the-art methods.
For the evaluation of inductive capabilities, we construct several new inductive benchmarks by either removing or adding relations from knowledge graph datasets.
Extensive experimental comparisons on these benchmarks exhibit the superiority of our method under both transductive and inductive settings.
B RIDGING L OGICAL E XPRESSIONS AND G RAPH P ATTERNS
We begin with bridging logical expressions and graph patterns based on two intuitions/assumptions:
• A relation can often be inferred from other relations with a combination of simple logical rules that do not involve too many nodes.
• One can predict relation existence by ﬁnding whether the subgraphs containing the pair are similar to the subgraphs containing an edge with the same relation.
Take Figure 1 as an example where we wish to predict whether Person E lives in London .
An example of the ﬁrst intuition goes as follows: if Person E watches the soccer matches with club Arsenal , which is based in London , then Person E may also live in London .
However, such logical rules (so called chain-like logic rules in (Yang et al., 2017)) may not always hold, requiring probabilistic inference.
For instance, Person G watches the matches with FC
Barcelona , which is Based in Barcelona , but he does not live in the same city.
To make more accurate predictions, we need to combine other possible logical rules, such as “Friends often live in the same city” - in this case, Person E and Person H are friends , and Person H lives in London (some of them construct certain patterns are called tree-like logic rules (Yang & Song, 2019) or graph-like rules (Shi et al., 2020)).
The second intuition tells us that we can predict whether Person E lives in London by checking if the subgraphs containing the pair ( Person E , London ) (named target subgraphs, e.g. Person E - Arsenal - London ) are similar to the subgraphs containing a live relation (named supporting subgraphs, e.g. Person I - Chelsea - London ).
Similarity computation is done by a neural network.
To determine whether the two sets of subgraphs are similar, we additionally compare the target subgraphs against another set of subgraphs that do not contain the relation (named refuting subgraphs, e.g. Person E - Person G - Barcelona ) as a baseline.
A variety of options exist for selecting the refuting subgraphs, ranging from selecting those having the same topology regardless of the actual relation types, to those having both the same shape and relation types.
In addition, we do not include the edge of live in the supporting subgraphs to avoid information leakage.
We also require the supporting subgraphs and refuting subgraphs to have the same shape as that of the target subgraphs in order to make similarity computation focus more on the node features and relation types rather than the topology, and that is why we call both supporting subgraphs and refuting subgraphs analogy subgraphs.
This enables us to predict live relation with the pipeline above without learning an explicit embedding for live relation, unlike prior works.
The reason is that the information of live relation can be implicitly expressed by the other relations found in the supporting and refuting subgraphs, e.g. live can be expressed by watch and based .
This serves as the basis for our model’s generalizability to relations that have no occurrence in the training set.
We can describe what kind of subgraphs we are looking for in the above example with logical expressions.
The target subgraphs above will have the logical expression Source ( x ) ∧ Target ( y ) ∧
Edge ( x, z ) ∧ Edge ( z, y ) , where Source ( x ) means if node x is the source node s , Target ( y ) means if node y is the target node t , and Edge ( x, z ) means if there exists an edge regardless of relation type between x and z .
Our task is to determine whether an edge of relation type r exists between s and t .
The supporting subgraphs will have Edge ( x, z )
∧ Edge ( z, y )
∧ Edge r ( x, y ) , meaning that the subgraph can be any 3-cycle except that there must be an edge with relation type r .
The refuting subgraphs will have Edge ( x, z )
∧ Edge ( z, y ) ∧
¬ Edge r ( x, y ) , meaning that it can be any 2-path except that the starting node and ending node must not have an edge with relation r in between.
This leads to the concept of graph patterns , deﬁned as follows:
Deﬁnition 1 (Graph Pattern) .
A graph pattern is a logical function that takes in a subgraph as input and returns a boolean value, consisting of logical operators ( ¬ , ∧ , ∨ ) as well as indicator operators determined by the existence of nodes and edges, the latter includes:
• Source ( x ) and Target ( x ) , returning true iff x is the source node and target node, respectively.
• Edge ( x, y ) , returning true iff there exists an edge between node x and y .
• Edge r ( x, y ) , returning true iff there exists an edge of relation r between node x and y .
We call a subgraph S matches a graph pattern Π if true is returned when S is applied to Π .
Such operation is already well supported in graph databases (Francis et al., 2018), and in the following sections we give more efﬁcient algorithms that perform matching with simple patterns. [Table]
M ODELING R ELATION WITH A NALOGY S UBGRAPH E MBEDDINGS
For each graph G =
( V , E , R ) where V denotes node set, E denotes edge set and R is relation set, as Figure 2 shows, we outline how GraphANGEL works for each triplet (cid:104) s, r, t (cid:105) to predict the existence for the edge of type r connecting source node s and target node t as follows.
(a) We start by determining P target patterns denoted as Π ∗ 1 , · · · , Π ∗ P .
For each Π ∗ p we also determine its corresponding supporting pattern Π + p and refuting pattern Π − p .
(b) We then sample a set of
K target subgraphs {S ∗ p,k } Kk =1 matching Π ∗ p , Q supporting subgraphs {S + p,q } Qq =1 matching Π + p , and Q refuting subgraphs {S − p,q } Qq =1 matching Π − p .
(c) For each pattern Π p , we next compute the representation of each sampled subgraph and obtain the set of target subgraph embeddings { e ∗ p,k } Kk =1 , supporting subgraph embeddings { e + p,q } Qq =1 and refuting subgraph embeddings { e − p,q } Qq =1 .
(d) We ﬁnally compute a similarity score between the set { e ∗ p,k } P,Kp =1 ,k =1 and the set { e + p,q } P,Qp =1 ,q =1 , as well as { e ∗ p,k } P,Kp =1 ,k =1 and { e − p,q } P,Q p =1 ,q =1 , to get the relation prediction result.
We summarize the notations in Appendix A1, show the training algorithm in Algorithm 1, describe the details in the following subsections, and provide the overall time complexity in Appendix A3.
Input: Graph G , Patterns Π 1 , . .
, Π P .
for each tuple (cid:104) s, r, t (cid:105) do for each pattern Π p do
Construct Π ∗ p , Π + p , Π − p .
Retrieve K subgraphs {S ∗ p,k } Kk =1 matching Π ∗ p .
Retrieve Q subgraphs {S + p,q } Qq =1 matching Π + p .
Retrieve Q subgraphs {S − p,q } Qq =1 matching Π − p .
Compute e ∗ p,k , e + p,q , e − p,q via Eq.
end
Compute s + , s − via Eq.
Update parameters according to Eq.
end
Pattern Construction.
During training, we set for each Π p the target pattern Π ∗ p , the supporting pattern Π + p , and the refuting pattern Π − p using
Table 1.
Since pattern matching is
NP-complete like subgraph matching (Lewis, 1983), we ﬁrst designate before training a set of patterns { Π p } Pp = whose pattern matching can be computed in manageable time.
In practical, we use Pairs , 3-cycles , and 4-cycles .
Subgraph Retrieval.
Although general graph pattern matching is supported in graph databases (Francis et al., 2018), more efﬁcient solutions exist for simpler graph patterns, especially when we only consider pairs, 3-cycle and 4-cycle shapes.
Searching and retrieving the subgraphs matching patterns in Pair shape is trivial since it reduces to ﬁnding edges with a given relation, so the following discussion only involves 3-cycle and 4-cycle patterns.
We pre-compute all subgraphs matching 3-cycle patterns and uniformly sample a number of subgraphs matching 4-cycle patterns, and store them into a buffer by our algorithms, summarized as follows, with pseudocode, correctness proofs and complexity analysis in Appendix A2.
• 3-cycles .
We ﬁrst partition the node set by the node degrees into two sets V = V 1 ∪V 2 .
V 1 contains the nodes whose degrees are less than |E| 12 , and V 2 contains the rest.
For each u, v, w ∈ V 1 , we check if they form a 3-cycle.
Then, for each u ∈ V 2 , we enumerate all pairs of its neighbors and see if they are connected.
The complexity to ﬁnd all the subgraphs is O ( |E| 32 ) .
• 4-cycles .
For each node, we ﬁnd all 2-paths ( u, v, w ) whose starting node u has the largest degree, i.e., d u = max
( d u , d v , d w ) , and then store them in T uw .
Each pair of 2-paths that shares the same ending node forms a 4-cycle.
In this regard, 4-cycles can be uniformly sampled by ﬁrst sampling u, w with the probability in proportion to |T uw | · |T uw − 1 | , and then choosing two distinct nodes v, x from T uw .
The complexity to sample n 4 − cycle subgraphs is O (max( n 4 − cycle , |E| 3 2 )) .
Then we perform pattern matching for 3-cycles and 4-cycles for each subgraph as follows
• Target patterns .
Matching 3-cycle target patterns reduces to ﬁnding common neighbors of s and t , which takes O ( d s + d t ) time where d s and d t are degrees of s and t .
Matching 4-cycle target pattern reduces to ﬁnding whether the neighbors of s and t are connected, taking O ( d s d t ) time.
[Table]
• Supporting patterns .
Matching supporting patterns reduces to ﬁnding a subgraph containing an edge of type r in the precomputed result.
One can efﬁciently retrieve with a precomputed inverted map with relation r as key and the actual subgraphs containing it as value.
• Refuting patterns .
Matching refuting patterns reduces to random walks, followed by checking whether the starting node and the ending node has an edge of type r .
We further design a series of novel uniform sampling algorithms such that the time complexity of sampling n refuting cases of Π 3 − cycle or Π 4 − cycle reduces to O ( |V| + |E| + n ) .
See detailed descriptions in Appendix A2.
Representation Computation.
We apply a neural network Φ( · ) over each subgraph S ∗ p,k , S + p,q and
S − p,q to obtain graph-level representations e ∗ p,k , e + p,q , e − p,q , following
[Equation]
In the implementation, we adopt single layer R-GCN (Schlichtkrull et al., 2018) followed by any readout function, e.g., Mean ( · ) , Max ( · ) as Φ( · ) .
We also empirically study the effect of using other
GNNs with different number of propagation layers as Φ( · ) in Appendix A6.5.
Similarity Computation.
We deploy a neural network Ψ( · ) to measure the similarity s + between the set of subgraphs matching Π ∗ 1 , . . .
, Π ∗ P and the set of subgraphs matching Π +1 , . .
, Π + P ; and the similarity s − between the set of subgraphs matching Π ∗ 1 , . . .
, Π ∗ P and the set of subgraphs matching Π − 1 , . .
, Π − P , which can be formulated as
[Equation]
There are many choices of measuring the similarity between two sets.
In the implementation, we adopt a co-attention mechanism (Lu et al., 2016) as Ψ( · ) , because even within the same pattern, the subgraphs having similar node features are more important than others.
We put the concrete formulation of Φ( · ) and Ψ( · ) in the Appendix A3.3 and A3.4.
Loss Function.
For each tuple (cid:104) s, r, t (cid:105) , we have the binary label y in the training dataset D to denote the relation existence.
We here train the model by logistic loss with negative sampling:
[Equation]
where ˆ y is the ﬁnal prediction calculated with normalized similarity as ˆ y =
[Equation]
Inference.
We perform pattern matching to ﬁnd the target, supporting, and refuting subgraphs on the testing graph, and compute the prediction score
ˆ y exactly as what we do in training.
We predict that an edge exists if ˆ y is larger than a threshold (as a hyper-parameter, 0 .
5 in our implementation).
Limitations.
There are two main limitations of GraphANGEL.
The ﬁrst one is that GraphANGEL would not work reliably if the assumptions (intuitions) in Section 2 are violated.
Examples include when node s and t are disconnected or topologically far away when (cid:104) s, r, t (cid:105) is removed, so that one may not be able to ﬁnd 3-cycle or 4-cycle target subgraphs.
The prediction in this case can only rely on Pair patterns, i.e., comparing if the source-target pair is similar to the incident nodes of edges with relation r or not.
The second one is that current online subgraph sampling algorithms are slow.
To make sampling efﬁcient, the subgraph retrieval stage requires ﬁnding and storing all 3-cycles and 4-cycles (See Appendix A3.1 for the space complexity analysis).
For static graphs, it is a one-time preprocessing step, although the results can be stored on external storage.
We also provide the incremental searching and retrieving algorithms to address the dynamic graphs in the real-world scenario in Appendix A3.2.
See further discussions on these limitations in Appendix A4.1.
[Table]
E XPERIMENT 4. E XPERIMENT S ETUP AND C OMPARED A LGORITHMS
Recommendation on Heterogeneous Graph .
We evaluate our model on four heterogeneous graph benchmark datasets in various ﬁelds: LastFM (Hu et al., 2018a), Yelp (Hu et al., 2018b), Amazon (Ni et al., 2019), and Douban Book (Zheng et al., 2017).
The baselines we compare against are: HetGNN (Zhang et al., 2019a), HAN (Wang et al., 2019b), TAHIN (Bi et al., 2020), HGT (Hu et al., 2020), and R-GCN (Schlichtkrull et al., 2018).
As the recommendation task can naturally be regarded as the relation predictions between each user and item pair, for each triplet, we formulate the task as a binary classiﬁcation task.
We split each dataset into 60%, 20%, and 20% for training, validation and test sets, respectively.
Following the setting of (Zhang et al., 2019a), we generate an equal number of negative triplets with the same relation type in the test set, and report Area Under ROC Curve (AUC),
Accuracy (ACC), and F1 score.
More details of the datasets and experimental conﬁgurations as well as the implementation details for baselines are reported in Appendix A5.
Knowledge Graph Completion .
We compare different methods on two benchmark datasets: FB15k237 (Toutanova & Chen, 2015) and WN18RR (Dettmers et al., 2018), which are constructed from
Freebase (Bollacker et al., 2008) and WordNet (Miller, 1995), respectively.
The baselines we compare against are: MLN (Singla & Domingos, 2005), TransE (Bordes et al., 2013), ConvE (Dettmers et al., 2018), ComplEx (Trouillon et al., 2016), pLogicNet (Qu & Tang, 2019), RotatE (Sun et al., 2019a),
RNNLogic (Qu et al., 2020), ComplEx-N3 (Lacroix et al., 2018), GraIL (Teru et al., 2020) and QuatE (Zhang et al., 2019b).
For each triplet, we mask the source or target node, and let each method predict the masked node.
We use the ﬁltered setting during evaluation on the standard training-validation-test split, randomly break ties for triplets with the same score (Sun et al., 2019b) and report Mean Rank (MR), Mean Reciprocal Rank (MRR), and Hit@K (K=1,3,10).
In each task, we implement GraphANGEL as we proposed in Section 3.
Concretely, we imply
GraphANGEL with Π p upon the logical patterns shown in Table 2.
For further investigations on the inﬂuence of different graph patterns, we here introduce GraphANGEL 3 − cycle , a variant of
GraphANGEL without using patterns in 3-cycle shapes; and GraphANGEL 4 − cycle , another variant without using patterns in 4-cycle shapes.
For recommendation tasks on heterogeneous graphs, since the number of edge types is usually small, we can enumerate all the relation type combination in each pattern.
This allows us to make the pattern Π p speciﬁc to relation types.
We denote this variant as
GraphANGEL ∗ .
As the main advantage of our model is that it can generalize to relation types unseen during training without ﬁne-tuning , we both evaluate the overall performance of GraphANGEL in standard relation prediction tasks and its generalizability to unseen relation types against other state-of-the-art methods.
We exam the robustness of GraphANGEL by adding the Gaussian noise into the heterogeneous graphs and report the results in Appendix A6.5.
We also compare the training and inference time of GraphANGEL against baseline models and report results in Appendix A6.7.
4. R ESULT A NALYSIS OF S TANDARD T ASKS
Heterogeneous Graph Based Recommendation .
In recommendation scenarios, edges between user and item nodes are generally more likely to exist if they share neighboring users or items.
In other words, users close in the graph may share similar interests and items close usually share similar attributes.
Table 3 summarizes the performances of GraphANGEL and baselines on four different kinds of recommendation tasks.
We observe that GraphANGEL signiﬁcantly outperforms the baselines across all datasets in terms of AUC, ACC, and F1 metrics.
Almost all prevailing baseline methods on heterogeneous graph are based on sampling through metapath.
One explanation is that [Table] given a speciﬁc pattern, these metapaths can be roughly regarded as target patterns, but without constructing supporting and refuting patterns in Table 1.
Knowledge Graph Completion .
In knowledge graphs, the connection between two nodes is determined by both logic and node attributes.
Table 4 summarizes all experimental results.
As can be seen, GraphANGEL outperforms all baselines across all datasets.
One explanation is that most knowledge graph embedding techniques focus on mining the hidden information in each tuple, which is similar to only considering the patterns in Pair shapes in Table 1.
However, other patterns contain information involving multiple relations, which enables to model the logics.
In order to better illustrate our performance of these relations with few occurrences in the training set, we solely report the results of testing each model on the 20 % relations with few occurrence in
Table A4 (see Appendix A6.1 for details).
From the comparison between Tables 4 and A4, we can observe that with few shots of relations, GraphANGEL can have a better generalization ability.
One reason is that embeddings of the relations with few occurrences cannot be trained with plenty of data samples, resulting in low expressive power of the relations.
In contrast, GraphANGEL does not learn the embeddings directly, but learns to represent the relations of the related logics.
However, it is still more challenging to model these relations that lead to a drop in performance.
[Figure]
4. E FFECT OF D IFFERENT P ATTERNS
We systematically investigate the effect of three shapes of patterns used in GraphANGEL.
For each dataset, we evaluate the effect of the patterns in 3-cycle and 4-cycle shapes by the performance of GraphANGEL 3 − cycle and GraphANGEL 4 − cycle .
Since different shapes of patterns represent different composition logical rules, as shown in Table [Figure] patterns in Pair shape are the most general but include the least structure information, while those in 4-cycle shape are rich in the structure but less common.
Hence, these patterns have their unique power in representing logics.
Although it is hard to determine whether 3-cycle or 4-cycle shaped patterns is more powerful, as shown in Tables 3, 4 and A4, GraphANGEL with patterns in all shapes achieves the best performance.
Besides the pattern type, we also investigate how the number of sampled subgraphs affects the performance.
Taking Amazon and Douban Book datasets as examples, we show the performance of GraphANGEL under different K and Q in terms of ACC and AUC in Figure 3.
One explanation is that the subgraphs following target patterns are constricted within the neighborhood of source and target nodes, the number of which is much smaller than subgraphs following supporting and refuting patterns. [Table]
4. R ESULT A NALYSIS OF G ENERALIZATION S TUDY
We further evaluate these models in a scenario where generalizing from existing relations to unseen relations is required.
Concretely, we use the same datasets with knowledge graph completion and randomly split R into two partitions R seen and R unseen .
Each model is trained and validated only with the relations in R seen .
During testing, the training and validation triplets in R unseen are added back to the original graph.
We report results on test triples with relations in R unseen only.
[Figure]
We cannot directly use the baselines above for unseen relations since those relation embeddings are never trained.
Therefore, we combine them with EmbedRule, which estimates the relation embedding by ﬁnding a number of most common relation sequences that cooccur with the unseen relation, and composing those embeddings thereafter (Yang et al., 2014).
We superscript the name with an asterisk for models enhanced by EmbedRule (e.g. TransE ∗ ).
Results in Table 5 illustrate that our model is signiﬁcantly less affected than other models when we drop 20% relations from the training and validation sets.
We additionally report the results of dropping or adding 5%, 10%, 15% relations in Appendix A6.2.
These results on FB15kare summarized in Figure 4, where we see that our model obtains the better generalization ability.
4. R ESULT A NALYSIS OF E XPLAINABLE A TTENTION M AP
𝑏,𝑎 𝑏,𝑎𝑏,𝑎 𝑏,𝑎 𝑏,𝑎𝑏,𝑎 𝑏,𝑦𝑏,𝑎 𝑏,𝑎 𝑏,𝑎 𝑏,𝑦𝑏,𝑎 𝑏,𝑎𝑢,𝑢 𝑏,𝑎 𝑢,𝑢 𝑏,𝑎𝑢,𝑢 𝑏,𝑎𝑢,𝑢 𝑏,𝑎 𝑢,𝑢 𝑏,𝑎𝑢,𝑢
𝑏,𝑎𝑢,𝑢
𝑏,𝑎𝑢,𝑢 𝑏,𝑎𝑢,𝑢
𝑏,𝑎𝑢,𝑢 𝑏,𝑎 𝑢,𝑢 𝑏,𝑎 𝑢,𝑢 𝑏,𝑎𝑢,𝑢 𝑏,𝑎 𝑢,𝑢 𝑏,𝑎𝑢,𝑢
𝑏,𝑎𝑢,𝑢 𝑏,𝑎𝑢,𝑢
𝑏,𝑎𝑢,𝑢 𝑏,𝑎 𝑢,𝑢
𝑏,𝑎𝑢,𝑢
𝑢,𝑢 𝑢,𝑢 𝑢,𝑢 𝑢,𝑢 𝑢,𝑢 𝑢,𝑢 𝑢,𝑢 𝑢,𝑢 𝑢,𝑢 𝑢,𝑢 𝑏,𝑦 𝑏,𝑦 𝑏,𝑦 𝑏,𝑦 𝑏,𝑦 𝑏,𝑦 𝑏,𝑦 𝑏,𝑦 𝑏,𝑦 𝑏,𝑦
Besides the performance, we further show that our model can produce explainable heat maps of attention scores across the discovered logic.
We here provide an illustration on the recommendation task based on Douban Book graph, where we are required to predict the relation existence between each user and book pair.
In Douban Book, we can deﬁne the graph patterns based on the node types, such as 4-cycle shaped patterns: User − Book − Author − Book denoted as ( b, a ) , User − Book − Year − Book denoted as ( b, y ) , and User − User − User − Book denoted as ( u, u ) .
In Figure 5, the rows represent the supporting subgraphs while the columns represent the target subgraphs.
Each cell represents the similarity between a target subgraph (at the top) and a supporting subgraph (at the bottom).
The color of each cell shows the attention weight for corresponding pair of supporting and target subgraphs.
We can observe that the deep color of the cell located at target and supporting subgraphs following ( b, a ) patterns, which indicates that the logic User ∧ Book ∧ Author ∧ Book ⇒ User ∧ Book has high conﬁdence and can be strong evidence to support the relation prediction.
R ELATED W ORK
Heterogeneous graphs such as knowledge graphs (Bordes et al., 2013; Schlichtkrull et al., 2018) and social networks (Zhang et al., 2019a; Jin et al., 2020) that encode facts about the world surrounding us, have motivated work on automatically predicting new statements based on known ones.
Roughly speaking, existing approaches can be summarized into three main branches, namely matrix factorization techniques (Nickel et al., 2011; 2012), statistical relational learning approaches (Richardson & Domingos, 2006; Singla & Domingos, 2005), and neural-embedding-based methods (Bordes et al., 2013; Dettmers et al., 2018). Our work focuses on the study of neural-embedding-based models as they embed nodes and relations into low-dimensional spaces (Bordes et al., 2013; Schlichtkrull et al., 2018; Trouillon et al., 2016; Yan et al., 2019; Teru et al., 2020), showing good scalability and strong generalization ability. One dominant paradigm (Ji et al., 2015; Lin et al., 2015) in this branch is constructed based on translation (Bordes et al., 2013) or rotation (Sun et al., 2019a) assumptions.
The key idea behind this kind of models is that for a positive instance, the source node should be as close as possible to the target node through the relation, serving as a translation or rotation.
Although there are multiple successful stories in this line of works, these models are all trained on individual instances, regardless of their local neighborhood structures.
As stated in (Schlichtkrull et al., 2018), explicitly modeling local structure can be an important supplement to help recover missing relations.
Inspired by the success of GNNs in modeling structured neighborhood information, another line of literature (Schlichtkrull et al., 2018; Zhang & Chen, 2018) learns the relation embedding based on its neighborhood subgraph using graph convolution layers.
With a similar approach, recent work (Zhang & Chen, 2019; Teru et al., 2020) illustrates the inductive capabilities of generalization to unseen nodes.
However, these approaches fail to generalize to unseen relation types, as it is indirect to introduce the “neighborhood” for a relation type upon the graph structure.
Meanwhile, our work also relates to previous researches (Qu et al., 2019;
Qu & Tang, 2019; Zhang et al., 2020; Qi et al., 2018; Zheng et al., 2019) studying to effectively combine GNNs techniques with symbolic logic rule-based approaches (Giarratano & Riley, 1998; Jackson, 1998; Lafferty et al., 2001;
Taskar et al., 2012; Richardson & Domingos, 2006; Singla & Domingos, 2005).
However, these methods, as originally proposed, are transductive in nature.
Unlike our method, they still require learning relation type speciﬁc embeddings, whereas we treat the relation prediction as a graph pattern matching problem, independent of any particular relation type identity.
There are also other work (Yang et al., 2014) that designs a logical rule extraction approach based on the potential generalization ability of transition-based methods.
As this method can incorporate with various neural-embedding-based models, this set of models constitute our baselines in the inductive setting.
Notably, comparing to the message passing neural network (Gilmer et al., 2017), the current prevailing GNN framework, as illustrated in Figure A2, GraphANGEL is able to avoid the GNNs’ limitation of using neighbor nodes by using nodes satisfying certain logical patterns (See detailed discussion in Appendix A4.1).
Another popular link prediction approach is to directly infer the likelihood of relation existence using a local neighborhood or a local subgraph (Schlichtkrull et al., 2018; Zhang & Chen, 2018;
Hu et al., 2020).
A signiﬁcant difference between those works and our approach is in the formulation of the loss function.
Let f r be any function that maps a vector to a scalar score parametrized by r and (cid:96) r be a scalar function, then the prior works’ loss function often take the form L = (cid:80) (cid:104) s,r,t (cid:105)∈D (cid:96) ( f r ( h ∗ ) , y ) , meaning that the score computation explicitly depends on the embedding of relation to be predicted.
Ours, in contrast, has the form L = (cid:80) (cid:104) s,r,t (cid:105)∈D (cid:96) ( d ( h ∗ , h + ) , d ( h ∗ , h − ) , y ) where (cid:96) is a scalar function and d is a distance function between two vectors, which are embeddings of a set of subgraphs.
Consequently, our loss function (cid:96) does not require representation of r .
C ONCLUSION
We propose a novel relation prediction framework that predicts the relations between each node pair based on the subgraph containing the pair and other subgraphs with identical graph patterns, and has a strong inductive bias for the generalization to unseen relation types.
With these graph patterns, we introduce several graph pattern searching and sampling techniques, which can efﬁciently ﬁnd subgraphs matching the patterns in triangle and quadrangle shapes.
In the future, we plan to further extend GraphANGEL to more complex structures (i.e., compositional logical rules). Acknowledgments.
The Shanghai Jiao Tong University Team is supported by Shanghai Municipal
Science and Technology Major Project (2021SHZDZX0102) and National Natural Science Foundation of China (62076161, 62177033).
We would also like to thank Wu Wen Jun Honorary Doctoral
Scholarship from AI Institute, Shanghai Jiao Tong University
A I LLUSTRATIONS OF N OTATIONS
[Figure]
notation logical pattern notation supporting subgraph set target subgraph set supporting subgraph embedding set refuting subgraph set ?
... target subgraph embedding set ... target subgraph embedding set ...
[Figure]
In this paper, we begin by bridging the logical expression and the graph pattern.
Taking Figure A1 as example, we aim to predict the relation live between the source and target nodes (i.e., Person E and London ).
We consider the 3-cycle graph pattern.
Formally, we use Π p where p
= 1 , 2 , . .
. , P to denote a certain graph pattern.
For each Π p , we use Π ∗ p to denote the corresponding target pattern, Π + p to denote the corresponding supporting pattern, Π − p to denote the corresponding refuting pattern.
Concretely, as shown in Table 1, we can deﬁne Π p =
Edge ( x, z ) ∧ Edge ( z, y ) , Π ∗ p =
Edge ( x, z ) ∧ Edge ( z, y )
∧ Source ( x ) ∧ Target ( y ) , Π + p
=
Edge ( x, z ) ∧ Edge ( z, y )
∧ Edge live ( x, y ) , Π − p =
Edge ( x, z ) ∧ Edge ( z, y )
∧¬ Edge live ( x, y ) .
According to these patterns, we can retrieve the corresponding subgraphs on the graph (as shown in Figure 1) and construct the corresponding subgraph sets.
Figure A1 illustrates a target subgraph representing the logical expression Watch ( Person E , Arsenal ) ∧ Base ( Arsenal ,
London )
∧ Source ( PersonE )
∧ Target ( London ) .
We use S ∗ p,k to denote the k -th sampled target subgraph following the target pattern Π ∗ p where k = 1 , 2 , . .
, K , S + p,q to denote the q -th sampled supporting subgraph following the supporting pattern Π + p , S − p,q to dentoe the q -th sampled refuting subgraph following the refuting pattern Π − p
where q = 1 , 2 , . . .
, Q .
For each Π p , after applying pattern retrieval and sampling introduced in Appendix A2, we build a set of target subgraphs denoted as {S ∗ p,k } Kk =1 , a set of supporting subgraphs denoted as {S + p,q } Qq =1 , a set of refuting subgraphs denoted as {S − p,q } Qq =1 .
We then learn a representation for each subgraph, and obtain a set of corresponding target subgraph embeddings denoted as { e ∗ p,k } Kk =1 , a set of corresponding supporting subgraph embeddings denoted as { e + p,q } Qq =1 , a set of corresponding refuting subgraph embeddings denoted as { e − p,q } Qq =1 .
A P SEUDOCODE , C ORRECTNESS P ROOFS , AND C OMPLEXITY A NALYSIS FOR 3- CYCLE AND 4- CYCLE P ATTERN R ETRIEVAL AND S AMPLING
In this section, we ﬁrst summarize the complexity of retrieving and sampling subgraphs following the graph patterns in 3-cycle and 4-cycle with the following theorem:
Theorem 1.
(Time Complexity of Retrieval and Sampling)
Given a graph G =
( V , E ) and a graph pattern Π 3 − cycle in 3-cycle and a graph pattern Π 4 − cycle in 4-cycle, the time complexity of retrieving all the (supporting) subgraphs satisfying Π 3 − cycle is O ( |E| 32 ) , of retrieving all the (supporting) subgraphs satisfying Π 4 − cycle is O (max( |E| 32 , N 4 − cycle )) where N 4 − cycle is the number of quadratic cycles in G and also the trivial lower bound of the time complexity.
For uniform sampling algorithms, the time complexity of sampling n 4 − cycle supporting cases of Π 4 − cycle is O
( |E| 32 + n 4 − cycle ) .
As there are usually more refuting cases than supporting ones, the time complexity of sampling n refuting cases of Π 3 − cycle or Π 4 − cycle is O ( |V| + |E| + n ) .
In the following subsections, we show the proof of Theorem 1 by providing the pseudocode of retrieval and sampling algorithms along with the correctness proof and complexity analysis in Appendix A2. and A2.2 respectively.
A2. P SEUDOCODE , C ORRECTNESS P ROOFS , AND C OMPLEXITY A NALYSIS FOR 3- CYCLE AND 4- CYCLE P ATTERN R ETRIEVAL
In this part, we ﬁrst provide two algorithms for searching and retrieving supporting subgraphs following the graph patterns in 3-cycle (i.e., Π 3 − cycle ) in Part 1; and one algorithm for searching and retrieving supporting subgraphs following the graph patterns in 4-cycle (i.e., Π 4 − cycle ) in Part 2.
Part 1.
In Algorithm 2, we present the pseduocode for searching and retrieving all the supporting subgraphs following the pattern Π 3 − cycle .
The input is a graph G =
( V , E ) and the output is a set B containing all the supporting subgraphs satisfying Π 3 − cycle .
We denote the degree of node u by d u , and the set of neighbor nodes of node u in graph G by N G ( u ) .
Algorithm 2: Search and Retrieval Algorithm A for Π 3 − cycle
V 1 ← { u | d u > |E| 12 } , V 2 ← { u | d u ≤ |E| 12 }
B ← {} foreach u ∈ V 1 do foreach v ∈ V 1 do foreach w ∈ V 1 do if (cid:104) u, v
(cid:105) ∈
E and (cid:104) u, w
(cid:105) ∈
E and (cid:104) v, w (
cid:105) ∈
E then
B ← B ∪ { ( u, v, w ) } end end end endforeach u ∈ V 2 do foreach v ∈ N G ( u ) do foreach w ∈ N G
( u ) do if (cid:104) v, w (
cid:105) ∈ E then
B ← B ∪ { ( u, v, w ) } end end end end
Now, we investigate the correctness and the time complexity of Algorithm 2.
Proof.
For any 3-cycle (i.e., 3 nodes connected by 3 edges) ( u, v, w ) , if the degrees of all three nodes are greater than |E| 12 (i.e., d u > |E| 12 and d v > |E| 12 and d w > |E| 12 ), it will be enumerated in the ﬁrst loop.
Otherwise, if the degree of any node is not greater than |E| 12 , the 3-cycle will be enumerated in the second loop.
In order to analyze the time complexity, we ﬁrst investigate an upper bound of V 1 = { u | d u > |E| 12 } : 2 |E| = (cid:88)
u ∈V d u ≥ (cid:88) u ∈Vd u > (cid:88) u ∈V|E| 12 ⇒ |V 1 | < 2 |E| 12 .
For the ﬁrst main loop of the algorithm, we have
[Equation]
For the second main loop, we have
[Equation]
Summarizing the analysis above, we conclude that the overall time complexity is O ( |E| 32 ) .
Alternatively, we provide another searching and retrieving algorithm for all the supporting subgraphs following Π 3 − cycle in Algorithm 3.
The inputs and outputs of the algorithm are consistent with
Algorithm 2.
Algorithm 3: Search and Retrieval Algorithm B for Π 3 − cycle
B ← {} foreach u ∈ V do foreach v ∈ N G ( u ) where
d v ≤ d u do foreach w ∈ N G
( v ) where d w ≤ d v do if (cid:104) v, w (
cid:105) ∈ E then
B ← B ∪ { ( u, v, w ) } end end end end
Next, we analyze and prove the correctness and the time complexity of Algorithm 3.
Proof.
It is not difﬁcult to ﬁnd that each 3-cycle ( u, v, w ) satisfying d u ≥ d v ≥ d w is enumerated.
To analyze the time complexity, let us consider how many w s are enumerated:
[Equation]
[Equation]
From Eq.
[Equation]
we know that for each v satisfying d v > |E| 12 , since d u ≥ d v > |E|
[Equation]
there are at most 2 |E| 12 different u s in the outer loop.
Hence
[Equation]
we have (cid:88)
[Equation]
(8) where [ · ] represents the boolean indicator function (i.e., [ x ] = {+
+} if x is true; otherwise false).
Therefore, the overall time complexity is also O ( |E| 3 2 ) .
Part 2.
In Algorithm 4, we demonstrate the algorithm for searching and retrieving all the supporting subgraphs following graph patterns in 4-cycles (i.e., Π 4 − cycle ).
The inputs, outputs, and notations are similar to those in Algorithm 2.
← {} foreach u ∈ V do
T x ← {} for each x ∈ V foreach v ∈ N G ( u ) where d v ≤ d u do foreach w ∈ N G
( v ) where d w ≤ d u do foreach x ∈ T w do
B ← B ∪ { ( u, v, w, x ) } end
T w ← T w ∪ { v } end end end
We provide the correctness and the time complexity of Algorithm 4 as follows.
Proof.
For correctness, we can see from the algorithm that every 4-cycle ( u, v, w, x ) satisfying d u ≥ max( d v , d w , d x ) is enumerated.
For time complexity, let N 4 − cycle denote the number of quadratic cycles in G .
We ﬁrst have the time complexity Θ( N 4 − cycle ) of the innermost loop.
For the outer three loops, let us consider how many w s are enumerated:
[Equation]
From Eq.
(8), we have
[Equation]
Thus, the overall time complexity is O (max( |E| 32 , N 4 − cycle )) .
A2. P SEUDOCODE , C ORRECTNESS P ROOFS , AND C OMPLEXITY A NALYSIS FOR 3- CYCLE AND 4- CYCLE P ATTERN S AMPLING
From the subsection above, we know that given a graph G =
( V , E ) , we have different algorithmic complexities depending on the graph pattern Π .
For Π 3 − cycle (i.e., 3-cycles), the time complexity of retrieving all supporting subgraphs is O ( |E| 32 ) .
For Π 4 − cycle (i.e., 4-cycles), the time complexity is
O (max( |E| 32 , N 4 − cycle )) , where N 4 − cycle is the number of quadratic cycles in G .
However, for large-scale graphs, it is impractical to retrieve all quadratic-cycle graph patterns since
N 4 − cycle could be too large.
In addition, the number of refuting cases for both Π 3 − cycle and Π 4 − cycle may also be large.
These problems may lead to the high computational cost of these search algorithms.
Realizing this, we further introduce the technique of uniform sampling to keep the time complexity within an acceptable level.
Therefore, in the following, we ﬁrst provide one algorithm for uniformly sampling and retrieving graph patterns following Π + 4 − cycle in Part 1; and two algorithms for uniform sampling and retrieving graph patterns following the refuting patterns in 3-cycle shape (i.e., Π − 3 − cycle ) and 4-cycle shape (i.e., Π − 4 − cycle ) in Part 2.
As a reminder, while practical, another approach to obtain graph patterns following Π − 3 − cycle and Π − 4 − cycle is random walk, although random walk is often not efﬁcient enough.
Part 1.
We present the algorithm for sampling and retrieving all the supporting subgraphs following Π 4 − cycle in Algorithm 5.
The inputs to the algorithm are a graph G =
( V , E ) and n 4 − cycle , rep-
resenting the number of subgraphs to sample.
The output is a set B containing all sampled graph patterns.
B ← {} foreach u ∈ V do
T uw ← {} for w ∈ V foreach v ∈ N G ( u ) where
d v ≤ d u do foreach w ∈ N G
( v ) where d w ≤ d u do
T uw ← T uw ∪ { v } end end endforeach ( u, w ) where T uw (cid:54) = ∅ do c uw ← |T uw | ( |T uw | − 1) end c ← (cid:80) uw c uw foreach ( u, w ) where T uw (cid:54) = ∅ do p uw ← c uw /c
endfor
i ← 1 to n 4 − cycle do
Sample ( u, w ) where T uw (cid:54) = ∅ with probability p uw
Uniformly sample two different nodes u and x from T uw if (cid:104) u, w
(cid:105) ∈
E then
B ← B ∪ { ( u, v, w, x ) } end end
We provide the correctness and the time complexity of Algorithm 5 as follows.
Proof.
For correctness, we can observe from the algorithm that every 4-cycle ( u, v, w, x ) satisfying d u ≥ max( d v , d w , d x ) is counted in T uw exactly once, where v ∈ T uw and x ∈ T uw .
We sample ( u, w ) with the probability of occurrence of the 4-cycle ( u, v, w, x ) , then sample different v and x uniformly from T uw .
For time complexity, it has been proved in Algorithm 4 that the time complexity of computing T is
O ( |E| 32 ) .
Notably, there are O ( |E| 32 ) non-empty T s.
Therefore, if we sample ( u, w ) through Alias Method (Walker, 1977), then the overall time complexity is O ( |E| 32 + n 4 − cycle ) .
Part 2.
We further provide two path sampling algorithms.
In most practical cases, there are many more refuting examples than supporting ones, and if we assume so, then Algorithm 6 can be used to sample graph patterns following refuting patterns in 3-cycle shape (i.e., Π − 3 − cycle ), and Algorithmto sample graph patterns following refuting patterns in 4-cycle shape (i.e., Π − 4 − cycle ).
Algorithm 6 displays the algorithm for uniformly sampling n subgraphs following Π − 3 − cycle in G .
The inputs, outputs, notations are identical to those in Algorithm 5.
We provide the correctness and the time complexity of Algorithm 5 as follows.
Proof.
For correctness, as shown in the algorithm, we formulate the sampling of refuting cases of Π 3 − cycle into uniformly sampling three-node paths, namely ( u, v, w ) .
We ﬁrst sample the intermediate node of the path (i.e., v ), where there are d v ( d v − 1) / 2 three-node paths with v as its intermediate node.
Next, we sample v with probability p v to ensure uniformity, where p v =
d v ( d v − 1) / (cid:80) v d v ( d v − 1) , and then sample its two neighbors to form the path.
Using Alias Method, the overall time complexity is O ( |V| + |E| + n ) , where the computation for the degree requires O ( |E| ) , the ﬁrst two loops require O ( |V| ) , and the last loop requires O ( n ) .
B ← { foreach u ∈ V do c u ← d u ( d u − 1) end c ← (cid:80)
u c u foreach u ∈ V do p u ← c u /c endfor i ← 1 to n do
Sample v with probability p v
Uniformly sample two different neighbor nodes (i.e., u and w ) of v if (cid:104) u, w (cid:105) (cid:54)∈ E then
B ← B ∪ { ( u, v, w ) } end end foreach (cid:104) u, v
(cid:105) ∈
E do if d v = 2 and d u = 2 and |N G ( u ) ∩ N G ( v ) |
= 1 then c uv ←endelse c uv ← ( d u − 1) · ( d v − 1) end end c ← (cid:80) uv c uv foreach (cid:104) u, v (cid:105) ∈
E do p uv ← c uv /c endfor
i ← 1 to n do
Sample (cid:104) v, w (cid:105) with probability p vw
Uniformly sample a neighbor node u of v
Uniformly sample a neighbor node x of w if u (cid:54) =
x and (cid:104) u, x (cid:105) (cid:54)∈ E then
B ← B ∪ { ( u, v, w, x ) } end end
In Algorithm 7, we present the algorithm for uniformly sampling graph patterns following Π − 4 − cycle in G .
We provide the correctness and the time complexity of Algorithm 7 as follows.
Proof.
Analogous to Algorithm 6, we formulate the sampling of refuting cases of Π 4 − cycle into uniformly sampling four-node paths namely ( u, v, w, x ) .
To ﬁrst analyze the correctness of the algorithm, we ﬁrst sample the intermediate edge of the path (i.e., (cid:104) v, w (cid:105) ).
There are ( d v − 1) · ( d w − 1) three-edge paths (including 3-cycles) where the intermediate edge is (cid:104) v, w (cid:105) .
Hence, we sample (cid:104) v, w (cid:105) with the probability proportional to ( d v − 1) · ( d w − 1) , and then sample a neighbor node u from v and another one x from w , and reject the case when it forms a 3-cycle.
To guarantee the time complexity, we do not sample the edge (cid:104) v, w (cid:105) when v and w have the only common neighbor, thus |N G ( v ) ∩N G ( w ) | ( d v − 1) · ( d w − 1) ≤ 12 when c (cid:54) = 0 , i.e., we reject with probability no more than 1 2 , the expected time complexity is O ( |V| + |E| + n ) . Input:
Source for each pattern Π p do
Construct Π ∗ p
Π p
Π −
Search/Sample and retrieve subgraphs following Π p through Algorithms 2,3,4,5,6,7.
Store all the subgraphs in B .
endfor
D do for each logic Π p do Sample and retrieve K subgraphs S p k matching Π p in B .
Sample and retrieve Q subgraphs S
,q matching Π
B .
Sample and retrieve Q subgraphs S
B .
Compute e end
Compute
Calculate via
= Re-sample and retrieve subgraphs following Π p through Algorithms 6,7,5.
Update B with the re-sampled subgraphs.
Update parameters by minimizing loss L shown in Eq.
end
Notably, these algorithms are valuable because the number of the refuting cases are considerable in practice, and they make GraphANGEL more relevant for practical deployment.
A I MPLEMENTATION A LGORITHM OF G RAPH ANGEL A3. A LGORITHM FOR I MPLEMENTATION OF G RAPH ANGEL ON S TATIC G RAPHS
For the heterogeneous graph based recommendation task, we follow (Hu et al., 2020; Wang et al., 2019b; Zhang et al., 2019a) to predict the existence of edge between each given node pair (cid:104) s, t (cid:105) , while in the knowledge graph completion task, we follow (Bordes et al., 2013; Sun et al., 2019a;
Dettmers et al., 2018) to mask head and tail entities instead of relation prediction.
As the input of our model is a triplet (cid:104) s, r, t (cid:105) , one can score all the possible triplets no matter either relation r or tail entity (i.e., node) t is masked.
We then rank all these tuples for evaluation.
Overall Time Complexity of Original Algorithm.
Theorem 1 shows the time complexity of searching and retrieving the subgraphs following the pre-deﬁned 3-cycle and 4-cycle shaped patterns:
O (max( |E| 3 2 , n 4 − cycle )) for sampling n 4 − cycle subgraphs satisfying Π + 4 − cycle , O ( |V| + |E| + n ) for sampling n refuting cases of Π − 3 − cycle or Π − 4 − cycle .
Besides the subgraph retrieval part, the main components of GraphANGEL are the representation computation part (i.e., Φ ) and the similarity computation part (i.e., Ψ ).
We further analyze the time complexity of these parts as follows.
For Φ , as described in Section 3, we employ R-GCN (Schlichtkrull et al., 2018)
(See Appendix A3.
for details).
One can use other GNN architectures (e.g., R-GAT (Busbridge et al., 2019)).
Hence, without loss of generality, we analyze the time complexity of the general GNNs following the message passing framework (Gilmer et al., 2017), most of whose update functions can be formulated as H ( l +1) =
ReLU (( L + I ) H ( l ) W ( l ) ) , where H ∈ R |V|× d l +1 is the embedding matrix obtained after l + 1 steps of propagation, and L represents the Laplacian matrix for the graph, deﬁned by
L = D − 12 AD − 12 , A is the adjacency matrix and D is the diagonal degree matrix,
W l ∈ R d
l × d l −is the trainable transformation matrix.
The l -th propagation layer, the matrix multiplication has computational complexity O ( | L + | · d l × d l − 1 ) , where | L + | is the number of nonzero entities in the
Laplacian matrix, and the d l and d l − 1 are the current and previous transformation size.
Then, the time complexity of Φ with L layers is O ( (cid:80) Ll =1 | L + | · d l × d l − 1 ) .
[Figure]
For Ψ , as Section 3 shows, we use a co-attention mechanism (Lu et al., 2016)
(See Appendix A3.4 for details).
Without loss of generality, let N head denote the number of heads.
Then, the time complexity of Ψ is O (( P × K ) · ( P × Q ) · d 2 · N head ) .
Then, the overall time complexity of Algorithm 1 is to sum up all the above computation costs.
In Algorithm 1, we retrieve and sample subgraphs for every source-target node pair, which can be time-consuming.
Algorithm 8 introduces additional steps that precompute all the graph patterns and their satisfying subgraphs as a data preprocessing step .
The intuition behind this is very straightforward that, as shown in Table 2, the graph patterns considered in this paper are quite simple, involving at most four nodes.
For knowledge graphs, it’s acceptable to pre-compute all the supporting subgraphs of 3-cycle shaped graph pattern using
Algorithm 2 or 3.
We also develop a novel search and retrieval algorithm (as shown in Algorithm 4) to retrieve all the supporting subgraphs with O (max( |E| 32 , N 4 − cycle )) .
However, as the number of supporting cases of 4-cycle shaped graph pattern (i.e., N 4 − cycle ) is too large (as shown in Table A3), while practical, we uniformly sample n
4 − cycle subgraphs following supporting graph patterns in 4-cycle shape using Algorithm 6.
n 4 − cycle is a hyper-parameter, which we set as the number of 3-cycles in graph (i.e., n 4 − cycle =
N 3 − cycle ).
All these subgraphs into a buffer B .
For heterogeneous graphs, since the graph schema of these graphs are usually not complex.
We can cover all possible graph patterns by enumerating on the graph schema.
Similarly, this enables us to pre-compute the graph patterns and store all the retrieved and sampled subgraphs following each graph pattern in the buffer B .
We then can build an index of the buffer by the graph patterns as well as the subgraphs following each graph pattern.
During training, we sample and retrieve subgraphs in the buffer instead of the whole graph for each source-target node pair.
Compared with the original one in Algorithm 1, we reduce the complexity and improve the efﬁciency.
We can further set a update interval to refresh B by re-sampling the supporting cases of Π 4 − cycle and refuting cases of Π 3 − cycle and Π 4 − cycle .
While practical, we ﬁnd that whether refreshing or not would not bring much inﬂuence, and establish an ablation study for the buffer size in Appendix A6.6.
Let n − 3 − cycle , n − 4 − cycle denote the number of refuting samples of Π − 3 − cycle and Π − 4 − cycle respectively.
In practice, we set n = n − 3 − cycle =
n − 4 − cycle = 2 · N 3 − cycle .
We sample n refuting cases of Π 3 − cycle and Π 4 − cycle respectively and store them into B .
Space Complexity of Storing subgraphs.
From the above analysis, we know that if we choose to store all the supporting subgraphs, the space complexity of storing these subgraphs is O ( N 3 − cycle
+
N 4 − cycle ) .
While practical, it is O
( N 3 − cycle ) .
And for the subgraphs of refuting cases, the space complexity is O ( n − 3 − cycle + n − 4 − cycle ) , and O ( n ) for implementation.
Notably, this space complexity is different from the space complexity of the algorithms in Theorem 1, which is the same as its time complexity.
V 1 ← { u ∈ V old | d u > σ } , V 2 ← { u ∈ V old | d u ≤ σ } where σ ← |E old | 1 2 / ( | ∆ V + new | 1 4 |V old | 1 4 )
B ← {} foreach u ∈ V 1 do foreach v ∈ V 1 do foreach w ∈ ∆ V + new do if (cid:104) u, v
(cid:105) ∈
E old and (cid:104) u, w (cid:105) ∈ ∆ E + old and (cid:104) v, w (cid:105) ∈
∆ E + old then
B ← B ∪ { ( u, v, w ) } end end end endforeach u ∈ V 2 do foreach v ∈ N G old ( u ) do foreach w ∈ N ∆ G + old ( u do end end end end
A3. A LGORITHM FOR I MPLEMENTATION OF G RAPH ANGEL ON D YNAMIC G RAPHS
Note that all the algorithms introduced in Appendix A2 mainly focus on searching and retrieving the subgraphs in the context of the static graphs.
However, the graphs used in the real-world scenario almost are dynamic graphs, which always need to be modiﬁed and updated.
In the followings, we show the (incremental) subgraph retrieval algorithms which updates the old buffer (denoted as B old ) calculated by the old graph (denoted as G old ) to the new buffer (denoted as B new ) computed by new graph (denoted as G new ).
Formally, we use G new =
( V new , E new ) and G old =
( V old , E old ) to denote the new graph and the old graph respectively, where V old and V new denote node sets, E old and E new denote node sets.
We do not introduce the relation sets, as our subgraph retrieval does not consider the relations.
We use A old and A new to denote the adjacency matrices of G new and G old regardless of the edge types (i.e., relations).
Let ∆ G − =
(∆ V − , ∆ E − ) denote a graph including all the nodes and edges removed from G old comparing to G new ; and ∆ G + = (∆ V + , ∆ E + ) denote a graph consisting of all the nodes and edges added to G old comparing to G new .
As some edges in ∆ G + may connect the a node in G old and a new node, we can further divide ∆ G + into two graphs.
One is a graph including all the new nodes and edges connecting these nodes, denoted as ∆ G + new =
(∆ V + new , ∆ E + new ) .
And the rest edges together with corresponding nodes construct another graph denoted as ∆ G + old = (∆ V + old , ∆ E + old ) .
Hence, putting ∆ G − aside, the original new graph G new can be divided into three parts to consider: G old , foreach u ∈ V do c u ← d u ( d u − 1) end c ← (cid:80)
u c u foreach u ∈ V do p u ← c u /c end
# Following is Generation of Uniform Sampling foreach (cid:104) u, v
(cid:105) ∈
E do if d v = 2 and d u = 2 and |N G ( u ) ∩ N G ( v ) |
= 1 then c uv ←endelse c uv ← ( d u − 1) · ( d v − 1) end end c ← (cid:80) uv c uv foreach (cid:104) u, v (cid:105) ∈
E do p uv ← c uv /c end ∆ G new and ∆ G old .
Notably, these three parts have overlapping nodes but do not have overlapping edges, as shown in Figure A3.
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
To efﬁciently obtain ∆ G − , ∆ G + old and ∆ G − new from G old and G new , we can ﬁrst compute ∆ G − by comparing G new to
G old (or comparing A new to A old ).
We can then calculate ∆ V + new consisting of the nodes in V new
but not in V old by ∆ V + new =
V new + ∆ V − − V old .
Next, we can calculate ∆ E + new by referring to A new .
And, ∆ E + old can be derived by ∆ E + old =
E new + ∆ E − − E old − ∆ E + new .
From Algorithms 2,3,4,6,7,5, we can see that the buffer B only store the of nodes (i.e., node IDs) of each subgraph instance.
These IDs for subgraphs following the supporting graph patterns in 3-cycle shape represent all the 3-cycles in the graph, and the IDs for subgraphs following the supporting graph patterns in 4-cycle shape represent a number of uniformly sampled 4-cycles in the graph.
Therefore, we are seeking for an efﬁcient way to update the buffer from
B old , the buffer computed by G old , to B new , the buffer computed by G new .
From the above analysis, we are required to search and retrieve all the new subgraphs following Π + 3 − cycle and uniformly sample the new subgraphs following Π + 4 − cycle .
As the time complexity of sampling n refuting subgraphs following Π − 3 − cycle or Π − 4 − cycle in
G new is O ( |V new | + |E new | + n ) (as shown in Theorem 1), and thus it’s yet efﬁcient to directly re-compute these refuting subgraphs.
We summarize the pipeline of updating the buffer B in
Algorithm 11 where [ · ] denotes the ﬂooring function, and provide detailed descriptions as follows.
Firstly, we deal with “delete modiﬁcations” from G old to
G new (i.e., ∆ G − ) by removing all the instances that include the nodes in ∆ V − .
Secondly, for new
[Figure]
subgraphs following Π + 3 − cycle , these 3-cycles might either in ∆ G + new or include the nodes and edges across G old and ∆ G + new .
For the former one, we can directly apply Algorithm 2 or 3 in ∆ G + new , while for the latter one, we need to consider two different cases, shown as (a) and (b) in Figure A3.
For searching and retrieving subgraphs in (b) shape, it’s practical to directly enumerate all the edges in ∆ V + new , as | ∆ V + new | is expected to be small.
However, |V old | is large in the most time and thus we develop Algorithm 9 to efﬁciently search and retrieve subgraphs in (a) shape, whose design is analogous to Algorithm 2.
Thirdly, for Π + 4 − cycle , similar to Π + 3 − cycle , there are also two classes of subgraphs.
For the ﬁrst one, we can uniformly sample a number of subgraphs following Π + 4 − cycle by directly applying Algorithmin ∆ G + new .
For the second one, we need to consider ﬁve different cases, shown as (c), (d), (e), (f), (g) in Figure A3.
We can simultaneously deal with (c), (d), (e) by uniformly sampling edges in ∆ V + old where the sample rate of edges can be computed by using Algorithm 10.
For (f) and (g), we ﬁrst uniformly sample a 3-cycle in one of G old and G + new , which is similar to Algorithm 6, and then uniformly sample one node in the other one of G old and G + new .
The sample rate in these cases can also obtain by using Algorithm 10.
For Π − 3 − cycle and Π − 4 − cycle , considering Algorithm 7 and 5 only have the linear complexity, we directly re-sample the refuting cases of Π − 3 − cycle
and Π − 4 − cycle by running these algorithms on G new .
Time Complexity of Incremental Updating Algorithm.
As shown in Algorithm 11, there are mainly four parts: removing the instances including the nodes in ∆ V − ; searching and retrieving the supporting cases of Π + 3 − cycle ; sampling the the supporting cases of Π + 4 − cycle ; sampling the refuting cases of Π − 3 − cycle
and Π − 4 − cycle .
Let |B| denote the number of instances in the buffer B .
The ﬁrst part requires O ( |B old | · | ∆ V − | ) computations, the second part requires O ( | ∆ E + new | 32 + | ∆ E + new |·|V old | + |E old |· min( | ∆ V + new | , [ |V old | 12 | ∆ V + new | 12 ]) computations, the third part requires O ( | ∆ E + new | 32 + [ | ∆ V + new | 4 |V old | 4 n 4 − cycle ]
+ |V old | + |E old | + | ∆ V old | + | ∆ E old | + | ∆ V new | + | ∆ E new | + [ | ∆ E + old | [Table] ] +
[Table] [Table]
n 4 − cycle ]) , the fourth part requires O ( |V new | + |E new | +[ |
[Table]
n ]) , where [ · ] denote the ﬂoor function.
The overall complexity is to sum up the above computations.
One can easily conclude that when the numbers of new nodes and edges are relatively much smaller than the numbers of old nodes and edges, Algorithm 11 is more efﬁcient than re-running the subgraph retrieval algorithms on the new graph, which are the common cases in the real-world scenario.
A3. D ETAILED F ORMULATION OF Φ IN G RAPH ANGEL
As introduced in Section 3, we apply a neural network Φ( · ) over each subgraph.
These subgraphs can be from either target subgraph set {S ∗ p,k } P,Kp =1 ,k =1 , supporting subgraph set {S + p,q } P,Qp =1 ,q =1 , or refuting subgraph set {S − p,q } P,Qp =1 ,q =1 , as shown in Eq.
Formally, Φ maps each subgraph S ∗ p,k , S + p,q or
S − p,q to the corresponding graph-level representations e ∗ p,k , e + p,q or e − p,q .
In the implementation, we adopt single layer R-GCN (Schlichtkrull et al., 2018) followed by any readout function (e.g., Mean ( · ) ,
Max ( · ) ) as Φ( · ) .
Taking S ∗ p,k as an instance, we use h v,p,k to denote the embedding of node v in the subgraph namely v ∈ S ∗ p,k , and N r S p,k ( v ) to denote the set of neighbor nodes of node v in S p,k regarding relation type r .
We compute h v,p,k by
[Equation]
where x · denote the input node features or learnable node embeddings, W · denote trainable parameters, and σ is non-linearity function such as ReLU .
After updating the embedding vectors of all the nodes, we further compute e ∗ p,k by e ∗ p,k
= Mean ( { h v,p,k | v ∈ S ∗ p,k } ) .
Computations of e + p,q and e − p,q are similar.
Speciﬁcally, in the case of Pair shape, R-GCN reduces to an MLP, and thus Φ here generally would not be worse than what an MLP could achieve.
A3. D ETAILED F ORMULATION OF Ψ IN G RAPH ANGEL
As stated in Section 3, we deploy a neural network Ψ( · ) to measure the similarity s + between the set of subgraphs matching Π ∗ 1 , Π ∗ 2 , . .
, Π ∗ P and the set of subgraphs matching Π +1 , Π +2 , . .
, Π + P ; and the similarity s − between the set of subgraphs matching Π ∗ 1 , Π ∗ 2 , . . .
, Π ∗ P and the set of subgraphs matching Π − 1 , Π − 2 , . .
, Π − P .
Take the computation of s + as an example.
We ﬁrst reshape the subgraph embedding tensors e ∗ p,k into matrices: E ∗ =
CONCAT [ { e ∗ p,k , p = 1 , . .
, P ; k = 1 , . . .
, K }
] ∈ R d × ( P × K ) , and similarly form e + p,q to E + ∈ R d × ( P × Q ) , where d is the hidden representation size.
We then compute the afﬁnity matrix as
[Equation]
(12) where W c ∈ R
d
×
d are trainable weights.
With the afﬁnity matrix A , one possible way of computing attention is to simply maximize out the afﬁnity, namely a + [ j ] =
Max i ( A i,j ) and a ∗ [ i ] =
Max j ( A i,j ) .
[Table]
Instead of using the max activation, we follow (Lu et al., 2016), considering this afﬁnity matrix as a feature and learning the attention map via following:
[Equation]
which are used to compute the attention map among E ∗ and E + through
[Equation]
where W ∗ ∈ R d × d and W + ∈ R d × d , w ∗ ∈ R d and w + ∈ R
d are the weight parameters.
a ∗ ∈
R ( P × K ) and a + ∈ R ( P × Q ) .
We ﬁnally obtain the set embedding via
[Equation]
15) which is the weighted average of the subgraph embeddings in matrix E + and E ∗ .
The similarity of supporting patterns is s + = h
+ (cid:62) h ∗ .
Computation of s − is similar.
A F URTHER A NALYSIS OF G RAPH ANGEL A4. L IMITATIONS OF G RAPH ANGEL
In order to further analyse the limitations of GraphANGEL, we ﬁrst compare our framework with exsiting GNN methods.
As shown in Figure A2, we provide an illustrated example of nodes and edges activated and not activated in various GNN models, including GCN (Kipf & Welling, 2016),
GraphSAGE (Hamilton et al., 2017), GAT (Veli ˇ ckovi ´ c et al., 2017), SEAL (Zhang & Chen, 2018),
PPNP and APPNP (Approximate PPNP)
(Klicpera et al., 2018), and our model GraphANGEL.
With a limited number of GNN layers, one can easily observe that these existing approaches only involve all context within the “neighborhood”.
Instead, GraphANGEL leverages the graph structure by searching and retrieving the subgraphs in the whole graph, which not only removes the limitation of neighborhood constriction but also selects pattern (i.e., logic) related information.
Figure A2 depicts an illustrated example of the edges activated in GraphANGEL comparing to other GNNs, where we can see that our method can be regarded as to restrict the message passing within the subgraphs satisfying certain pre-deﬁned patterns, (e.g., Π 3 − cycle , Π 4 − cycle ).
Although from the analysis above, we can see that GraphANGEL can succeed in those cases where existing GNN models fail; however, leveraging logically-induced graph patterns can also be a limitation of GraphANGEL, especially when it is hard to ﬁnd or design such graph patterns.
For example, if the two nodes for the link prediction are far away from each other, the graph patterns considered in our experiments may be not able to cover the cases, and it is also hard to deﬁne what a graph pattern should be, which is previously pointed out in Section 3.
We argue that such cases are rare in real-world graphs as they are usually scale-free, meaning that the topological distance is usually small.
The topological distance of graphs are well measured by Wiener index, and detailed theoretical description and empirical study can be found in (Dobrynin et al., 2001).
Due to the high computation costs for shortest path algorithms (Floyd, 1962; Gallo & Pallottino, 1988), in this paper, instead, we calculate the fraction of 3-hop and 4-hop neighborhoods over all the nodes for every node in each dataset, average them, and report the results in Table A3, which shows that 3-hop and 4-hop neighborhoods can cover almost all the nodes in the graph.
Since there is no connectivity requirement between the source and target nodes in the target pattern as shown in Table 1, it is unlikely to fail to ﬁnd at least one subgraph satisfying the target patterns.
The other limitation pointed out in Section 3 is the time complexity of Algorithm 1 and the storage complexity of Algorithm 8.
We provide the complexity analysis in Appendix A3 and report the overall time of training and inference in Appendix A6.7.
We argue that there is a recent trend in building language models (Khandelwal et al., 2019;
He et al., 2021; Khandelwal et al., 2020; Zheng et al., 2021) by searching and storing the representation vectors of neighbor nodes.
Comparing to these methods, our subgraph retrieval only search, retrieve and further store the node IDs, which
[Table] [Table]
are relatively much more retrieval-efﬁcient and storage-efﬁcient than searching and storing the representation vectors of nodes.
Besides the above discussions about the limitations pointed out in Section 3, we also provide the following discussions regarding the assumptions proposed in Section 2.
One of the assumptions introduced in Section 2 is that existing relations is enough to cover unseen relations by logical inference, which motivates us to not explicitly construct embeddings for the relation to predict.
However, this assumption also disables GraphANGEL to consider the correlations among unseen relations.
In other words, it’s not possible for current GraphANGEL to incorporate unseen relations as a part of the graph after actually observing them during inference.
One possible solution is to incorporate our model with EmbedRule
(Yang et al., 2014) to ﬁrst generate explicit embeddings for each unseen relations by EmbedRule, and then apply GraphANGEL to generate the ﬁnal predictions.
Considering that both EmbedRule and GraphANGEL have strong generalizability to unseen relations, it is hard to say which one plays a more important role in this case.
Hence, we leave it as future work.
The other one of the assumptions is to bridge the graph patterns with logical rules.
Therefore,
GraphANGEL is capable of distinguishing different semantics of the same relation as long as the local subgraphs, hence contexts, are different.
A4. E XTENSIONS OF G RAPH ANGEL TO H OMOGENEOUS G RAPH
One may consider that heterogeneous graph may be another potential limitation of GraphANGEL.
However, we note that extending GraphANGEL to homogeneous graphs is possible with graph patterns that are not type-sensitive, i.e., not containing type-speciﬁc indicator operators
Edge r ( x, y ) .
In other words, the graph patterns will be purely based on topology.
For example, we can deﬁne 6-cycle for the graph in Figure A4(d). [Table] [Table]
Following the analysis above
[Table]
although nodes t and v in both Figure A4(a) and (d) have similar neighborhood structures
[Table]
our GraphANGEL is able to project these two source-target node pairs (i [Table] e., (cid:104) s, r, t (cid:105) and (cid:104) v, r, t (cid:105) ) into different embeddings on both heterogeneous and homogeneous graphs.
A4. E XTENSIONS OF G RAPH ANGEL TO C OMPLEX L OGIC
Another potential limitation of GraphANGEL is that the graph searching stage limits the model’s capability of exploring more complex graph patterns.
We admit that the graph pattern involved in our paper is simple and rudimentary [Table] but we argue that efﬁcient ﬁnding arbitrary graph patterns including those complex ones should be mainly investigated by graph database and pattern matching literature, which is deﬁnitively not the focus of our paper.
Our key idea is general since one can pre-deﬁne any pattern to search and retrieve subgraphs.
Moreover, the following corollary shows that the representations of complex logical formulas can be derived from those of simpler ones.
This, to some extent, provides the theoretical support of only involving simple graph patterns (e.g., Pair, 3-cycle, 4-cycle) in the implementation.
Lemma 1.
(Barceló et al., 2019)
(Expressive Power on Logic Expressions of Nodes)
Each FirstOrder Classiﬁer with 2 variables (FOC 2 ) can be captured by Aggregation and Combination GNN formulated as
[Equation]
where AGG ( X ) =
(cid:80)
x ∈X x and COM ( x , y )
=
σ ( xC + yA + b ) .
Corollary 1.
(Expressive Power on Logic Expressions of Subgraphs) Let e x = { e 1 , . .
, e P } ∈ R P be the representation vector of subgraph S x , where e p = 1 if subgraph S x matches graph pattern Π xp and e p = 0 otherwise, and Π x 1 , · · · , Π xP are sub-formulas of Π x , in the order where if Π xk is a sub-formula of Π xl then k ≤ l .
Suppose that the representation vectors e x and e y of subgraph S x and
S y matching pattern Π x and Π y respectively, are given.
If Π z takes either of the following forms:
• Π z = ¬ Π x
[Table] [Table]
• Π z
=
Π x ∧ Π y • Π z
= Π x ∨ Π y
Then there exists W x ∈ R P × P , W y ∈ R P × P , and b ∈ R P , such that
[Equation]
is a representation vector of a subgraph S
z matching Π z .
Proof.
The entries of the p -th columns of W x , W y , and b depend on the formulas of Π x,y as follows.
• if Π x,xp = Π xi , then W xip = 1 .
• if Π x,yp =
Π xi
∧ Π yj , then W xip
= 1 , W xjp = 1 , and b p = − 1 .
• if Π x,yp =
Π xi ∨ Π yj , then W xip = 1 , W xjp = 1 , and b p = 0 .
• if Π x,xp = ¬ Π xi , then W xip = − 1 and b p = 1 .
All other values in the p -th columns of W x , W y , and b are 0 .
Our proof follows the proof of
Lemma 1 in (Barceló et al., 2019).
The only differences are: (1) as the input of the basic logic formula in (Barceló et al., 2019) is nodes while in our paper is subgraph, thus the basic logic in (Barceló et al., 2019) is the color of nodes (i.e., Col ( · ) ), but in our setting is the graph pattern for subgraphs; (2) we only involve the basic logical operators ∧ , ∨ and ¬ in our paper.
Besides the prior literature (Barceló et al., 2019) studying the theoretical expressive power of GNNs, our work is also kind of related to a thread of works (Minervini et al., 2020a;b; Weber et al., 2019) on Neural Theorem Provers, where they ﬁrst assume template rules and then ﬁnd which relations can satisfy them.
In contrast, our work bridges the connection between logical expression and graph patterns and use the subgraphs to represent the rules.
This intuition enables our model empowered by the expressive power of GNNs, and also allows our work to identify new relation types.
[Table] [Table]
[Table]
A D ETAILED E XPERIMENTAL S ETTINGS A5. D ATASET S TATISTICS
The statistics of the datasets of heterogeneous graph based recommendation and knowledge graph completion used in our paper are displayed in Table A1 and Table A2, respectively.
We also report the statistic of the patterns used in the datasets in Table A3, which can be regarded as the supporting evidence of using these patterns, since it is unlikely to fail to ﬁnd at least one subgraph satisfying these patterns.
A5. B ASELINES
For the baseline methods, we employ these baseline methods based on Deep Graph Library (DGL) (Wang et al., 2019a) following their original setting or directly use their original implementations: (1) HetGNN (Zhang et al., 2019a) is a uniﬁed framework that jointly considers heterogeneous structured information as well as heterogeneous content information.
(2) HAN (Wang et al., 2019b) is a hierarchical attention mechanism designed on heterogeneous graph to capture node-level and semantic-level information.
(3) TAHIN (Bi et al., 2020) is a cross domain model from both source and target domain, which is then incorporated with three-level attention aggregations to get node representation.
(4) HGT (Hu et al., 2020) is a heterogeneous graph transformer architecture that characterizes the heterogeneous attention over each relation.
(5) R-GCN (Schlichtkrull et al., 2018) combines graph neural networks with factorization models for link prediction.
(6) pLogicNet (Qu & Tang, 2019) is an EM-based algorithm that simultaneously trains a rule generator as well as reasoning predictor with logic rules.
(7) TransE
(Bordes et al., 2013) is a translation-based relation prediction model.
(8) ConvE (Dettmers et al., 2018) is convolution-based relation prediction model.
(9) ComplEx (Trouillon et al., 2016) uses the composition of complex embeddings to handle a large variety of binary relations.
(10) MLN (Singla & Domingos, 2005) leverages domain knowledge to predict relations.
(11) RotatE
(Sun et al., 2019a) is a rotation-based relation prediction model.
RNNLogic (Qu et al., 2020) deﬁnes the joint distribution of all possible triplets by using Markov logic network which can be efﬁciently optimized with the variational EM algorithm.
(13) ComplEx-N (Lacroix et al., 2018) introduces a regularizer based on tensor nuclear p -norms for relation prediction.
(14) GraIL (Teru et al., 2020) reasons over local subgraph structures and holds a strong inductive bias to learn entity-independent relational semantics.
(15) QuatE (Zhang et al., 2019b) introduces [Table] [Table] quaternion embeddings, whose framework can be regarded as a generalization of ComplEx on hypercomplex space while offering better geometrical interpretations
[Table]
A5. H YPER -P ARAMETERS
We keep all hyper-parameter settings in their original implementations, except the following ones for fair comparisons.
All models involving GNN layers are set to have similar layers (2 orlayers, depending on the performance) and are trained for the same number of epochs.
We ﬁx model conﬁgurations across all experiments.
For all baseline GNN models and GraphANGEL, the dimension size of the embeddings is set to 128, and the learning rate space is { 0.001, 0.0001 } .
The dropout of input is set to 0.6, of edge is set to 0.3.
Notably, the results reported in our paper are comparable with those in their original papers.
In the implementation of subgraph retrieval, the graph patterns in the implementation are reported in Table 2, and we search and retrieve all the supporting cases of 3-cycle and sample the same number of the supporting cases of 4-cycle, whose numbers are reported in Table A3.
The number of the refuting cases of 3-cycle and 4-cycle is twice of the number of the supporting cases.
The number of target subgraphs K is set as 32 and the number of analogy subgraphs Q (for both supporting and refuting patterns) is set as 64.
The experimental comparisons are conducted with 10 different random seeds.
We report the main statistics of patterns in each datasets in Table A3, which includes the numbers of 3-cycles and 4-cycles.
We further report the buffer sizes of each dataset as follows, which is in proportion to the numbers of the patterns.
The buffer sizes of the supporting subgraphs of 3-cycles are: 4.5MB for LastFM, 8.7MB for Amazon, 21.2MB for Yelp, 96.2MB for Douban Book, 80.5MB for FB15k-237, 161.4MB for WN18RR where we sample twice number of the supporting subgraphs of 3-cycles in FB15k-237 for WN18RR.
And, the buffer size of the supporting subgraphs of 4-cycles, and the refuting subgraphs of 3-cycles and 4-cycles can be easily derived by the above buffer size, since we store the node IDs into the buffer.
A5. H ARDWARE S ETTINGS
The models are trained under the same hardware settings with an Amazon EC2 p3.8 × large instance 1 , where the GPU processor is NVIDIA Tesla V100 processor and the CPU processor is Intel Xeon
E5-2686 v4 (Broadwell) processor.
[Table]
[Table]
[Table]
GraphANGEL 3 − cycle 0.
[Table]
[Table]
[Table]
580 GraphANGEL 4 − cycle
[Table]
[Table]
[Table]
[Table]
[Table]
574 GraphANGEL 0.
[Table]
[Table] [Table]
[Table] [Table] [Table]
[Table]
A A DDITIONAL R ESULTS A6. A DDITIONAL R ESULTS OF S TANDARD T ASK
We show the results of comparisons between GraphANGEL against the baseline methods testing with 20% least frequent relations in terms of MR, MRR, Hit@1, Hit@3 and Hit@10 in Table A5.
A6. A DDITIONAL R ESULTS OF G ENERALIZATION S TUDY
Table A6 shows the results of performance in terms of MR, MRR, Hit@K (K=1,3,10) after randomly removing 5% relations.
We also report the results after randomly dropping 10%, 15% and 20% in Tables A7, A8 and A9.
In addition to randomly dropping the relations, we also evaluate the performance when we randomly adding 5% relations by combining the existing relations through Composition
Rules, Inverse Rules, Symmetric Rules, Subrelation Rules discussed in (Qu & Tang, 2019).
The results are reported in Table A10.
These experimental results verify that GraphANGEL is signiﬁcantly less affected by dropping or adding relations, which demonstrates the better generalizability of our model to unseen relation types against these state-of-the-art methods.
A6. A DDITIONAL R ESULTS OF H EAT M AP
We show the detailed attention heat map of Douban Book dataset in Figure A5 and Figure 5 in the main text is a simpliﬁcation of this ﬁgure.
One can see that based on the heat map, we can not only have a global view for each graph pattern, but also see the importance of each subgraph.
Speciﬁcally, Figure A5 visualizes the attention heat map between target subgraphs and supporting subgraphs when predicting relations between users and books.
The rows represent the supporting subgraphs while the columns represent the target subgraphs.
We denote each subgraph as a tuple of node IDs, e.g., S +7 = ( u 107 , b 11 , a 33 , b 12 ) .
Each cell represents the similarity between a target subgraph (at the top) and a supporting subgraph (at bottom).
The color of each cell shows the attention weight for corresponding pair of supporting and target subgraphs.
We can observe that the deep color of the cell located at the interaction of S ∗ 7 and S +32 indicates more similarity.
We can see from the ﬁgure that the cells with darkest color have both target subgraph and supporting subgraph as ( u · , b · , a · , b · ) , which indicates a subgraph with User − Book − Author − Book pattern, meaning that this particular pattern is more inﬂuential in predicting User − Book relation.
Notably, this result reveals a simple logic User ∧ Book
∧ Author ∧ Book ⇒ User ∧ Book indicating a simple factor that the users are likely to be interested in the book sharing the same authors with his reading ones.
Hence, our method can intuitively be regarded as the a logical collaborative ﬁltering technique working on graph.
Similarly, we can ﬁnd that 3-cycle pattern ( Person ) − Impersonate − 1 − ( Person ) ∧ ( Person ) − Nationality − ( Nation )
⇒
( Person ) − Nationality − ( Nation ) plays an important role in FB15k-237 dataset, where inverse relations are denoted with a superscript − 1 .
A6. A DDITIONAL R ESULTS OF A BLATION S TUDY OF S UBGRAPH E MBEDDING M ETHODS
In Section 3, we use R-GCN (Schlichtkrull et al., 2018) with single layer (denoted as
GraphANGEL ∗ 1 R − GCN ) as an example of Φ .
In this subsection, we further investigate the effect of Φ 𝑢 !" # ,𝑏 !$% , 𝑎 !"! 𝑏 !& 𝑢 !" # ,𝑏 !$% , 𝑎 !"!
𝑏 !& 𝑢 !"
# ,𝑏 !$% , 𝑎 !"!
𝑏 !& 𝑢 !"
# ,𝑏 !$% , 𝑎 !"!
𝑏 !& 𝑢 !"
# ,𝑏 !$% , 𝑎 !"!
𝑏 !& 𝑢 !"
# ,𝑏 !$% , 𝑎 !"!
𝑏 !& 𝑢 !"
# ,𝑏 !
# , 𝑎 $’& ,𝑏 !& 𝑢 !"
# ,𝑏 !
# , 𝑎 $’& ,𝑏 !& 𝑢 !"
# ,𝑏 !
# , 𝑎 $’& ,𝑏 !& 𝑢 !"
# ,𝑏 !
# , 𝑎 $’& ,𝑏 !& 𝑢 !"
# ,𝑏 !
# , 𝑎 $’& ,𝑏 !& 𝑢 !"
# ,𝑏 !
# , 𝑎 $’& ,𝑏 !& 𝑢 !"
# ,𝑏 !!
, 𝑎 ’’ ,𝑏 !& 𝑢 !"
# ,𝑏 !!
, 𝑎 ’’ ,𝑏 !& 𝑢 !"
# ,𝑏 !!
, 𝑎 ’’ ,𝑏 !& 𝑢 !"
# ,𝑏 !!
, 𝑎 ’’ ,𝑏 !& 𝑢 !"
# ,𝑏 !!
, 𝑎 ’’ ,𝑏 !& 𝑢 !"
# ,𝑏 !!
, 𝑎 ’’ ,𝑏 !& 𝑢 !"
# ,𝑏 !$% , 𝑎 %(# ,𝑏 !& 𝑢 !"
# ,𝑏 !$% , 𝑎 %(# ,𝑏 !& 𝑢 !"
# ,𝑏 !$% , 𝑎 %(# ,𝑏 !& 𝑢 !"
# ,𝑏 !$% , 𝑎 %(# ,𝑏 !& 𝑢 !"
# ,𝑏 !$% , 𝑎 %(# ,𝑏 !& 𝑢 !"
# ,𝑏 !$% , 𝑎 %(# ,𝑏 !& 𝑢 !"
# ,𝑏 !$% , 𝑦 ! ,𝑏 !& 𝑢 !"
# ,𝑏 !$% , 𝑦 ! ,𝑏 !& 𝑢 !"
# ,𝑏 !$% , 𝑦 ! ,𝑏 !& 𝑢 !"
[Table]
[Table]
[Table]
𝑢 !"
# ,𝑏
[Table]
[Table]
[Table]
[Table]
𝑏
[Table]
𝑢 !"
[Table]
𝑏
[Table]
[Table]
[Table]
[Table]
𝑢
[Table]
[Table]
[Table]
% ,𝑢 ## , 𝑢 *! ,𝑏 %’( 𝑢 #(% ,𝑢 ## , 𝑢 *! ,𝑏 %’( 𝑢 #(% ,𝑢 ## , 𝑢 *! ,𝑏 %’( 𝑢 #(% ,𝑢 ## , 𝑢 *! ,𝑏 %’( 𝑢 #(% ,𝑢 ## , 𝑢 *! ,𝑏 %’( 𝑢 )
#% ,𝑢 )’$ , 𝑢 %(& ,𝑏 !
$( 𝑢 )#% ,𝑢 )’$ , 𝑢 %(& ,𝑏 !
$( 𝑢 )#% ,𝑢 )’$ , 𝑢 %(& ,𝑏 !
$( 𝑢 )#% ,𝑢 )’$ , 𝑢 %(& ,𝑏 !
$( 𝑢 )#% ,𝑢 )’$ , 𝑢 %(& ,𝑏 !
$( 𝑢 )#% ,𝑢 )’$ , 𝑢 %(& ,𝑏 !
$( 𝑢 $)* ,𝑢 *#" , 𝑢 ’&( ,𝑏 %’ 𝑢 $)* ,𝑢 *#" , 𝑢 ’&( ,𝑏 %’ 𝑢 $)* ,𝑢 *#" , 𝑢 ’&( ,𝑏 %’ 𝑢 $)* ,𝑢 *#" , 𝑢 ’&( ,𝑏 %’ 𝑢 $)* ,𝑢 *#" , 𝑢 ’&( ,𝑏 %’ 𝑢 $)* ,𝑢 *#" , 𝑢 ’&( ,𝑏 %’ by applying different GNNs including GCN (Kipf & Welling, 2016) (denoted as GraphANGEL ∗ 1 GCN ),
GAT (Hamilton et al., 2017) (denoted as GraphANGEL ∗ 1 GAT ), R-GAT (Busbridge et al., 2019) (denoted as GraphANGEL ∗ 1 R − GAT ).
We do not choose knowledge embedding based methods including TransE (Bordes et al., 2013), RotatE (Sun et al., 2019a), ComplEx (Trouillon et al., 2016), as these approaches need to introduce additional supervisions/gredients on the subgraph embeddings, which require signiﬁcant modiﬁcations on the framework of GraphANGEL.
We do not change any other settings of
GraphANGEL for fair comparison, and report the results on heterogeneous recommendation task and knowledge graph completion task in Tables A11 and A12 respectively.
Results shows that there are slight differences between GraphANGEL ∗ 1 R − GCN and GraphANGEL ∗ 1 R − GAT .
One possible explanation is that as our retrieved subgraphs only includes few nodes (e.g. 2,3,or 4 nodes), there is no requirement of GraphANGEL to mine some complex structure (e.g., high-order neighborhood).
We further study the impact of using the multiple layers of GNNs.
We establish a series of variants using two layers of of R-GCN and R-GAT, denoted as GraphANGEL ∗ 2 R − GCN and GraphANGEL ∗ 2 R − GAT .
Results reported in Tables A11 and A12 show that there are only slight differences when comparing to GraphANGEL ∗ 1 R − GCN and GraphANGEL ∗ 1 R − GAT , which is consistent with the above explanation.
A6. A DDITIONAL R ESULTS OF E FFECT OF N OISY G RAPHS
In order to further investigate the robustness of GraphANGEL, we randomly generate some noises and add them into the training graphs.
Concretely, for heterogeneous graphs and knowledge graphs shown in Tables A1 and A2, we randomly change the source or target node of each triplet (cid:104) s, r, t (cid:105) with probability 5 % .
Since the heterogeneous graphs have the node types, our changes are made by replacing the current source or target node with a randomly sampled node in the same node type.
We compare our model with all the baselines and report the results for heterogeneous graph recommendation and knowledge graph completion tasks in Tables A13 and A14.
These results show that our method can consistently outperform these baseline methods, sometimes, even are less inﬂuenced by the noises.
One explanation is that if we introduce noises in the labels y , then the learnable subgraph embeddings s + and s − will simultaneously be inﬂuenced by the noises.
These inﬂuences might be counteracted according to Eq.
A6. A DDITIONAL R ESULTS OF A BLATION S TUDY OF B UFFER S IZE
We establish an ablation study to investigate the impact of the buffer size B in Algorithm 8.
We here focus on the number of the subgraphs following Π + 3 − cycle and Π + 4 − cycle .
From Appendix A3, we know that we search and sample N 3 − cycle subgraphs following Π + 3 − cycle and Π + 4 − cycle , where
[Figure]
N 3 − cycle is the number of subgraphs following Π + 3 − cycle .
We reduce the number of either following Π + 3 − cycle or Π + 4 − cycle .
Results reported in Figure A6 shows that once the numbers of supporting 3-cycles and 4-cycles surpass 0.5 · N 3 − cycle and 0.75 · N 3 − cycle respectively, there will be no signiﬁcant difference to further increase the buffer size [Figure] r a L n L n g T L m H ( s H c R n
G )
[Figure]
A6. A DDITIONAL R ESULTS OF T IME C OMPLEXITY
We ﬁrst compare the time complexity of GraphANGEL in term of both training and inference times on Amazon dataset against aforementioned baseline methods.
Result in Figure A7(a) reveals that
GraphANGEL is more efﬁcient than HGT and HAN in term of training time and is more efﬁcient than R-GCN and HGT in term of inference time.
One possible explanation is that HAN, HGT, and
R-GCN contain the whole graph, which is much more complex than sampled subgraphs.
We further evaluate the training and inference time of GraphANGEL against the baselines on FB15k-237 dataset and report the results in Figure A7(b).
As the prevailing methods on knowledge completion task are mainly based on either on translation (Bordes et al., 2013) or rotation (Sun et al., 2019a) assumptions regardless of the graph structure, they are naturally more efﬁcient than GNN based relation prediction models.
We do note that our efﬁciency is comparable against other graph neural networks (i.e.,
R-GCN) which have also been tried on knowledge graphs.
B ← B old for each v ∈ ∆ V − do
Remove all the instances including v in B .
end
# Following is Search and Retrieval for Π + 3 − cycle in ∆ G + new
Search, retrieve and store the instances following Π + 3 − cycle in ∆ G + new using Algorithm 2 or 3.
# Followings is Search and Retrieval for Π + 3 − cycle in (a) Shape
Search, retrieve and store the instances following Π + 3 − cycle in (a) shape using Algorithm 9.
# Followings is Search and Retrieval for Π + 3 − cycle in (b) Shape for each u ∈ V old do for each (cid:104) v, w
(cid:105) ∈
∆ V + new do if (cid:104) u, v
(cid:105) ∈
∆ E + old and (cid:104) u, w (cid:105) ∈
∆ E + old then
Add ( u, v, w ) into B .
end end end
# Following is Uniform Sampling for Π
+ 4 − cycle in ∆ G + new
Sample and store [ | ∆ V + new | 4 |V old | 4 n 4 − cycle ] instances following Π + 4 − cycle in ∆ G + new using Algorithm 5.
# Following is Uniform Sampling for Π + 4 − cycle in (c), (d), (e) Shapes
Compute sample rate p edge ∆ G + old by running purple part in Algorithm 10 on ∆ G + old .
for i ← 1 to [ | ∆ E
+ old | 2 |E old | 2 n 4 − cycle ] do
Sample (cid:104) u, v (cid:105) ∈
∆ E + old with probability p edge ∆ G + old .
Sample (cid:104) w, x (cid:105) ∈
∆ E + old with probability p edge ∆ G + old .
if { (cid:104) u, w (cid:105) ∈
E old ∪ ∆ E + new and (cid:104) v, x
(cid:105) ∈
E old ∪ ∆ E + new } or { (cid:104) u, w
(cid:105) ∈
E old ∪ ∆ E + new and (cid:104) v, x
(cid:105) ∈
E old ∪ ∆ E + new } then
Add ( u.v, w, x ) into B .
end end
# Following is Uniform Sampling for Π + 4 − cycle in (f), (g) Shapes
Compute sample rate p edge G old and p node ∆ G + new by running blue part in Algorithm 10 on G old and ∆ G + new .
for i ← 1 to [ | ∆ E
+ old | 2 |E old | 2 n 4 − cycle ] do
Sample x ∈ ∆ V + new with probability p node ∆ G + new .
Sample v ∈ V old with probability p node G old .
Uniformly sample two different neighbor nodes u, w ∈ V old of v .
if (cid:104) u, x (cid:105) ∈
∆ E + old and (cid:104) w, x (cid:105) ∈
∆ E + old then
Add ( u, v, w, x ) into B .
end end for i ← 1 to [ | ∆ E
+ old | 2 | ∆ E + new | 2 |E old |n 4 − cycle ] do
Sample x ∈ V old with probability p node G old .
Sample v ∈ ∆ V + new with probability p node ∆ G + new .
Uniformly sample two different neighbor nodes u, w ∈ ∆ V + new of v .
if (cid:104) u, x
(cid:105) ∈
E + old and (cid:104) w, x (cid:105) ∈ E + old then
Add ( u, v, w, x ) into B .
end end
# Following is Uniform Sampling for Π − 3 − cycle
and Π − 4 − cycle
Re-sample and store [ |E new | 3 |E old | 3 n − 3 − cycle ] and [ |E new | 3 |E old | 3 n − 4 − cycle ] refuting cases of Π − 3 − cycle
and Π − 4 − cycle from G new using Algorithms 7 and 5 with linear computation costs.