SAFRAN: An interpretable, rule-based link prediction method outperforming embedding models
Anonymous authors
Abstract
Neural embedding-based machine learning models have shown promise for predicting novel links in knowledge graphs.
Unfortunately, their practical utility is diminished by their lack of interpretability.
Recently, the fully interpretable, rule-based algorithm AnyBURL yielded highly competitive results on many general-purpose link prediction benchmarks.
However, current approaches for aggregating predictions made by multiple rules are affected by redundancies.
We improve upon AnyBURL by introducing the SAFRAN rule application framework, which uses a novel aggregation approach called Non-redundant
Noisy-OR that detects and clusters redundant rules prior to aggregation.
SAFRAN yields new state-of-the-art results for fully interpretable link prediction on the established generalpurpose benchmarks FB15K-237, WN18RR and YAGO3-10.
Furthermore, it exceeds the results of multiple established embedding-based algorithms on FB15K-237 and WN18RR and narrows the gap between rule-based and embedding-based algorithms on YAGO3-10.
1. Introduction
Link prediction in knowledge graphs is a vibrant area of research and development and encompasses a wide range of diﬀerent approaches.
Currently, most state-of-the-art approaches use low-dimensional representations (embeddings) of knowledge graphs to predict new knowledge.
However, as such approaches are black boxes, the validity of their predictions can be questioned due to lack of interpretability.
Recent research has focused on learning symbolic rules from knowledge graphs that try to detect correlations of entities in knowledge bases and express them as ﬁrst-order Horn rules.
In contrast to neural models, such rule-based methods are fully transparent and predictions can be explained intuitively.
However, while current research on rule-based approaches puts a lot of emphasis on designing rule-mining algorithms, research on methods for applying learned rules and aggregating their predictions in a meaningful way is comparatively neglected.
AnyBURL [Meilicke et al., 2019, 2020] is currently one of the best performing symbolic rule learners and demonstrated results competitive with embedding methods on link prediction benchmarks such as FB15k-237.
However, the application of rules generated by Any-
BURL is currently limited by functional redundancies, making it diﬃcult to meaningfully aggregate predictions made by diﬀerent rules and thereby limiting predictive performance.
In this paper, we introduce the high-performance rule application framework SAFRAN (’Scalable And fast non-redundant rule application’) that employs a novel aggregation approach.
We propose an algorithm called Non-redundant Noisy-OR that detects redundant rules prior to aggregation, mitigating their negative eﬀects and improving predictive performance.
This algorithm for aggregating predictions is not necessarily restricted to rules learned by AnyBURL and could also be applied to other rule-based systems.
2. Preliminaries 2.1 Link prediction
Knowledge graphs contain facts, such as “Amsterdam is the capital of the Netherlands” or “Marie Curie was born in Warsaw”, which are represented as semantic triples containing a subject (head h ), predicate (relation r ) and object (tail t ).
Formally, a knowledge graph is a directed heterogeneous multigraph G =
( E, R, T ) where E and R are the set of entities and relations respectively and T is a set of triples { ( h, r, t ) } ⊆
E × R × E .
The task of link prediction is to predict either the tail of a given head and relation ( h, r, ?) or the head of a given tail and relation (? , r, t ) such that the resulting triple describes a correct fact which has not been in G before.
2.2 AnyBURL
We now give an overview about the main AnyBURL algorithm, which is explained in detail in [Meilicke et al., 2019] and [Meilicke et al., 2020].
AnyBURL (Anytime Bottom Up
Rule Learning) is a novel walk-based method and is currently one of the best symbolic rule learners, competing with current state-of-the-art embedding approaches [Rossi et al., 2021].
Walk-based (bottom-up) methods are based on the idea that sampled paths from a knowledge graph (random walks) are examples of very speciﬁc rules, which can be transformed into more general rules.
Learned rules can then be applied to the knowledge graph for the prediction of novel links between entities. [Table]
AnyBURL learns rules in the form of ﬁrst-order logic Horn rules of length n .
In each iteration of the algorithm for mining rules the algorithm samples a random triple h ( c 0 , c 1 ), f [Table]
e. lives ( max, uk ) from the training set. Starting from either the head or the tail of this triple, AnyBURL performs a random walk of length n , resulting in a ground path in the form of h ( c 0 , c 1 ) ← b 1 ( c 1 , c 2 ) , . . .
, b n ( c n , c n +1 ), f.e. speaks ( max, english ) ← lives ( max, uk ) , lang ( uk, english ).
The sampled ground path is subsequently generalized into three predeﬁned types of rules shown in Table 1.
Cyclic rules can be generalized from cyclic paths {+( c 
+}
=
c n +1 ), while AC2 rules can be generalized from acyclic paths ( c 0 (cid:54)
=
c n +1 ).
AC1 rules are hybrid, as they can be both generalized from cyclic (if c [-
-]0 = c n +1 ) and acyclic (if c 0 (cid:54) =
c n +1 ) paths.
All rules that can be generalized from the previous example ground path can be seen in Table 2.
[Table]
The following applies for inferring a set of head triples ˆ H r from the body of a rule r : ˆ H r = [Equation] if type [Equation]
if
[Equation]
AC
[Equation] +1 ) ∈ T } , if type = AC AnyBURL calculates the conﬁdence of a rule as the size of the union of inferred triples by a rule and the triples of the training set divided by the number of inferred triples conf ( r ) =
| ˆ H r ∩ T | / | ˆ H r | , where T is the set of triples contained in the training set.
2.3 Aggregation
Rule-based link prediction methods, such as AnyBURL, produce many rules with diﬀerent conﬁdences that are subsequently applied to the knowledge graph in order to generate candidates for a link prediction task.
Upon a link prediction task p ( s, ?), rule r generates the set of predictions { y | p ( s,
y ) ∈
ˆ H r } that could act as substitutions for the question mark.
Rules can either predict zero, one or multiple entities.
As the same entities can be proposed by multiple rules which may diﬀer in conﬁdence, an aggregation of these conﬁdences is required to assess the ﬁnal conﬁdence of a prediction.
Table 3 shows the conﬁdences and results of ﬁve ﬁctional rules.
[Table]
One aggregation approach called Maximum aggregation is using the maximum of these conﬁdences [Meilicke et al., 2019].
The score of an entity
[Equation] is the maximum conﬁdence of all rules r 1 , . . .
, r k that predict entity e .
If the maximum conﬁdences of two or more entities are the same, these entities are further ranked by their second best conﬁdence and so on, until all top-k candidates can be distinguished or all rules are processed.
This approach results in the following list of ranked candidates when applied to the solutions of Table 3: ranking max = (cid:104) ( b, 0 . 9) , ( a, 0 . 9) , ( c, 0 .
8) (cid:105) .
As the second best rule that proposes b has a higher conﬁdence than the second best rule which proposes a , b is ranked before a .
Unfortunately, the Maximum aggregation is constrained to relatively simple predictions:
Each prediction is primarily informed by only a single rule with the highest conﬁdence; it is not possible to make predictions based on a weighted combination of diﬀerent rules.
These limitations of maximum aggregation are potentially addressed by noisy-or aggregation, ﬁrst proposed in this context by [Gal´arraga et al., 2015].
Instead of taking the maximum of conﬁdences, the probability that at least one of the rules proposed the correct candidate is used for generating a list of ranked candidates.
The score of an entity e can thus be calculated as
[Equation] where r 1 , . . .
, r k are rules that predict entity e .
Applying the Noisy-OR approach to the results of Table 3, the entities are ranked as follows: ranking noisy = (cid:104) ( c, 0 . 94) , ( b, 0 . 93) , ( a, 0 . 9) (cid:105) .
This ranking diﬀers from the maximum approach.
As c is proposed by the second and third most conﬁdent rules, its aggregated conﬁdence is higher than a or b , entities proposed by the most conﬁdent rule.
In theory, Noisy-OR allows for more sophisticated conﬁdence aggregation.
However, it assumes independence/non-redundancy of all rules.
As most rules that can be derived from real-world knowledge bases are in some way redundant with other rules, Noisy-OR was found to perform worse than maximum aggregation in practice.
2.4 The detrimental eﬀect of redundancy on Noisy-OR aggregation
Redundancies in rules can lead to overestimation of conﬁdences of predicted entities when aggregating using Noisy-OR.
As two redundant rules generate the same predictions for the same reasons, the conﬁdences of entities predicted by both rules are overestimated due to double counting.
Consider the rules shown in Table 4 where this problem becomes apparent.
If the training set contains the ground path lives ( john, uk ) , lang ( uk, english ), each rule would predict speaks ( john, english ) upon the prediction task speaks ( john, ?) for the same reasons.
Aggregating the conﬁdences of all three rules using Noisy-OR would result in an overestimated conﬁdence of 0 .
988 for the entity english .
While these redundancies are apparent, redundancies between rules are often less obvious in practice.
Redundancy between two rules can arise from special relationships between relations, such as relations that are nearly equivalent, symmetric relations or relations entailing other relations.
Consider the rules shown in Table 5.
The ﬁrst example shows two [Table] rules being redundant based on entailment, as every capital of a country is a city within that country.
The second example shows redundancy based on the symmetric relationship married .
Both rules generate predictions for languages spoken by a person, based on what language its spouse speaks.
The third example shows that such relationships are not restricted to single relations, but can arise from a combination of relations, as the combination parent → brother is a duplicate of the relation uncle . [Table]
It is important to understand that there is usually no explicit information available about the symmetry of married nor do we know that capital is more speciﬁc than city .
It is not even possible to grasp this information from the dataset itself as it might be incomplete and noisy.
This makes it impossible to decide redundancy via a procedure that makes use of relevant background knowledge
[Table]
3. Approach 3.1 Algorithm
To overcome the problem of redundancy when using the Noisy-OR aggregation method, we propose an approach to cluster rules based on their redundancy degree prior to aggregation.
Predictions of rules in a cluster are aggregated using the Maximum approach, as this approach is not susceptible to redundancies.
Predictions of the diﬀerent clusters are then further aggregated using the Noisy-OR approach.
We call the resulting aggregation technique Non-redundant Noisy-OR . A pseudocode of Non-redundant Noisy-OR can be seen in Appendix G.
As a metric for redundancy between two rules r i , r j the Jaccard Index sim ( r i , r j ) = | ˆ H r i ∩
ˆ H r
j | / | ˆ H r
i ∪ ˆ H r
j | of the sets of inferred triples is used.
As the calculation of the
Jaccard coeﬃcient is very ineﬃcient for large sets, the Jaccard coeﬃcient is estimated using the MinHash scheme [Broder, 1997], which makes time complexity linear and memory usage constant.
Naturally rules are pre-clustered by their relation of the head atom, as two rules having a diﬀerent relation in the head are never used in the same prediction task and the union of their inferred triples is always empty h r i (cid:54) =
h r j
→ ˆ H r i ∩ ˆ H r j = ∅ .
Thus, similarity matrices of rules are calculated for each head relation and used to cluster the rules.
Two rules r i , r j are considered redundant and assigned to the same cluster if sim ( r i , r j ) > t , where t is a threshold.
However a global threshold t is not capable of deﬁning the cutoﬀ point for redundancy for all relations, rule types and directions of prediction (prediction of head entities or tail entities).
As the sets of rules that predict for diﬀerent relations may vary in distribution of type, length or speciﬁcity, the threshold that generates the optimal clustering for a relation may not generate the optimal clustering for a diﬀerent relation.
Furthermore as relations may vary in multiplicity the direction of the prediction has to taken into account.
Therefore, a distinct threshold for the prediction of heads and prediction of tails for each relation is required in order to be able to optimally cluster an entire rule set.
Furthermore, within a set of rules that predict for the same relation, optimal thresholds may vary between types of rules.
As can be seen by the examples in Table 4, the type combination of C and AC1 often tends to form set-subset relation.
The Jaccard Index is very insensitve to such relationships.
While e.g. a Jaccard index of 0.25 between a rule of type C and a rule of type AC1 may indicate a high redundancy due to the dominance of set-subset relationships within rules of this type combination, it may be considered as a fairly low evidence level for a redundancy between two rules of type C, where such relationships are not as prominent.
Thus a diﬀerentiation of rule types is needed when deﬁning the cutoﬀ value for redundancy between two rules of a certain relation.
For each combination of rule types an independent threshold parameter is used to determine redundancy.
Conclusively, the threshold of similarity for rules that predict entities of direction d (head or tail) of a h -triple
(we use h to refer to a relation, while r is used to refer to rules) is given by t hd =
[ t C/C , t C/AC 1 , t C/AC 2 , t AC 1 /AC 2 , t AC 1 /AC 1 , t AC 2 /AC 2 ].
Note that if t hd =
[0 , 0 , 0 , 0 , 0 , 0] only the Maximum aggregation is used and if t hd =
[1 , 1 , 1 , 1 , 1 , 1] only Noisy-OR.
An optimal clustering of rules that predict h -triples is given by the type combination thresholds t hd that maximize the ﬁtness of the clusters for predicting new links.
This ﬁtness is evaluated on the validation set using the mean-reciprocal rank based on the topk predicted entities.
SAFRAN uses one of two search strategies for ﬁnding the optimal thresholds: grid search (parameter sweep) and random search.
For grid search the range of possible thresholds [0 . 0 , 1 . 0] is divided by n equally distant steps and each threshold is subsequently used for clustering.
For the grid search, we do not distinguish between rule-type speciﬁc parameters but use a single relation-speciﬁc ﬁx parameter t hd to limit the search space.
Contrary to this, the random search randomly samples the six rule-typespeciﬁc thresholds of t hd .
3.2 An Example and its Explanation
Figure 1 demonstrates the eﬀect of our algorithm.
It has two parts that both illustrate the list of predicted entities for the prediction task genre artist(?, Bryan Adams) .
The upper half of the ﬁgure shows the result of applying the Non-redundant Noisy-OR approach, while the lower part presents the results of the maximum aggregation.
In addition to the candidate lists some explanations in terms of the rules or rule clusters that predict these entities are shown.
For the Non-redundant Noisy-OR we depict the two clusters that have the highest maximum conﬁdence.
By aggregating these clusters via the Noisy-OR multiplication, the highest rank candidate is Pop rock .
A diﬀerent result can be observed for the maximum aggregation.
The correct entity Pop rock is ranked at the third position, while the ﬁrst position is occupied by Heavy Metal .
The example illustrates nicely that the inﬂuence of a single rule can be too strong in the maximum approach, while the Non-redundant Noisy-OR approach allows to aggregate the results in a more meaningful way.
This holds in particular if the clusters make sense and combine similar reasons for a prediction within the same cluster.
Therefore, in addition to improving predictive performance, our algorithm provides explanations that are easier to understand and that provide deeper insights about the underlying data and the rules derived from them.
4. Experiments
We conducted experiments with three established link prediction datasets, including FB15k237 [Toutanova and Chen, 2015], WN18RR
[Dettmers et al., 2018] and YAGO3-10 [Dettmers et al., 2018].
An overview of these datasets can be seen in Table 6.
For our experiments we selected only datasets without train-test leakage.
We abstained from performing experi- [Table] ments on the parent datasets FB15k (WN18) [Bordes et al., 2013] of FB15k-237 (WN18RR) as they were found to suﬀer from heavy testing leakage (f.e.
some triples in the test data are present as inverse triples in the training data).
Training, inferencing and evaluations were run on a machine with 24 physical (48 logical)
Intel(R)
Xeon(R) CPU E5-2650 v4 @
2.20GHz cores and 264 GB of RAM.
As SAFRAN and
AnyBURL do not utilize GPU-accelerated computing, only CPUs were used.
For optimal comparison with the best results using the maximum aggregation approach presented in [Meilicke et al., 2020], the same learning parameters were adopted for the same datasets.
Rulesets used for the results were learned for 1000 seconds each, using 22 threads.
For rules learned from WN18RR the maximum length of cyclic rules was set to 5, while for all other datasets a maximum length of 3 was used.
The maximum length of acyclic rules was set to 1 for all datasets.
As FB15k-237 contains reﬂexive triples in the form of r ( c, c ) a ﬂag was set that allows AnyBURL to use such reﬂexive triples for learning rules.
The same rulesets were used for inference and evaluation with AnyBURL and SAFRAN.
Random search was performed using a n of 10 and 10000 iterations, while grid search was performed with a n of 200.
We compare our results with the established latent models RESCAL [Nickel et al., 2011],
TransE [Bordes et al., 2013], DistMult
[Yang et al., 2015], ComplEx [Trouillon et al., 2016],
ConvE [Dettmers et al., 2018], RotatE
[Sun et al., 2019], TuckER [Balazevic et al., 2019] and the rule-based methods AMIE+ [Gal´arraga et al., 2015], RuleN [Meilicke et al., 2018],
C-NN [Ferr´e, 2020], Neural LP
[Yang et al., 2017], RLvLR [Omran et al., 2018], DRUM [Sadeghian et al., 2019], GPFL
[Gu et al., 2020b].
For details on these approaches we refer to their respective original papers.
For each approach we report the ﬁltered mean reciprocal rank (MRR), ﬁltered hits@1 and ﬁltered hits@10, where predicted triples that are already known are ﬁltered out, except the test triple itself.
The results for FB15k-237, WN18RR and YAGO3-10 can be seen in Table 7.
All results reported by us were evaluated according to the average policy [Rossi et al., 2021] (see Appendix A), where the rank of a target entity within a group of same score entities is the average rank of this group.
Reported results from [Rossi et al., 2021], [Ruﬃnelli et al., 2020] were evaluated using the same protocol.
Results of
C-NN, GPFL, AMIE+, RuleN were evaluated using ordinal (random) policy which results in very similar results than average policy, while to the best of our knowledge there is no information available about the policy used for evaluating RLvLR.
Results of DRUM and
Neural LP were evaluated using the top policy which is biased towards producing better evaluations results and cannot be compared directly to other results. Our aggregation method in the more exhaustive random search outperforms both symbolic and subsymbolic approaches on FB15k-237 and WN18RR. This is a surprising result as both data sets have been used for years to evaluate and improve performance of latent methods that are build on the concept of embeddings. On YAGO3-10 the results are slightly worse.
For this data set ComplEx achieves an MRR that is 0.012 higher.
However, AnyBURL together with the Non-redundant Noisy-OR is still among the top performing techniques. With our approach we oﬀer a fully interpretable symbolic approach, that allows to explain the results in terms of the rules that generated them, while we are still able to compete with or even outperform latent approaches. The improvement against the
AnyBURL default maximum aggregation technique diﬀers a lot between datasets.
While we observe a clear improvement of more than 3% in terms of MRR for FB15k-237, the improvement for the other datasets is around 0.5% to 1%.
Currently we do not understand the reason for this diﬀerences.
The most probable explanation is a lower degree of redundancy in the rules that describe the regularities encoded in these datasets.
Another reason for this may be the small validation set sizes of WN18RR and YAGO310.
While the training set of YAGO3-10 is the biggest of the three datasets used in the experiments, the size of its validation set is only 0.46% of the dataset, compared to 6.6% in FB15k-237.
However, non-redundant Noisy-OR needs an appropriate sized validation set that represents a broad spectrum of relations and entities to be able to fully recognize redundancies.
The validation set size of WN18RR is 3.4% of the dataset.
However as the overall dataset is considerably small, the absolute size of its validation set is only 3, triples.
As rules generated by AnyBURL can generalize to unseen entities [Ferr´e, 2020], this size may not be suﬃcient to generate the most optimal clustering.
[Table]
t [Table] (cid:63) [Table]
[Table] [Table] .673 DistMult (cid:63) . .661 ComplEx (cid:63) .
.704 ConvE (cid:63) .
.657 RotatE ¶ .
.670 TuckER ¶ .
.680 HAKE (cid:53) .
I n t e r p r e t a b l e
C-NN (cid:52) .
.519 DRUM ‡ .343 † .255 † .516 † .486 † .425 † .586 †
Neural LP ♦ .240 † .362 † .435 † .371 † .566 †
GPFL ♦ .
.552 AMIE+ ♣ .
.388 RuleN ♣ .
.536 RLvLR (cid:93) .
AnyBURLwithSAFRAN
Maximum .
.686 Noisy-OR .
[Table]
[Table]
[Table]
[Table]
In our experiments we used two diﬀerent search techniques.
The random search performs better in most cases.
As explained above, it learns thresholds that distinguish between diﬀerent rule types.
Thus, only a random selection of all possible threshold combinations is visited during the search.
Nevertheless, type-speciﬁc thresholds seem to make sense as the results are in most cases slightly better compared to the grid search, which learns a single threshold per relation.
To support our claim that Non-redundant Noisy-OR could potentially be applied to other rule-based approaches, we performed experiments on all used data sets using rules learned by AMIE+.
The results can be seen in Appendix
F.
We used AMIE in its default setting.
This means that it mines only cyclic rules.
This could be the reason that we observed no or only a very limited improvement on each data sets with the exception of
FB15k-237.
To ascertain the importance of optimal thresholds, we performed another experiment.
We evaluated each relation in the validation set using Noisy-OR and Maximum aggregation and apply the approach with the maximum MRR of each relation to the test set.
Results for the datasets can be seen in Table 7 denoted as VS.
All metrics improved compared to the Maximum aggregation but could not achieve the gain of Non-redundant Noisy-OR.
5. Related Work
One of the most prominent rule mining systems is AMIE, which has been described ﬁrst in [Gal´arraga et al., 2015] and later, including several extension and improvements in [Lajus et al., 2020].
These papers focus mainly on mining rules (including the conﬁdence computation) rather than their application.
This is typical for most rule-mining or inductive logic programming (ILP) papers.
The authors touch the topic of how to aggregate conﬁdences only in a single experiment where they compare maximum and the Noisy-OR aggregation.
According to their results the Noisy-OR approach works better.
In [Ferr´e, 2020] another interesting variation of Noisy-OR, the Dempster–Shafer score [Denoeux, 1995], has been applied.
This approach is not based on rules, but relies on another symbolic representation using concepts of nearest neighbours.
In their experiments the authors found that the aggregation using the Dempster–Shafer score performs at best 1.3 % worse than the maximum score [Ferr´e, 2020].
It might make sense to decouple the learning of rules from the model used for their application.
ProbLogic [De Raedt et al., 2007] oﬀers a well-deﬁned model for applying probabilistic rules to a given set of facts (or triples).
ProbLog evaluates probabilistic logic programs by combining Selective Linear Deﬁnite clause resolution (SLD-resolution) with methods for computing the probability of Boolean formulae.
Using ProbLog for applying the rules learned by a rule miner would be similar to the Noisy-OR aggregation we described above.
As ProbLog is based on a model-theoretic notion of entailment, predictions created by a rule might enable other rules to ﬁre, which might again result into further predictions (and so on).
While this makes sense from a conceptual point of view, such an approach results in runtime problems for larger datasets.
In [Kuˇzelka and Davis, 2020] the authors proposed Markov Logic as a powerful framework for solving knowledge base completion tasks.
Instead of learning rules, the authors propose to learn the weights of a Markov logic network.
As a consequence, dependencies between regularities are modeled in a complex way, which would lead to a non-trivial aggregation of evidences.
However, the analysis remains entirely theoretical and, so it is doubtful whether the approach is applicable to larger knowledge graphs.
The concept of rule redundancy is crucial for rule aggregation.
If two rules predict the same candidate for the same reason, a Noisy-OR aggregation makes no sense.
In [Gu et al., 2020a] a rule hierarchy was used to detect redundant rules.
However, the application model is based on the maximum aggregation strategy, which means that the removal has no positive impact on the predictive quality.
Indeed, the authors focus their experiments on runtime performance and report about a positive impact without a decrease in the predictive quality.
It is important to understand that the notion of rule redundancy is not limited to logical or extentional equivalence.
Approaches to detect equivalent duplicate rules are well known in ILP, and have for example been implemented in QuickFOIL [Zeng et al., 2014].
Instead of that, we have to deal with problems where two rules are only partially or nearly redundant.
We note that there are a few approaches that try to generate post-hoc explanations of predictions made by latent models such as knowledge graph embeddings. However, as opposed to AnyBURL/SAFRAN which provides ad-hoc explanations for every predicted triple, such approaches often fail to ﬁnd explanations. F.e. the approach proposed in [Zhang et al., 2019] applied to the knowledge graph embedding CrossE can only ﬁnd explanations for 40% of triples in the test set of the FB15K dataset.
Furthermore it is not fully transparent if generated explanations actually represent the reason for a prediction of a latent model.
6. Conclusion
Interpretable models drastically improve the trust and practical utility of predictions, and help to address current concerns about intransparency and unwanted bias in machine learning models.
We introduced a novel method for fully interpretable link prediction that yields new state-of-the-art results among interpretable methods and outperforms state-of-the-art embedding models on some of the established benchmarks.
Furthermore, our approach provides explanations that reveal clustering patterns in underlying data and derived rulesets.
While our approach can signiﬁcantly improve predictive performance of rule-based approaches, a more in-depth analysis is needed to assess the quality of generated clusters, which we plan to do in future work.
We hope that beyond their direct practical utility, our results help to reinvigorate interest into novel rule-based approaches to complement the embedding- and deep learning-based approaches that have been at the center of attention in machine learning research over the past years
Appendix A. Evaluation metrics and same score policy
We now formally deﬁne used metrics in our experiments.
Let t = 2 ∗ | T | be the number of prediction tasks in test set T and rank i be the rank of the correct entity in the predicted list of entities for completion task i , then Hits@k and
mean-reciprocal rank (MRR) are deﬁned as follows: Hits
[Equation]
t t (cid:88) i = 1 if rank i < k
MRR = 1 t t (cid:88) i =
rank i There are multiple policies for dealing with entities that have the same score as the correct entity [Rossi et al., 2021]:
• Top : The correct entity is given the best rank among a group of same score entities.
• Bottom : The correct entity is given the worst rank among a group of same score entities.
This policy is the most conservative.
• Average : The rank of the correct entity is the mean rank of all predictions that tie their score with the correct entity.
• Ordinal : The rank of the correct entity is given the rank in which it appears in the group of same score entities.
This policy usually corresponds to the random policy.
• Random :
The rank of the correct entity is given a random rank of the ranks within the group of same score entities.
This policy usually corresponds to the average policy.
When comparing results from diﬀerent sources it is important to use comparable policies for dealing with same score entities.
F.e. results evaluated with the top policy can be artiﬁcially boosted and thus they cannot be compared to results evaluated with the ordinal/random or
average policy.
Appendix B. Ruleset Statistics
[Table]
Appendix C. Runtime comparison AnyBURL - SAFRAN
Table 9 shows a comparison of the runtimes of AnyBURL and SAFRAN, when inferencing ﬁltered predictions of all relevant rules for each prediction task in the test set of a dataset.
The experiment was performed with 8 physical (16 logical) AMD Ryzen 7 PRO 4750U @ 1700 MHz cores and 16 GB of RAM.
SAFRAN could drastically improve the runtime of rule application
[Table]
Appendix D. Threshold Statistics
[Table]
[Table]
Appendix E. Clustering Statistics
Table 11 shows statistics of formed clusters for a selection of relations.
As can be seen, the average cluster size varies greatly between relations.
Extreme examples are the relations happenedIn where primarily small clusters were formed and isAﬃliatedTo where one big cluster and several smaller clusters were formed.
[Table]
happenedIn
[Table]
Appendix F. Experiments with AMIE+
Table 12 shows the results of Non-redundant Noisy-OR applied to rules learned with
AMIE+.
Rules were learned with default settings, in which AMIE+ only learns cyclic rules.
The results can be seen in Table 12.
As with rules learned by AnyBURL, Nonredundant Noisy-OR could achieve the biggest performance gain on the FB15k-237 dataset.
However, the gains on other datasets are signiﬁcantly small due to the restriction of the default setting to only cyclic rules.
As the rule set only contains only rules of one type (i.e. cyclic rules), only a grid search was performed with the same settings as in Table 7. [Table] FB15KMaximum . Noisy-OR .
[Table]
[Table]
WN18RR
Maximum .
Noisy-OR .
NRNO .
YAGO
[Table]
Maximum .
Noisy-OR .
NRNO .
Non-redundant Noisy-OR on rules learned by AMIE+.
Appendix G. Pseudocode of Non-redundant Noisy-OR Algorithm 1 Non-redundant Noisy-OR: Clustering
Input: Set of entities E , Set of relations R , Rule set RS , Thresholds TR of size | R | × 6, Test set T for relation rel in R do (cid:46) Calculation of clusters of redundant rules relevantRules ← rules in RS having relation rel in head; for r i in relevantRules do for r j in relevantRules do similarityMatrix [ r i ][ r j ] ← jaccard ( r i ,r j ) end for end for clusters [ rel ] ← Ø for r i in relevantRules do visited [ r i ] ←end for for r i in relevantRules do if visited [ r i ] = 0 then cluster ← Ø
Q ← { r i } while Q is not empty do r j ← Q.dequeue () cluster ← cluster ∪ { r j } visited [ r j ] ← 1 for r k in relevantRules do if similarityMatrix [ r j ][ r k ] > TR [ rel ][ type ( r j ,r k )] then if visited [ r k ] = 0 then
Q.enqueue ( r k ) end if end if end for end while clusters [ rel ] ← clusters [ rel ] ∪ { cluster } end if end for end for for prediction task t in T do (cid:46) Application of clusters of redundant rules for e in E do nrnoisy
[ e ] ← 0 end for for cluster in clusters [ relation ( t )] do for e in E do maximum
[ e ] ← 0 end for for rule in cluster do predictions ← prediction ( rule,t ) for e in E do if e ∈ predictions then maximum [ e ] ← max ( maximum [ e ] ,confidence ( rule )); end if end for end for for e in E do nrnoisy [ e ] ← noisy ( nrnoisy [ e ] ,maximum [ e ]) end for end for end for