Original: Why a naive way to Combine Symbolic and Latent Knowledge Base Completion works surprisingly well
Revised: Why a Naive Way to Combine Symbolic and Latent Knowledge Base Completion Works Surprisingly Well

Original: Anonymous authors
Revised: Anonymous authors

Original: Abstract
Revised: Abstract

Original: We compare a rule-based approach for knowledge graph completion against current state-of-the-art, which is based on embbedings.
Revised: We compare a rule-based approach for knowledge graph completion against current state-of-the-art, which is based on embeddings.

Original: Instead of focusing on aggregated metrics, we look at several examples that illustrate essential diﬀerences between symbolic and latent approaches.
Revised: Instead of focusing on aggregated metrics, we look at several examples that illustrate essential diﬀerences between symbolic and latent approaches.

Original: Based on our insights, we construct a simple method to combine the outcome of rule-based and latent approaches in a post-processing step.
Revised: Based on our insights, we construct a simple method to combine the outcome of rule-based and latent approaches in a post-processing step.

Original: Our method improves the results constantly for each model and dataset used in our experiments.
Revised: Our method improves the results constantly for each model and dataset used in our experiments

Original: 1. Introduction and Motivation
Revised: 1. Introduction and Motivation

Original: In this paper we compare knowledge graph completion (KGC) methods, that are based on non-symbolic representations in terms of embeddings, against a symbolic approach that is based on rules.
Revised: In this paper we compare knowledge graph completion (KGC) methods, that are based on non-symbolic representations in terms of embeddings, against a symbolic approach that is based on rules.

Original: Given a knowledge graph G , which is a set of triples, a KGC task is to predict the question mark in the incomplete triple r ( e, ?), where r is a relation (binary predicate) used in G and e is an entity described by (usually) several triples in G .
Revised: Given a knowledge graph G , which is a set of triples, a KGC task is to predict the question mark in the incomplete triple r ( e, ?) or r (? , e ), where r is a relation (binary predicate) used in G and e is an entity described by (usually) several triples in G .

Original: KGC methods solve this task by generating a ranking of possible candidates, which are evaluated by metrics that focus on the rank of the correct prediction within this ranking.
Revised: KGC methods solve this task by generating a ranking of possible candidates, which are evaluated by metrics that focus on the rank of the correct prediction within this ranking.

Original: A ﬁrst famous model for solving the KGC problem via a representation in an embeddings space is known as TransE [Bordes et al., 2013].
Revised: A ﬁrst famous model for solving the KGC problem via a representation in an embeddings space is known as TransE [Bordes et al., 2013].

Original: The basic idea of such a model, that we call in the following knowledge graph embedding (KGE) model, is to ﬁrst randomly map the entities and relations from G to a multidimensional space, e.g., to IR n .
Revised: The basic idea of such a model, termed knowledge graph embedding (KGE) model in the following, is to ﬁrst randomly map the entities and relations from G to a multidimensional space, e.g., to IR n .

Original: Then the triples from G are used to set up a large optimization problem that is solved via gradient descent or a similar algorithm.
Revised: Then the triples from G are used to set up a large optimization problem that is solved via gradient descent or a similar algorithm.

Original: How to transfer a triple into a part of the objective function is usually the deﬁning characteristic of the model.
Revised: How to transfer a triple into a part of the objective function is usually the deﬁning characteristic of the model.

Original: By doing this, the ﬁnal embedding becomes an alternative representation of the knowledge graph and it can be used to answer the completion task.
Revised: By doing this, the ﬁnal embedding becomes an alternative representation of the knowledge graph and it can be used to answer the completion task.

Original: Contrary to this, a rule-based approach searches for regularities in the graph and expresses them usually in the form of horn clauses.
Revised: Contrary to this, a rule-based approach searches for regularities in the graph and expresses them usually in the form of horn clauses.

Original: These rules are then used to propose candidates for solving the completion task.
Revised: These rules are then used to propose candidates for solving the completion task.

Original: Within this paper we try to understand the essential diﬀerences between both families of approaches.
Revised: Within this paper we try to understand the essential diﬀerences between both families of approaches.

Original: We ﬁrst tackle this problem by analysing the rankings for several concrete completion tasks.
Revised: We ﬁrst tackle this problem by analysing the rankings for several concrete completion tasks.

Original: This analysis results into several insights that motivates a speciﬁc method to combine both types of approaches.
Revised: This analysis results into several insights that motivates a speciﬁc method to combine both types of approaches.

Original: This aggregation method improves each of the incoming rankings, no matter whether rule or embeddings-based input has been better.
Revised: This aggregation method improves each of the incoming rankings, no matter whether rule or embeddings-based input has been better.

Revised: As an additional beneﬁt our method does not loose the explanatory power of a rule-based approach but integrates it into the combined approach.

Original: 2. Related Work
Revised: 2. Related Work

Original: In [Zhang et al., 2021] the authors are concerned with diﬀerent KGC methods with a focus on symbolic reasoning, neural reasoning techniques, and everything in between.
Revised: In [Zhang et al., 2021] the authors are concerned with diﬀerent KGC methods with a focus on symbolic reasoning, neural reasoning techniques, and everything in between.

Original: However, they discuss diﬀerences from a rather abstract level and try to give a comprehensive overview without any connection to their empirical analysis.
Revised: However, they discuss diﬀerences from a rather abstract level and try to give a comprehensive overview without any connection to their empirical analysis.

Original: Symbolic approaches and latent approaches that are based on a projection to an embedding space have already been compared in an experimental study in [Meilicke et al., 2018], where the authors identify types of completion tasks for which there are diﬀerences between these approaches.
Revised: Symbolic approaches and latent approaches that are based on a projection to an embedding space have already been compared in an experimental study in [Meilicke et al., 2018], where the authors identify types of completion tasks for which there are diﬀerences between these approaches.

Original: However, within the last three years signiﬁcant improvements have been made.
Revised: However, within the last three years signiﬁcant improvements have been made.

Original: For example, in the 2018 publication the authors reported about top hits@10 scores for the FB237 dataset in the range 0.4 to 0.43, while in [Broscheit et al., 2020] and [Rossi et al., 2021] scores between 0.5 and 0.55 are reported.
Revised: For example, in the 2018 publication the authors reported about top hits@10 scores for the FB237 dataset in the range 0.4 to 0.43, while in [Broscheit et al., 2020] and [Rossi et al., 2021] scores between 0.5 and 0.55 are reported.

Original: This makes insights that follow from these experiments to some degree unreliable, as it might be the case many ﬂaws are meanwhile ﬁxed.
Revised: This makes insights that follow from these experiments to some degree unreliable, as it might be the case many ﬂaws are meanwhile ﬁxed.

Original: Note also that our main interest is not to ﬁnd types of completion tasks where rules are superior or vice versa but to understand at least partially why this is the case.
Revised: Note also that our main interest is not to ﬁnd types of completion tasks where rules are superior or vice versa but to understand at least partially why this is the case.

Original: In [Rossi et al., 2021] the authors compare a large set of embeddings based methods and included the rule-based approach AnyBURL [Meilicke et al., 2019] as a baseline.
Revised: In [Rossi et al., 2021] the authors compare a large set of embeddings based methods and included the rule-based approach AnyBURL [Meilicke et al., 2019] as a baseline.

Original: This paper describes an extensive experimental study and comes to the conclusion that AnyBURL is an eﬃcient alternative to non-symbolic methods.
Revised: This paper describes an extensive experimental study and comes to the conclusion that AnyBURL is an eﬃcient alternative to non-symbolic methods.

Original: The paper does not explain the reasons for the partially unexpected result.

Original: Unfortunately, AnyBURL isto our knowledge the only symbolic approach that has proven to achieve results that are on the same level with the currentKGE state-of-the-art.
Revised: To our best knowledge, AnyBURL is the only symbolic approach that has proven to achieve performances on par with the current

Original: For that reason, we use AnyBURL as theonly representative of a rule-based approach in our experiments.
Revised: KGE state-of-the-art and therefore we choose AnyBURL as the representative for rule-based approaches within our experiments.

Original: In the second half of this paper we combine the results of rule and embedding based approaches.
Revised: In the second half of this paper we combine the results of rule and embedding based approaches.

Original: The core principle of our strategy, combining multiple models into an ensemble, has been studied in [Wang et al., 2018] and in [Meilicke et al., 2018].
Revised: The core principle of our strategy, combining multiple models into an ensemble, has been studied in [Wang et al., 2018] and in [Meilicke et al., 2018].

Original: Although the approach in this paper can be seen as an extension, it diﬀers fundamentally from a typical ensemble.
Revised: Although the approach in this paper can be seen as an extension, it diﬀers fundamentally from a typical ensemble.

Original: As we discuss in Section 4, we exclusively allow for candidates which are within the language bias of the rule based approach by only considering its top-k candidates.
Revised: As we discuss in Section 4, we exclusively allow for candidates which are within the language bias of the rule based approach by only considering its top-k candidates.

Original: Our combination strategy ensures that the rule based method and the KGE model operate independently and are able to focus on their strengths before the results are aggregated to achieve an overall improvement.
Revised: Our combination strategy ensures that the rule based method and the KGE model operate independently and are able to focus on their strengths before the results are aggregated to achieve an overall improvement.

Original: There are several papers that propose, on the contrary, to tightly integrate rule learning and embedding based approaches.
Revised: There are several papers that propose, on the contrary, to tightly integrate rule learning and embedding based approaches.

Original: Examples can be found in KALE [Guo et al., 2016] and RUGE [Guo et al., 2018].
Revised: Examples can be found in KALE [Guo et al., 2016] and RUGE [Guo et al., 2018].

Original: However, approaches that belong to this category are mostly restricted to a type of rule which does not allow for constants.
Revised: However, approaches that belong to this category are mostly restricted to a type of rule which does not allow for constants.

Original: More recently diﬀerentiable rule learning has been proposed [Rockt¨aschel and Riedel, 2017, Yang et al., 2017, Sadeghian et al., 2019, Minervini et al., 2020].
Revised: More recently diﬀerentiable rule learning has been proposed [Rockt¨aschel and Riedel, 2017, Yang et al., 2017, Sadeghian et al., 2019, Minervini et al., 2020].

Original: These approaches are not the focus of this work.
Revised: These approaches are not the focus of this work.

Original: Nevertheless, the aforementioned papers either have not been applied to the common KGC datasets or thethe results in this paper are better.
Revised: Nevertheless, the aforementioned papers either have not been applied to the common KGC datasets or the results in this paper are better.

Original: 3. Rule-Based Knowledge Base Completion with AnyBURL
Revised: 3. Rule-Based Knowledge Base Completion with AnyBURL

Original: We abstainhere from a detailed description of the AnyBURL algorithm and refer the reader to [Meilicke et al., 2019, 2020].
Revised: We abstain from a detailed description of the AnyBURL algorithm and refer the reader to [Meilicke et al., 2019, 2020].

Original: For our purpose  it isespecially important to know what kind of rules AnyBURL learns and how AnyBURL applies them to create  rankings.
Revised: Nevertheless, for the purpose of this work it is important to know what kind of rules AnyBURL learns and how they are applied to create the ﬁnal rankings.

Original: 3.1 Language Bias
Revised: 3.1 Language Bias

Original: We give concrete examples for the most important rule types within this section.
Revised: We give concrete examples for the most important rule types within this section.

Original: There are two additional rule types that are mainly used for ﬁlling up the rankings which are explained in the Appendix in Section C. Each of the supported rules is a horn rule and can be written as h ← b .
Revised: There are two additional rule types that are mainly used for ﬁlling up the rankings which are explained in the Appendix in Section C. Each of the supported rules is a horn rule and can be written as h ← b .

Original: We call h the head of the rule and b , which is a conjunction of atoms, the body of the rule.
Revised: We call h the head of the rule and b , which is a conjunction of atoms, the body of the rule.

Original: The ﬁrst and probably most prominent type of rules are called closed connected rules in [Gal´arraga et al., 2013] or cyclic rules according to [Meilicke et al., 2019].
Revised: The ﬁrst and probably most prominent type of rules are called closed connected rules in [Gal´arraga et al., 2013] or cyclic rules according to [Meilicke et al., 2019].

Original: The attribute cyclic refers to the head variables X and Y being directly connected in the head of the rule, while there is an alternative path expressed in the body of the rule.
Revised: The attribute cyclic refers to the head variables X and Y being directly connected in the head of the rule, while there is an alternative path expressed in the body of the rule.

Original: These rules do not contain constants but variables only.
Revised: These rules do not contain constants but variables only.

Original: Here are some examples.
Revised: Here are some examples.

Original: hypernym ( X, Y ) ← hyponym
Revised: hypernym [Equation]

Original: ( Y, X ) (1) contains ( X, Y ) ← contains ( X, A ) , adjoins ( A, Y ) (2) contains ( X, Y ) ← administrative parent ( A, X ) , adjoins ( A, B ) , capital ( B, Y ) (3)
Revised: , X ) (1) contains ( X, Y ) ← contains ( X, A ) , adjoins ( A, Y ) (2) contains ( X, Y ) ← administrative parent ( A, X ) , adjoins ( A, B ) , capital ( B, Y ) (3)

Original: The ﬁrst rule expresses that the hypernym relation is inverse to the hyponym relation.
Revised: The ﬁrst rule expresses that the hypernym relation is inverse to the hyponym relation.

Original: Rule 2 expresses the transitivity of the contains relation.
Revised: Rule 2 expresses the transitivity of the contains relation.

Original: Rule 3 is a rather complex rule that expresses roughly that X contains Y if X is an administrative parent location of A ,
Revised: Rule 3 is a rather complex rule that expresses roughly that X contains Y if X is an administrative parent location of A ,

Original: A shares a border with B and B has capital Y .
Revised: A shares a border with B and B has capital Y .

Original: Note that this rule is an example of a rule that creates highly ranked correct answers in our experiments.
Revised: Note that this rule is an example of a rule that creates highly ranked correct answers in our experiments.

Original: Another important type of rules are acyclic rules with only one variable, that have a constant in the head and a (in most cases diﬀerent) constant in the body.
Revised: Another important type of rules are acyclic rules with only one variable, that have a constant in the head and a (in most cases diﬀerent) constant in the body.

Original: AnyBURL is restricted in its default setting, which we used in our experiments, to learn rules of this type with only one body atom.
Revised: AnyBURL is restricted in its default setting, which we used in our experiments, to learn rules of this type with only one body atom.

Original: Here are two examples.
Revised: Here are two examples.

Original: [Equation]
Revised: [Equation]

Original: Rule 4 expresses that someone born in London is (probably) a citizen of the UK.
Revised: Rule 4 expresses that someone born in London is (probably) a citizen of the UK.

Original: Rulesays that locations contained in Victoria are also contained in Australia.
Revised: Rulesays that locations contained in Victoria are also contained in Australia.

Original: The vast majority of the mined rules are longer cyclic rules (as Rule 3) and acyclic rules similar to the ones that we just presented.
Revised: The vast majority of the mined rules are longer cyclic rules (as Rule 3) and acyclic rules similar to the ones that we just presented.

Original: A regularity that cannot be expressed in terms of these rule types is completely invisible to AnyBURL.
Revised: A regularity that cannot be expressed in terms of these rule types is completely invisible to AnyBURL.

Original: 3.2 Applying Rules
Revised: 3.2 Applying Rules

Original: To compute a prediction for a given completion task r ( a, ?)
Revised: To compute a prediction for a given completion task r ( a, ?)

Original: AnyBURL applies all rules that might create a triple r ( a, c ) where c is the predicted candidate.
Revised: AnyBURL applies all rules that might create a triple r ( a, c ) where c is the predicted candidate.

Original: This is done by grounding the relevant rules against the training set, i.e., by replacing the variables with constants (in other words: entities) such that the resulting body atoms are triples contained in the training set.
Revised: This is done by grounding the relevant rules against the training set, i.e., by replacing the variables with constants (in other words: entities) such that the resulting body atoms are triples contained in the training set.

Original: Following this procedure 2 it will happen quite often that a candidate is predicted by several rules.
Revised: Following this procedure 2 it will happen quite often that a candidate is predicted by several rules.

Original: In this case the maximum of the conﬁdences of these rules is associated as conﬁdence of predicting c .
Revised: In this case the maximum of the conﬁdences of these rules is associated as conﬁdence of predicting c .

Original: If there are several candidates c and c (cid:48) that are predicted with the same conﬁdence, they are ordered in the ranking according to the conﬁdence of the second best rule, (if this conﬁdence is also the same, the third-best counts, and so on).
Revised: If there are several candidates c and c (cid:48) that are predicted with the same conﬁdence, they are ordered in the ranking according to the conﬁdence of the second best rule, (if this conﬁdence is also the same, the third-best counts, and so on).

Original: Two candidates that cannot be distinguished due to the fact that they are predicted by a set of rules that have exactly the same conﬁdences, are ranked randomly.
Revised: Two candidates that cannot be distinguished due to the fact that they are predicted by a set of rules that have exactly the same conﬁdences, are ranked randomly.

Original: We call this aggregation method in the following maximum-aggregation.
Revised: We call this aggregation method in the following maximum-aggregation.

Original: In [Meilicke et al., 2020] the authors also reported about a noisy-or-aggregation where a candidate predicted by a set of rules will have a higher conﬁdence than a candidate predicted by a subset.
Revised: In [Meilicke et al., 2020] the authors also reported about a noisy-or-aggregation where a candidate predicted by a set of rules will have a higher conﬁdence than a candidate predicted by a subset.

Original: It turned out that this method performed worse on all datasets compared to the simpler maximum-aggregation.
Revised: It turned out that this method performed worse on all datasets compared to the simpler maximum-aggregation.

Original: This was especially caused by the problem that the method cannot discriminate between redundant rules that ﬁre for the same reason and rules that capture diﬀerent aspects.
Revised: This was especially caused by the problem that the method cannot discriminate between redundant rules that ﬁre for the same reason and rules that capture diﬀerent aspects.

Original: Opposed to a symbolic approach, that requires to explicitly deﬁne an aggregation type, KGE approaches have an implicit aggregation technique that is based on the fact that each triple is part of a comprehensive objective function.
Revised: Opposed to a symbolic approach, that requires to explicitly deﬁne an aggregation type, KGE approaches have an implicit aggregation technique that is based on the fact that each triple is part of a comprehensive objective function.

Original: This is another advantage that KGE models might have compared to rule-based approaches as long as they are based on a rather simple aggregation method.
Revised: This is another advantage that KGE models might have compared to rule-based approaches as long as they are based on a rather simple aggregation method.

Original: 4. Case By Case Analysis
Revised: 4. Case By Case Analysis

Original: It is not easy to distinguish accidental diﬀerences, that might be caused by the stochastic nature of the approaches, from systematic diﬀerences.
Revised: It is not easy to distinguish accidental diﬀerences, that might be caused by the stochastic nature of the approaches, from systematic diﬀerences.

Original: We ﬁrst propose a method to spot essential diﬀerences before we discuss what we found with this method.
Revised: We ﬁrst propose a method to spot essential diﬀerences before we discuss what we found with this method.

Original: We report occasionally about datasets and KGE models that are ﬁrst introduced in Section 6.1.
Revised: We report occasionally about datasets and KGE models that are ﬁrst introduced in Section 6.1.

Original: 4.1 Spotting Essential Diﬀerences
Revised: 4.1 Spotting Essential Diﬀerences

Original: Let R be a candidate ranking for a completion task r ( e, ?).
Revised: Let R be a candidate ranking for a completion task r ( e, ?).

Original: A candidate ranking is a total order over the entities in the given knowledge graph.
Revised: A candidate ranking is a total order over the entities in the given knowledge graph.

Original: We use R [ n ] to denote the entity ranked at position n in R and R [ c ] # to denote the ranking position of a candidate c .
Revised: We use R [ n ] to denote the entity ranked at position n in R and R [ c ] # to denote the ranking position of a candidate c .

Original: Given the completion task r
Revised: Given the completion task r

Original: ( e, ?), let A denote the AnyBURL ranking for r ( e, ?) and let E be a ranking for r ( e, ?) generated by a KGE model.
Revised: ( e, ?), let A denote the AnyBURL ranking for r ( e, ?) and let E be a ranking for r ( e, ?) generated by a KGE model.

Original: We use conf ( c, r ( e, ?))
Revised: We use conf ( c, r ( e, ?))

Original: to denote the conﬁdence that AnyBURL assigns to c with respect to being the answer to r ( e, ?).
Revised: to denote the conﬁdence that AnyBURL assigns to c with respect to being the answer to r ( e, ?).

Original: Then
Revised: Then

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: denotes the anomaly degree of ranking c in E for r ( e, ?)
Revised: denotes the anomaly degree of ranking c in E for r ( e, ?)

Original: from the perspective of AnyBURL.
Revised: from the perspective of AnyBURL.

Original: The deﬁnition is based on the idea to compare the conﬁdence that AnyBURL assigns to c to the conﬁdence that AnyBURL assigns within its own ranking to the entity ranked at the position where c is ranked in the KGE ranking.
Revised: The deﬁnition is based on the idea to compare the conﬁdence that AnyBURL assigns to c to the conﬁdence that AnyBURL assigns within its own ranking to the entity ranked at the position where c is ranked in the KGE ranking.

Original: A score of around 1 means that there is noanomaly, high scores mean that AnyBURL is highly conﬁdent that c is ranked to low, and low scores mean that c ranked to high.
Revised: A score of around 1 means that there is no

Original: [Table]
Revised: [Table]

Revised: anomaly, high scores mean that AnyBURL is highly conﬁdent that c is ranked to low, and low scores mean that c ranked to high.

Original: One might argue that it would have been better to base the deﬁnition of Ψ on a direct comparison of the ranking positions.
Revised: One might argue that it would have been better to base the deﬁnition of Ψ on a direct comparison of the ranking positions.

Original: If we have a completion task as locatedIn(?, UK) a city as Bristol might be ranked by AnyBURL on position #12 while it might be ranked on #77 by a KGE model.
Revised: If we have a completion task as locatedIn(?, UK) a city as Bristol might be ranked by AnyBURL on position #12 while it might be ranked on #77 by a KGE model.

Original: However, the AnyBURL conﬁdences between rank #10 and # might be very close.
Revised: However, the AnyBURL conﬁdences between rank #10 and # might be very close.

Original: Our deﬁnition of Ψ takes this into account and would yield a score around 1, while we would get a high score if we would base the score on ranking positions.
Revised: Our deﬁnition of Ψ takes this into account and would yield a score around 1, while we would get a high score if we would base the score on ranking positions.

Original: 4.2 Selected Examples
Revised: 4.2 Selected Examples

Original: We ﬁrst look at two predictions c with Ψ( c, E , r ( e, ?))
Revised: We ﬁrst look at three predictions c with Ψ( c, E , r ( e, ?))

Original: < 0 .
Revised: < 0 .

Original: 2 that are ranked above the correct hit.
Revised: 2 that are ranked above the correct hit.

Original: In the last paragraph we talk about the opposite, in particular we report about a relation where we spotted high Ψ scores in general.
Revised: In the last paragraph we talk about the opposite, in particular we report about a relation where we spotted high Ψ scores in general.

Original: We sometimes mark assertions with ‘(in test/training)’.
Revised: We sometimes mark assertions with ‘(in test/training)’.

Original: This means that the corresponding triple can be found in the test set (or in the training set).
Revised: This means that the corresponding triple can be found in the test set (or in the training set).

Original: Additional examples are discussed in the appendix in Section B.
Revised: Additional examples are discussed in the appendix in Section B.

Original: Where is the city Darwin
Revised: Where is the city Darwin

Original: The city Darwin is a city in Australia (in test).
Revised: The city Darwin is a city in Australia (in test).

Original: Darwin is located in the Northern Territory (in training).
Revised: Darwin is located in the Northern Territory (in training).

Original: The training set contains another triple that states that Darwin is also the capital of the Northern Territory.
Revised: The training set contains another triple that states that Darwin is also the capital of the Northern Territory.

Original: The training set states also that Australia contains the Northern Territory.
Revised: The training set states also that Australia contains the Northern Territory.

Original: We are now concerned with the completion task contains(?, Darwin) .
Revised: We are now concerned with the completion task contains(?, Darwin) .

Original: As shown in Table 1 AnyBURL puts Australia ﬁrst with all other alternatives having a very low conﬁdence, while ComplEx ranks each Australian territory ﬁrst before Australia appears at #7.
Revised: As shown in Table 1 AnyBURL puts Australia ﬁrst with all other alternatives having a very low conﬁdence, while ComplEx ranks each Australian territory ﬁrst before Australia appears at #7.

Original: RESCAL puts it on #2, however, its scores are similar to the scores of other alternatives that are clearly wrong.
Revised: RESCAL puts it on #2, however, its scores are similar to the scores of other alternatives that are clearly wrong.

Original: AnyBURL ranks Australia ﬁrst due to Rule (3), a complex rule with three body atoms, presented in Section 3.1.
Revised: AnyBURL ranks Australia ﬁrst due to Rule (3), a complex rule with three body atoms, presented in Section 3.1.

Original: The rule that expresses the transitivity of the contains relation (a rule with two atoms in the body) would also be suﬃcient to put Australia on top with a conﬁdence of 0.273.
Revised: The rule that expresses the transitivity of the contains relation (a rule with two atoms in the body) would also be suﬃcient to put Australia on top with a conﬁdence of 0.273.

Original: It seems that both regularities have a rather limited impact on the
Revised: It seems that both regularities have a rather limited impact on the

Original: KGE rankings.
Revised: KGE rankings.

Original: The KGE ranking might be aﬀected by the fact that the other Australian territories are very similar to the Northern Territory (all are contained in Australia, some share borders with each other, ect.).
Revised: The KGE ranking might be aﬀected by the fact that the other Australian territories are very similar to the Northern Territory (all are contained in Australia, some share borders with each other, ect.).

Original: The KGE ranking can be explained by a substitution of Northern Territory in contains(Northern Territory, Darwin) by very similar entities.
Revised: The KGE ranking can be explained by a substitution of Northern Territory in contains(Northern Territory, Darwin) by very similar entities.

Original: Metropolis
Revised: Metropolis

Original: The completion task festivals(Metropolis,?)
Revised: The completion task festivals(Metropolis,?)

Original: asks at which festival the movie
Revised: asks at which festival the movie

Original: Metropolis has been shown.
Revised: Metropolis has been shown.

Original: The correct answer is the 39th Berlin International Film
Revised: The correct answer is the 39th Berlin International Film

Original: Festival.
Revised: Festival.

Original: AnyBURL places the correct answer at #18 with a low conﬁdence of 0.016,
Revised: AnyBURL places the correct answer at #18 with a low conﬁdence of 0.016,

Original: ComplEx ranks it at #25.
Revised: ComplEx ranks it at #25.

Original: This diﬀerence is not signiﬁcant.
Revised: This diﬀerence is not signiﬁcant.

Original: More interesting are the candidates that can be found in the complete ranking.
Revised: More interesting are the candidates that can be found in the complete ranking.

Original: The ﬁrst 23 AnyBURL ranks are ﬁlled with ﬁlm festivals, followed by a mixed list that contains both festivals and cities (the relation festivals has sometimes been used to express that a movie has been shown in a certain city).
Revised: The ﬁrst 23 AnyBURL ranks are ﬁlled with ﬁlm festivals, followed by a mixed list that contains both festivals and cities (the relation festivals has sometimes been used to express that a movie has been shown in a certain city).

Original: All candidates have a low conﬁdence.
Revised: All candidates have a low conﬁdence.

Original: Still, they are to a certain degree meaningful up to (at least) position #36.
Revised: Still, they are to a certain degree meaningful up to (at least) position #36.

Original: In the ComplEx ranking starting from position #23 weird candidates show up: USA (#23), natural death cause (#24), Chris Parnell (#31), electric guitar (#33).
Revised: In the ComplEx ranking starting from position #23 weird candidates show up: USA (#23), natural death cause (#24), Chris Parnell (#31), electric guitar (#33).

Original: What is interesting with respect to these candidates is not that they exists somewhere in the ranking but that they are ranked above some meaningful candidates without obvious reasons.
Revised: What is interesting with respect to these candidates is not that they exists somewhere in the ranking but that they are ranked above some meaningful candidates without obvious reasons.

Original: It seems that the signals for the other meaningful candidates are too weak to enforce a meaningful order.
Revised: It seems that the signals for the other meaningful candidates are too weak to enforce a meaningful order.

Original: Michael Fisher works for King’s College
Revised: Michael Fisher works for King’s College

Original: This example is concerned with the completion task employer(ﬁsher,?) .
Revised: This example is concerned with the completion task employer(ﬁsher,?) .

Original: Michael Fisher has been a mathematician and physicist, who studied at the Kings’s College (training set) and worked as an employee at the Kings’s
Revised: Michael Fisher has been a mathematician and physicist, who studied at the Kings’s College (training set) and worked as an employee at the Kings’s

Original: College (test set).
Revised: College (test set).

Original: Another relatively important triple given in the training set states that
Revised: Another relatively important triple given in the training set states that

Original: Michael Fisher worked also for the Leiden University.
Revised: Michael Fisher worked also for the Leiden University.

Original: AnyBURL ranks the correct answer,
Revised: AnyBURL ranks the correct answer,

Original: King’s College, at position #10 only, while ComplEx and most of the other KGE models rank King’s College at #1 or at least among the top-5.
Revised: King’s College, at position #10 only, while ComplEx and most of the other KGE models rank King’s College at #1 or at least among the top-5.

Original: When analysing the task from the perspective of AnyBURL we detected the following two rules, that result into the correct prediction.
Revised: When analysing the task from the perspective of AnyBURL we detected the following two rules, that result into the correct prediction.

Original: The rules are shown on the left, the triples that makes the rule ﬁre are shown on the right.
Revised: The rules are shown on the left, the triples that makes the rule ﬁre are shown on the right.

Original: [Equation]
Revised: [Equation]

Original: studiedAt

Original: [Equation]

Original: Y ) studiedAt ( fisher

Original: [Equation] ) (6) employer ( X, kingscoll ) ← employer ( X, leiden ) employer ( fisher [Equation] leiden ) (7)
Revised: employer ( fisher, leiden ) (7)

Original: Both rules have a conﬁdence of ≈ 8%.
Revised: Both rules have a conﬁdence of ≈ 8%.

Original: There are several rules with higher conﬁdences (up to 15%) that generate the candidates that are ranked above King’s College.
Revised: There are several rules with higher conﬁdences (up to 15%) that generate the candidates that are ranked above King’s College.

Original: All other rules, which allow to predict King’s College, have a conﬁdence lower than 2%.
Revised: All other rules, which allow to predict King’s College, have a conﬁdence lower than 2%.

Original: As both rules have nearly the same conﬁdence, the maximum aggregation in AnyBURL will score
Revised: As both rules have nearly the same conﬁdence, the maximum aggregation in AnyBURL will score

Original: King’s College with (nearly) the same score no matter if we have both or only one of the two triples.
Revised: King’s College with (nearly) the same score no matter if we have both or only one of the two triples.

Original: If we remove both, King’s college falls out of the top-50 ranking.
Revised: If we remove both, King’s college falls out of the top-50 ranking.

Original: This example allows us to shed light on two question: (1) Are the triples that determine the AnyBURL ranking the same triples that determines the KGE behaviour?
Revised: This example allows us to shed light on two question: (1) Are the triples that determine the AnyBURL ranking the same triples that determines the KGE behaviour?

Original: (2) To what extent can the KGE results be explained as a cumulative aggregation of these triples (and the rules that they ﬁre)?
Revised: (2) To what extent can the KGE results be explained as a cumulative aggregation of these triples (and the rules that they ﬁre)?

Original: For this purpose we have executed ComplEx and HittER* on the original dataset, on the variant where we removed ( t 1 )
Revised: For this purpose we have executed ComplEx and HittER* on the original dataset, on the variant where we removed ( t 1 )

Original: studiedAt(ﬁsher,kingscoll) , on the variant where we removed ( t 2 ) employer(ﬁsher,leiden) , and on the variant were we removed both.
Revised: studiedAt(ﬁsher,kingscoll) , on the variant where we removed ( t 2 ) employer(ﬁsher,leiden) , and on the variant were we removed both.

Original: As KGE models can vary a lot between diﬀerent runs 3 , we conducted six runs, based on a diﬀerent random initialisation, for each dataset variant.
Revised: As KGE models can vary a lot between diﬀerent runs 3 , we conducted six runs, based on a diﬀerent random initialisation, for each dataset variant.

Original: Results are reported in
Revised: Results are reported in

Original: Table 2.
Revised: Table 2.

Original: Contrary to the maximum aggregation of AnyBURL, both KGE models aggregate the evidence that lies within these triples in a beneﬁcial way.

Original: When suppressing t 1 the average

Original: [Table]
Revised: [Table]

Original: All triples
Revised: All triples

Original: [Table]
Revised: [Table]

Original: Without t
Revised: Without t

Original: [Table]
Revised: [Table]

Original: Without t
Revised: Without t

Original: [Table]
Revised: [Table]

Original: Without t 1 and t
Revised: Without t 1 and t

Original: [Table]
Revised: [Table]

Revised: Contrary to the maximum aggregation of AnyBURL, both KGE models aggregate the evidence that lies within these triples in a beneﬁcial way.

Original:  rank of the correct hit falls slightly for ComplEx and signiﬁcantly for HittER*.
Revised: When suppressing t 1 the average rank of the correct hit falls slightly for ComplEx and signiﬁcantly for HittER*.

Original: The impact of removing t 2 is similar, yet stronger.
Revised: The impact of removing t 2 is similar, yet stronger.

Original: If both triples are removed, King’s College drops to a rank below #50.
Revised: If both triples are removed, King’s College drops to a rank below #50.

Original: This shows that the triples that are relevant for the AnyBURL rankings have also a signiﬁcant impact on the rankings of ComplEx and HittER*.
Revised: This shows that the triples that are relevant for the AnyBURL rankings have also a signiﬁcant impact on the rankings of ComplEx and HittER*.

Original: This kind of aggregation seems to work in general better for the tail predictions related to the employer -relation.
Revised: This kind of aggregation seems to work in general better for the tail predictions related to the employer -relation.

Original: We computed the MRR for these completion tasks only.
Revised: We computed the MRR for these completion tasks only.

Original: The
Revised: The

Original: KGE models achieved a score between 0.38 and 0.43, while the MRR of AnyBURL is 0. only.
Revised: KGE models achieved a score between 0.38 and 0.43, while the MRR of AnyBURL is 0. only.

Original: With only few exception all rules learned for this relation have a conﬁdence less then 0.4.
Revised: With only few exception all rules learned for this relation have a conﬁdence less then 0.4.

Original: We further looked at several randomly selected examples and for most of them several low-conﬁdent rules ﬁred similar to the example we presented above.
Revised: We further looked at several randomly selected examples and for most of them several low-conﬁdent rules ﬁred similar to the example we presented above.

Original: 5. Aggregating Rankings
Revised: 5. Aggregating Rankings

Original: In the following we propose an approach to combine the rankings generated by a rule-based approach and the rankings generated by a KGE model.
Revised: In the following we propose an approach to combine the rankings generated by a rule-based approach and the rankings generated by a KGE model.

Original: Motivated by the analysis in the previous section, we try to achieve the following goals:
Revised: Motivated by the analysis in the previous section, we try to achieve the following goals:

Original: • If something appears in a KGE ranking (1) which is an artefact of an random initialisation (e.g., Metropolis ) or (2) which appears there due to a similarity consideration not backed by a regularity (e.g., Darwin ), it should not appear in the ﬁnal ranking.
Revised: • If something appears in a KGE ranking (1) which is an artefact of an random initialisation (e g., Metropolis ) or (2) which appears there due to a similarity consideration not backed by a regularity (e.g., Darwin ), it should not appear in the ﬁnal ranking.

Original: • Relations for which KGE methods work better (e.g., employer ) and relations for which rules work better should be treated diﬀerently.
Revised: • Relations for which KGE methods work better (e g., employer ) and relations for which rules work better should be treated diﬀerently.

Original: The method should choose a weighting between KGE and rules that ﬁts best to the relation and direction (head vs. tail).
Revised: The method should choose a weighting between KGE and rules that ﬁts best to the relation and direction (head vs. tail).

Original: The ﬁrst requirement can be fulﬁlled by suppressing any (top-ranked) KGE prediction that is not at all predicted (or predicted with a conﬁdence close to 0) by a rule based approach.
Revised: The ﬁrst requirement can be fulﬁlled by suppressing any (top-ranked) KGE prediction that is not at all predicted (or predicted with a conﬁdence close to 0) by a rule based approach.

Original: Thus, the approach that we describe in the next section is restricted to the top-k ranking of AnyBURL and uses the KGE scores only as an additional information to change the position in the ranking of AnyBURL.
Revised: Thus, the approach that we describe in the next section is restricted to the top-k ranking of AnyBURL and uses the KGE scores only as an additional information to change the position in the ranking of AnyBURL.

Original: This will directly ﬁlter out any predictions for which a rule-based approach does not see any evidence.
Revised: This will directly ﬁlter out any predictions for which a rule-based approach does not see any evidence.

Original: It is a rather risky approach, as it is build on the assumption that everything visible to KGE lies within the language bias of AnyBURL.
Revised: It is a rather risky approach, as it is build on the assumption that everything visible to KGE lies within the language bias of AnyBURL.

Original: In case our approach works well, this shows that the vast majority of KGE predictions can be backed up by a symbolic explanation.
Revised: In case our approach works well, this shows that the vast majority of KGE predictions can be backed up by a symbolic explanation.

Original: The second requirement can easily be implemented by using the validation set to determine for which relations KGE models support better results and for which AnyBURL creates better rankings.
Revised: The second requirement can easily be implemented by using the validation set to determine for which relations KGE models support better results and for which AnyBURL creates better rankings.

Original: As it might be the case that there is a diﬀerence between head and tail-predictions within the relation, we have to distinguish not only between diﬀerent relations but have to take the direction of the prediction additionally into account.
Revised: As it might be the case that there is a diﬀerence between head and tail-predictions within the relation, we have to distinguish not only between diﬀerent relations but have to take the direction of the prediction additionally into account.

Original: For each relation r we ﬁrst collect all triples from the validation set that use r .
Revised: For each relation r we ﬁrst collect all triples from the validation set that use r .

Original: We refer to this subset as V ( r ).
Revised: We refer to this subset as V ( r ).

Original: Now we search over possible values for an aggregation parameter β r,ht where r denotes the relation and ht determines whether we deal with head or tail predictions.
Revised: Now we search over possible values for an aggregation parameter β r,ht where r denotes the relation and ht determines whether we deal with head or tail predictions.

Original: We iterate over all head (or tail) completion tasks r
Revised: We iterate over all head (or tail) completion tasks r

Original: (? , e ) (or r ( e, ?))
Revised: (? , e ) (or r ( e, ?))

Original: resulting from the triples in V ( r ).
Revised: resulting from the triples in V ( r ).

Original: For each prediction task r ( e, ?)
Revised: For each prediction task r ( e, ?)

Original: we compute an aggregated score score agg for each candidate in A k , where A k denotes the top-k candidate ranking created by
Revised: we compute an aggregated score score agg for each candidate in A k , where A k denotes the top-k candidate ranking created by

Original: AnyBURL.
Revised: AnyBURL.

Original: We use the following formula where score norm ( c, r (? , e )) is the normalized score that the KGE model assigned to c in the context of r (? , e ).
Revised: We use the following formula where score norm ( c, r (? , e )) is the normalized score that the KGE model assigned to c in the context of r (? , e ).

Original: We explain in the appendix in
Revised: We explain in the appendix in

Original: Section E how we normalize the KGE score.
Revised: Section E how we normalize the KGE score.

Original: [Equation]
Revised: [Equation]

Original: The aggregated score is a linear combination of AnyBURL and normalized KGE score, where β r,ht determines the weighting.
Revised: The aggregated score is a linear combination of AnyBURL and normalized KGE score, where β r,ht determines the weighting.

Original: Based on the aggregated scores, we create a reordered ranking.
Revised: Based on the aggregated scores, we create a reordered ranking.

Original: Once we computed all aggregated rankings for the tail (or head) predictions in
Revised: Once we computed all aggregated rankings for the tail (or head) predictions in

Original: V ( r ) for a speciﬁc β r,ht , we compute the MRR for these rankings.
Revised: V ( r ) for a speciﬁc β r,ht , we compute the MRR for these rankings.

Original: We search for the best parameter β r,ht for each relation and direction (head vs. tail prediction) via a grid search.
Revised: We search for the best parameter β r,ht for each relation and direction (head vs. tail prediction) via a grid search.

Original: 6. Experiments
Revised: 6. Experiments

Original: We ﬁrst explain the settings and datasets that we used in our experiments in Section 6.1, followed by a presentation of the most important results in Section 6.2.
Revised: We ﬁrst explain the settings and datasets that we used in our experiments in Section 6.1, followed by a presentation of the most important results in Section 6.2 and further ablation experiments in Section 6.3.

Original: 6.1 Settings
Revised: 6.1 Settings

Original: We evaluate our approach on FB237 [Toutanova and Chen, 2015] (also called FB15k-237 or FB15KSelected) and WNRR [Dettmers et al., 2018], which are frequently used in the literature and have been created to overcome leakage and redundancy problems of FB15K and WN18 [Bordes et al., 2013], respectively.
Revised: We evaluate our approach on FB237 [Toutanova and Chen, 2015] (also called FB15kor FB15KSelected) and WNRR [Dettmers et al., 2018], which are frequently used in the literature and have been created to overcome leakage and redundancy problems of FB15K and WN18 [Bordes et al., 2013], respectively.

Original: Further, we use the CoDEx benchmark [Safavi and Koutra, 2020] which includes three knowledge graphs in varying sizes designed with the goal to be more diﬃcult than previously published datasets [Safavi and Koutra, 2020].
Revised: Furthermore, we use the CoDEx benchmark [Safavi and Koutra, 2020] which includes three knowledge graphs in varying sizes designed with the goal to be more diﬃcult than previously published datasets [Safavi and Koutra, 2020].

Original: Summary statistics for the datasets can be found in Table 7 in the appendix.
Revised: Summary statistics for the datasets can be found in Table 8 in the appendix.

Original: In regard to the KGE models, we use the libKGE library [Broscheit et al., 2020] which focuses on reproducibility and has shown to produce state-of-the-art results.
Revised: In regard to the KGE models, we use the libKGE library [Broscheit et al., 2020] which focuses on reproducibility and has shown to produce state-of-the-art results.

Original: We include
Revised: We include

Original: TransE [Bordes et al., 2013], RESCAL [Nickel et al., 2011], DistMult [Yang et al., 2015],
Revised: TransE [Bordes et al., 2013], RESCAL [Nickel et al., 2011], DistMult [Yang et al., 2015],

Original: ComplEx [Trouillon et al., 2016], ConvE [Dettmers et al., 2018], and TuckER [Balaˇzevi´c et al., 2019].
Revised: ComplEx [Trouillon et al., 2016], ConvE [Dettmers et al., 2018], and TuckER [Balaˇzevi´c et al., 2019].

Original: For these models, we use the pretrained embeddings from libKGE when available for the respective datasets.
Revised: For these models, we use the pretrained embeddings from libKGE when available for the respective datasets.

Original: Due to its very good results, we use additionally the transformer implementation of libKGE which is based on the HittER no-context model [Chenet al., 2020]. We refer to this implementation, which is mentioned as transformer in libKGE, as HittER* in the following. 4 More details can be found in the supplementary material.
Revised: Due to its very good results, we use additionally the transformer implementation of libKGE which is based on the HittER no-context model [Chen

Original: FB
Revised: FB

Original: MRR .
Revised: MRR .

Original: AnyBURL .
Revised: AnyBURL .

Original: WNRR
Revised: WNRR

Original: MRR .
Revised: MRR .

Original: AnyBURL .
Revised: AnyBURL .

Original: CoDEx-L
Revised: CoDEx-L

Original: MRR .
Revised: MRR .

Original: AnyBURL
Revised: AnyBURL

Original: TuckER
Revised: TuckER

Original: TransE
Revised: TransE

Original: RESCAL
Revised: RESCAL

Original: HittER*
Revised: HittER*

Original: DistMult
Revised: DistMult

Original: ConvE
Revised: ConvE

Original: ComplEx 
Revised: ComplEx et al., 2020]. We refer to this implementation, which is mentioned as transformer in libKGE, as HittER* in the following. 4 More details can be found in the supplementary material.

Original: With respect to the aggregation method described in Section 5 , we set k =100.
Revised: With respect to the aggregation method described in Section 5 , we set k =100.

Original: We search the best β r,ht value over each multiple of 0.05 in the range [0 , 1].
Revised: We search the best β r,ht value over each multiple of 0.05 in the range [0 , 1].

Original: We use the evaluation protocol that has ﬁrst been proposed in [Bordes et al., 2013].
Revised: We use the evaluation protocol that has ﬁrst been proposed in [Bordes et al., 2013].

Original: In particular, we report hits@1, hits@10 and MRR (mean reciprocal rank).
Revised: In particular, we report hits@1, hits@10 and MRR (mean reciprocal rank).

Original: In the following we always present the ﬁltered variants of these measures without adding the adjective ‘ﬁltered’.
Revised: In the following we always present the ﬁltered variants of these measures without adding the adjective ‘ﬁltered’.

Original: 6.2  Results
Revised: 6.2 Main Results

Original: In particular, we compare the MRR scores for KGE, AnyBURL and the aggregated results in Figure 1 for WNRR, FB237, and CoDEx-L (all CoDEx variants can be found in the appendix in Figure 2).
Revised: In particular, we compare the MRR scores for KGE, AnyBURL and the aggregated results in Figure 1 for WNRR, FB237, and CoDEx-L (all CoDEx variants can be found in the appendix in Figure 2).

Original: Detailed numbers can be found in the appendix in Table 3.
Revised: Detailed numbers can be found in the appendix in Table 4.

Original: We have depicted a vertical line for each KGE model.
Revised: We have depicted a vertical line for each KGE model.

Original: The lower endpoint refers to the MRR of the model itself, and the upper endpoint refers to the MRR that we measured after applying our aggregation method.
Revised: The lower endpoint refers to the MRR of the model itself, and the upper endpoint refers to the MRR that we measured after applying our aggregation method.

Original: The longer this vertical line, the stronger is the positive impact of our approach.
Revised: The longer this vertical line, the stronger is the positive impact of our approach.

Original: We explain the marker in the middle of each vertical line later.
Revised: We explain the marker in the middle of each vertical line in Section 6.3.

Original: We have depicted the AnyBURL results as a horizontal dashed line.
Revised: We have depicted the AnyBURL results as a horizontal dashed line.

Original: We need to point to some general diﬀerences ﬁrst of all.
Revised: We need to point to some general diﬀerences ﬁrst of all.

Original: The WNRR datasets is a dataset where AnyBURL clearly outperforms each of the KGE models in terms of MRR.
Revised: The WNRR datasets is a dataset where AnyBURL clearly outperforms each of the KGE models in terms of MRR.

Original: For FB237 the opposite is the case with the exception of TransE. The largest version of the
Revised: For FB237 the opposite is the case with the exception of TransE. The largest version of the

Original: CoDEx datasets is somewhere in between as there are some KGE models that are better and some that are worse than AnyBURL.
Revised: CoDEx datasets is somewhere in between as there are some KGE models that are better and some that are worse than AnyBURL.

Original: We observe a similar trend across all datasets and KGE models.
Revised: We observe a similar trend across all datasets and KGE models.

Original: Each KGE model is improved by at least 1% on each dataset (in average it is much more).
Revised: Each KGE model is improved by at least one percentage point on each dataset. On average the improvement is 2.6 percentage points excluding TransE and 4. including TransE.

Original: The same holds from the perspective of AnyBURL.
Revised: The same holds from the perspective of AnyBURL.

Original: Even in the case of
Revised: Even in the case of

Original: WNRR, each of the models, inlcuding those that do not perform well, help to improve the results  by at least 0 .
Revised: WNRR, each of the models, inlcuding those that do not perform well, help to improve the results of AnyBURL by at least 0.5 percentage points.

Original: The models that do not perform well (e.g., TransE on FB237, RESCAL on WNRR,
Revised: The models that do not perform well (e.g., TransE on FB237, RESCAL on WNRR,

Original: TransE on CoDEx-M) are improved signiﬁcantly.
Revised: TransE on CoDEx-M) are improved signiﬁcantly.

Original: As a result the gap between the results that we have after the aggregation do not diﬀer much between diﬀerent models anymore.
Revised: As a result the gap between the results that we have after the aggregation do not diﬀer much between diﬀerent models anymore.

Original: This in particular important, as we are talking about models that diﬀer a lot in terms of computational eﬀort required to ﬁnd a good hyperparameter setting.
Revised: This in particular important, as we are talking about models that diﬀer a lot in terms of computational eﬀort required to ﬁnd a good hyperparameter setting.

Original: One of the best models in our evaluation is HittER*.
Revised: One of the best models in our evaluation is HittER*.

Original: Here we observe an improvement of ≈ 2% for CoDEx-L, an improvement of more than 3% on WNRR and and improvement of  1 . 5% on FB237.
Revised: Here we observe an improvement of ≈ 2 percentage points for CoDEx-L, an improvement of more than 3 percentage points on WNRR and and improvement of roughly 1 . 5 percentage points on FB237.

Original: Especially the FB237 result is interesting, as it shows that the aggregation with AnyBURL can improve a top-score even though the AnyBURL result itself is signiﬁcantly worse.
Revised: Especially the FB237 result is interesting, as it shows that the aggregation with AnyBURL can improve a top-score even though the AnyBURL result itself is signiﬁcantly worse.

Original: Note also, that we were able to improve the HittER* results on FB237 and WNRR up to the performance of the full-ﬂedged HittER variant, that has been claimed to be state-of-the-art in [Chen et al., 2020] ).
Revised: Please also note that we were able to improve the HittER* results on FB237 and WNRR up to the performance of the full-ﬂedged HittER variant, that has been claimed to be state-of-the-art in [Chen et al., 2020].

Original: This is in particular interesting as our approach requires signiﬁcantly less computational resources to achieve the same results.
Revised: This is in particular interesting as our approach requires signiﬁcantly less computational resources to achieve the same results.

Original: Results for the full-ﬂedged HittER variant for CoDEx-L are not available, and it would be extremly costly to generate these results.
Revised: Results for the full-ﬂedged HittER variant for CoDEx-L are not available, and it would be extremly costly to generate these results.

Revised: 6.3 Ablation Study

Original: To better understand what causes these results, we have computed scores where we  ﬁxed the beta r,ht score to 0.
Revised: To better understand what causes these results, we conducted experiments where we analyzed the impact of the beta r,ht scores. First, we ﬁxed the beta r,ht score to 0.

Original: This means that the rankings are completely determined by the KGE scores, while the candidates that are ranked are those provided by AnyBURL.
Revised: This means that the rankings are completely determined by the KGE scores, while the candidates that are ranked are those provided by AnyBURL.

Original: By doing this, we are able to use our method as a ﬁltering technique that suppresses any artifact from the random initialisation and anything based on a similarity consideration not backed by observable regularities.
Revised: By doing this, we are able to use our method as a ﬁltering technique that suppresses any artifact from the random initialisation and anything based on a similarity consideration not backed by a regularity that corresponds to a rule learned by AnyBURL.

Original: It turned out that the resulting MRR is always between the original KGE score and the MRR of the aggregated results.
Revised: It turned out that the resulting MRR is always between the original KGE score and the MRR of the aggregated results.

Original: We depicted this score as a mark within the vertical lines in Figure 1.
Revised: We depict this score as a mark within the vertical lines in Figure 1.

Original: Detailed results of these experiments can be found in Table 4.
Revised: Detailed results of these experiments can be found in Table 5.

Original: It seems that 1 / 4 to 1 / 2 of the positive impactof our approach can be explained by ﬁltering out what is not predicted by AnyBURL.
Revised: Between 1 / 4 and 1 / 2 of the positive impact can be explained by ﬁltering out what is not predicted by AnyBURL.

Original: This result implies also that the KGE models are not capable of detecting anything that cannot be described in terms of the rules we presented in Section 3.1.
Revised: This result implies also that the KGE models are not capable of detecting anything that cannot be described in terms of the rules we presented in Section 3.1.

Revised: We further explore the behavior for diﬀerent values of β r,ht .

Original: 7. Conclusions
Revised: We show results of experiments for HittER* and ComplEx on CoDEx-L in Table 3. The β r,ht =0.0 setting corresponds to using AnyBURL as a ﬁlter as explained above. Setting β r,ht =1.0 corresponds to the original AnyBURL scores. In the line that shows results for β r,ht =0.5, both members of the ensemble have an equal and ﬁxed weight.

Revised: In the last row we show the results of our approach in its standard setting from Figure 1 where we allow to select the best β r,ht from { 0 , 0 . 05 , . . .

Revised: , 1 . 0 } based on the best MRRs on the validation set.

Revised: In the row above we restrict the search space to β r,ht ∈ { 0 , 1 } .

Original: Within this work we have not presented a sophisticated new knowledge base completion method.
Revised: The results show that an equally balanced approach can already improve the performance. However, learning an optimal β r,ht against the validation set yields further improvements. This holds especially for HittER*, which is the best model in our experiments. [Table]

Revised: Moreover, learning a relation and direction speciﬁc weighting is superior to selecting one of the two approaches for each relation/direction, which is reﬂected by the β r,ht ∈ { 0 , 1 } setting which is 0.7 and 1.1 percentage points worse compared to the main results.

Revised: In Tables 9 to 12 in the Appendix, for the 10 most frequent relations of CoDEx-L we present the β r,ht values that resulted in the best MRR scores against the validation set in regard to the last two settings of Table 3.

Original: Instead of  that  we tried to understand the pros and cons of KGE models and rule-based approaches by looking at the rankings that they generate.
Revised: A β r,ht of 0.0 means that the KGE model determines the ranking, while a value of 1.0 means that the ranking is completely determined by AnyBURL. The β r,ht scores vary between the values of the respective search spaces, indicating that it a weighting where none of the two ensemble members is ignored is beneﬁcial for most relations. 7. Conclusions Instead of presenting a sophisticated new knowledge base completion method, in this work, we tried to understand advantages and disadvantages of KGE models and rule-based approaches by analysing the rankings that they generate.

Original: Our means to achieve this goal was the use of the explanatory power of a rule based approach.
Revised: Our means to achieve this goal was the use of the explanatory power of a rule based approach.

Original: Thus, we were able to spot and understand several examples of interesting predictions that revealed some essential diﬀerences.
Revised: Thus, we were able to spot and understand several examples of interesting predictions that revealed some essential diﬀerences.

Original: As a consequence of our understanding we developed an approach that increases the quality of an already top-performing KGE approach very constantly by 1% to 3% in terms of the MRR.
Revised: As a consequence of our understanding, we developed an approach that increases the quality of an already top-performing KGE approach consistently by 1 to 3 percentage points in terms of the MRR.

Original: While the prediction quality of our aggregation method is in itself a valuable results, it is more important to understand what follow from these results:
Revised: While the prediction quality of our aggregation method is in itself a valuable results, it is more important to understand what follow from these results:

Original: • KGE models are good in combining/aggregating diﬀerent signals.
Revised: • KGE models are good in combining/aggregating diﬀerent signals.

Original: • KGE models can suﬀer from relicts of the random initialization.
Revised: • KGE models can suﬀer from relicts of the random initialization.

Original: • KGE scores are aﬀected by similarity considerations that are sometimes unreasonable.
Revised: • KGE scores are aﬀected by similarity considerations that are sometimes unreasonable.

Original: • Rule based approaches are better in detecting signals that can be explained in terms of (relatively) long rules.
Revised: • Rule based approaches are better in detecting signals that can be explained in terms of (relatively) long rules.

Original: • KGE models remain within the language scope described in Section 
Revised: • KGE models remain within the language scope described in Section 3.1, which means that we can use the rules of AnyBURL to explain the predictions of the KGE model.

Original: These insights explain why a naive way to combine symbolic and latent knowledge graph completion techniques works surprisingly well.
Revised: These insights explain why a naive way to combine symbolic and latent knowledge graph completion techniques works surprisingly well.

Original: Appendix A. Detailed Results
Revised: Appendix A. Detailed Results

Original: Table 3 shows detailed results in terms of ﬁltered hits@1, hits@10 and MRR (based on the top-100 predictions only) for all combinations of KGE models and datasets.
Revised: Table 4 shows detailed results in terms of ﬁltered hits@1, hits@10 and MRR (based on the top-100 predictions only) for all combinations of KGE models and datasets.

Original: Rules/KGE Results
Revised: Rules/KGE Results

Original: Aggregated Results
Revised: Aggregated Results

Original: Improvements
Revised: Improvements

Original: Approach h@ h@
Revised: Approach h@ h@

Original: MRR h@ h@ MRR h@ h@
Revised: MRR h@ h@ MRR h@ h@

Original: MRR
Revised: MRR

Original: F B
Revised: [Table]

Original: AnyBURL .
Revised: AnyBURL .

Original: ComplEx .
Revised: ComplEx .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: ConvE .
Revised: ConvE .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: DistMult .
Revised: DistMult .

Original: +.
Revised: +.

Original: +. +.
Revised: +. +.

Original: HittER* .
Revised: HittER* .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: RESCAL .
Revised: RESCAL .

Original: +.
Revised: +.

Original: +. +.
Revised: +. +.

Original: TransE .
Revised: TransE .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: WN
Revised: W

Revised: [Table]

Original: RR
Revised: RR

Original: AnyBURL .
Revised: AnyBURL .

Original: ComplEx .
Revised: ComplEx .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: ConvE .
Revised: ConvE .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: DistMult .
Revised: DistMult .

Original: +.
Revised: +.

Original: +. +.
Revised: +. +.

Original: HittER* .
Revised: HittER* .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: RESCAL .
Revised: RESCAL .

Original: +.
Revised: +.

Original: +. +.
Revised: +. +.

Original: TransE .
Revised: TransE .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: C o
Revised: C o

Original: D E x - S
Revised: D E x - S

Original: AnyBURL .
Revised: AnyBURL .

Original: ComplEx .
Revised: ComplEx .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: ConvE .
Revised: ConvE .

Original: +.
Revised: +.

Original: +. +.
Revised: +. +.

Original: HittER* .
Revised: HittER* .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: RESCAL .
Revised: RESCAL .

Original: +.
Revised: +.

Original: +. +.
Revised: +. +.

Original: TransE .
Revised: TransE .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: TuckER .
Revised: TuckER .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: C o
Revised: C o

Original: D E x - M
Revised: D E x - M

Original: AnyBURL .
Revised: AnyBURL .

Original: ComplEx .
Revised: ComplEx .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: ConvE .
Revised: ConvE .

Original: +.
Revised: +.

Original: +. +.
Revised: +. +.

Original: HittER* .
Revised: HittER* .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: RESCAL .
Revised: RESCAL .

Original: +.
Revised: +.

Original: +. +.
Revised: +. +.

Original: TransE .
Revised: TransE .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: TuckER .
Revised: TuckER .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: C o
Revised: C o

Original: D E x - L
Revised: D E x - L

Original: AnyBURL .
Revised: AnyBURL .

Original: ComplEx .
Revised: ComplEx .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: ConvE .
Revised: ConvE .

Original: +.
Revised: +.

Original: +. +.
Revised: +. +.

Original: HittER* .
Revised: HittER* .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: RESCAL .
Revised: RESCAL .

Original: +.
Revised: +.

Original: +. +.
Revised: +. +.

Original: TransE .
Revised: TransE .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: TuckER .
Revised: TuckER .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: Table 4 is structured similar to Table 3, however, this time the results are based using the
Revised: Table 5 is structured similar to Table 4, however, this time the results are based using the

Original: AnyBURL ranking only for the purpose of ﬁltering the KGE ranking.
Revised: AnyBURL ranking only for the purpose of ﬁltering the KGE ranking.

Original: This means that we keep the order of the KGE ranking and suppress each candidate that is not within the top-100 ranking of AnyBURL.
Revised: This means that we keep the order of the KGE ranking and suppress each candidate that is not within the top-100 ranking of AnyBURL.

Original: The empty positions in the ranking are ﬁlled up by the subsequent candidates.
Revised: The empty positions in the ranking are ﬁlled up by the subsequent candidates.

Original: Rules/KGE Results

Original: [Table]
Revised: [Table]

Revised: Improvements

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Revised: MRR

Original: [Table]
Revised: [Table]

Revised: AnyBURL .

Original: [Table] [Table]
Revised: ComplEx . +. +.

Original: [Table]
Revised: +. ConvE .

Original: [Table]
Revised: +. +. +. DistMult .

Original: [Table] [Table]
Revised: +. +. +.

Original: [Table] [Table]
Revised: HittER* . +. +.

Original: [Table] [Table]
Revised: +. RESCAL . +.

Original: [Table] [Table]
Revised: +. +. TransE .

Original: [Table]
Revised: +. +. +.

Original: W N
Revised: W N

Original: RR
Revised: RR

Original: [Table]
Revised: AnyBURL .

Original: [Table] [Table]
Revised: ComplEx . +. +.

Original: [Table]
Revised: +. ConvE . +. +.

Original: +.
Revised: +.

Revised: DistMult .

Original: +. +. 
Revised: +. +. +.

Original: HittER* .
Revised: HittER* .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: RESCAL .
Revised: RESCAL .

Original: +.
Revised: +.

Original: +. +.
Revised: +. +.

Original: TransE .
Revised: TransE .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: C o
Revised: C o

Original: D E x - M
Revised: D E x - M

Original: AnyBURL .
Revised: AnyBURL .

Original: ComplEx .
Revised: ComplEx .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: ConvE .
Revised: ConvE .

Original: +.
Revised: +.

Original: +. +.
Revised: +. +.

Original: HittER* .
Revised: HittER* .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: RESCAL .
Revised: RESCAL .

Original: +.
Revised: +.

Original: +. +.
Revised: +. +.

Original: TransE .
Revised: TransE .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: TuckER .
Revised: TuckER .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: C o
Revised: C o

Original: D E x - L
Revised: D E x - L

Original: AnyBURL .
Revised: AnyBURL .

Original: ComplEx .
Revised: ComplEx .

Original: ConvE .
Revised: ConvE .

Original: HittER* .
Revised: HittER* .

Original: [Table]
Revised: [Table]

Original: In addition to the visualisation shown in the main paper, we prepared the same diagram for CoDEx-S, M and L in Figure 2.
Revised: In addition to the visualisation shown in the main paper, we prepared the same diagram for CoDEx-S, M and L in Figure 2.

Original: Note that the validation set of CoDEx-S is relatively small, which might explain why the method works not so well for some models.
Revised: Note that the validation set of CoDEx-S is relatively small, which might explain why the method works not so well for some models.

Original: CoDEx-S
Revised: CoDEx-S

Original: MRR .
Revised: MRR .

Original: AnyBURL .
Revised: AnyBURL .

Original: CoDEx-M
Revised: CoDEx-M

Original: MRR .
Revised: MRR .

Original: AnyBURL .
Revised: AnyBURL .

Original: CoDEx-L
Revised: CoDEx-L

Original: MRR .
Revised: MRR .

Original: AnyBURL
Revised: AnyBURL

Original: TuckER
Revised: TuckER

Original: TransE
Revised: TransE

Original: RESCAL
Revised: RESCAL

Original: HittER*
Revised: HittER*

Original: ConvE
Revised: ConvE

Original: ComplEx
Revised: ComplEx

Original: Appendix B. Additional Examples
Revised: Appendix B. Additional Examples

Original: In the following we present two additional examples.
Revised: In the following we present two additional examples.

Original: The ﬁrst example illustrates a behaviour similar to what we discussed with the Darvin/Australia example .
Revised: The ﬁrst example illustrates a behaviour similar to what we discussed with the Darvin/Australia example .

Original: The second example is related to a speciﬁc pattern.
Revised: The second example is related to a speciﬁc pattern.

Original: We have also developed a simple method to measure the impact of suppressing this pattern and report about its results.
Revised: We have also developed a simple method to measure the impact of suppressing this pattern and report about its results.

Original: Who produced Tomb Raider
Revised: Who produced Tomb Raider

Original: We are now concerned with the completion task producedBy(TombRaiderI,?) .
Revised: We are now concerned with the completion task producedBy(TombRaiderI,?) .

Original: We use TombRaiderI/II to refer to the ﬁrst and second movie from the series (in training is a prequel relation between these movies).
Revised: We use TombRaiderI/II to refer to the ﬁrst and second movie from the series (in training is a prequel relation between these movies).

Original: The producer of both movies is Lawrence Gordon (in test for TombRaiderI; in training for TombRaiderII).
Revised: The producer of both movies is Lawrence Gordon (in test for TombRaiderI; in training for TombRaiderII).

Original: Angeline Jolie acted in both movies as Lara Croft (both triples are available in training).
Revised: Angeline Jolie acted in both movies as Lara Croft (both triples are available in training).

Original: It is also relevant that Angelina Jolie produced several other movies.
Revised: It is also relevant that Angelina Jolie produced several other movies.

Original: AnyBURL ranks the correct answer on position #1.
Revised: AnyBURL ranks the correct answer on position #1.

Original: This is caused by a rule with two atoms in the body that captures the regularity that the producer of a movie has produced the prequel of this movie with a probability of 0.445.
Revised: This is caused by a rule with two atoms in the body that captures the regularity that the producer of a movie has produced the prequel of this movie with a probability of 0.445.

Original: ComplEx ranks Angelina
Revised: ComplEx ranks Angelina

Original: Jolie ﬁrst.
Revised: Jolie ﬁrst.

Original: What might be the reason for this result?
Revised: What might be the reason for this result?

Original: It happens sometimes that a movie is produced by one of the persons acting in that movie.
Revised: It happens sometimes that a movie is produced by one of the persons acting in that movie.

Original: Moreover, the producedBy and the actedIn relations establish connections between the same types of entities.
Revised: Moreover, the producedBy and the actedIn relations establish connections between the same types of entities.

Original: In that sense the embedding of these relations might be similar.
Revised: In that sense the embedding of these relations might be similar.

Original: Thus, as it is known that Angelina Jolie acted in TombRaider I, the triples actedIn(TombRaider I, Angelina Jolie) must have a high score, and thus, the triple producedBy(TombRaider I, Angelina Jolie) will also receive a relatively high score.
Revised: Thus, as it is known that Angelina Jolie acted in TombRaider I, the triples actedIn(TombRaider I, Angelina Jolie) must have a high score, and thus, the triple producedBy(TombRaider I, Angelina Jolie) will also receive a relatively high score.

Original: On the other hand, the regularity that is captured by the rule that triggered AnyBURLs decision seems to aﬀect the results only to a limited degree.
Revised: On the other hand, the regularity that is captured by the rule that triggered AnyBURLs decision seems to aﬀect the results only to a limited degree.

Original: Robert Schumann did not inﬂuence Robert Schumann
Revised: Robert Schumann did not inﬂuence Robert Schumann

Original: A correct answer to the completion task inﬂuencedBy(schumann,?) is (beside others) the composer Felix Mendelsohn.
Revised: A correct answer to the completion task inﬂuencedBy(schumann,?) is (beside others) the composer Felix Mendelsohn.

Original: However, ComplEx puts Robert Schumann himself on the ﬁrst rank resulting in the triple inﬂuencedBy(schumann, schumann) .
Revised: However, ComplEx puts Robert Schumann himself on the ﬁrst rank resulting in the triple inﬂuencedBy(schumann, schumann) .

Original: The correct answer (Mendelsohn) is ranked on position #9.
Revised: The correct answer (Mendelsohn) is ranked on position #9.

Original: Note that we observed many similar cases where a triple r ( a, a ) was predicted, even though relation r is, as it is the case for the inﬂuencedBy relation, irreﬂexive.
Revised: Note that we observed many similar cases where a triple r ( a, a ) was predicted, even though relation r is, as it is the case for the inﬂuencedBy relation, irreﬂexive.

Original: While this sounds like a rather speciﬁc issue, it can again be explained by misleading similarity
Revised: While this sounds like a rather speciﬁc issue, it can again be explained by misleading similarity

Original: [Table]
Revised: [Table]

Revised: Angelina Jolie

Revised: [Table]

Revised: Michael

Revised: [Table]

Revised: Wilson

Revised: [Table]

Revised: Spielberg

Revised: [Table]

Original: Kathleen 
Revised: Kathleen Kennedy [Table] Lawrence [Table]

Revised: Lawrence

Original: [Table]
Revised: [Table]

Original:  Schumann
Revised: Robert Schumann

Revised: [Table]

Revised: Arnold Schoenberg

Original: [Table]
Revised: [Table]

Original: Victor Hugo
Revised: Victor Hugo

Original: [Table]
Revised: [Table]

Original: Bach
Revised: Bach

Original: [Table]
Revised: [Table]

Original: Thomas Mann
Revised: Thomas Mann

Original: [Table]
Revised: [Table]

Original: Spinoza
Revised: Spinoza

Original: [Table]
Revised: [Table]

Original: Schopenhauer
Revised: Schopenhauer

Original: [Table]
Revised: [Table]

Revised: Mendelssohn

Revised: [Table]

Original: Jean-Jacques Rousseau
Revised: Jean-Jacques Rousseau

Original: [Table]
Revised: [Table]

Original: Mendelssohn
Revised: Mendelssohn

Original: [Table]
Revised: [Table]

Original:  Freud
Revised: Sigmund Freud

Original: [Table]
Revised: [Table]

Revised: Mendelssohn

Revised: [Table]

Original: Aleksandr Pushkin
Revised: Aleksandr Pushkin

Original: [Table]
Revised: [Table]

Original: Friedrich Hayek
Revised: Friedrich Hayek

Original: [Table]
Revised: [Table]

Original: consideration.
Revised: consideration.

Original: There several triples given in the training set, which tell us about people that have inﬂuenced Robert Schumann.
Revised: There several triples given in the training set, which tell us about people that have inﬂuenced Robert Schumann.

Original: Robert Schumann seems to be very similar to the average inﬂuencer of Schumann.
Revised: Robert Schumann seems to be very similar to the average inﬂuencer of Schumann.

Original: For that reason he is proposed as a solution to the completion task.
Revised: For that reason he is proposed as a solution to the completion task.

Original: Some predictions for AnyBURL, ComplEx and RESCAL are shown in Table 5.
Revised: Some predictions for AnyBURL, ComplEx and RESCAL are shown in Table 6.

Original: Note also that AnyBURL puts the correct answer to a rather low ranking position.
Revised: Note also that AnyBURL puts the correct answer to a rather low ranking position.

Original: However, that is not the point here.
Revised: However, that is not the point here.

Original: The main point here is related to the fact that an alternative that is clearly wrong, given the fact that we never observe inﬂuencedBy(c,c) for any constant c in the training set, is placed at a top position by the KGE approach due to misleading similarity considerations.
Revised: The main point here is related to the fact that an alternative that is clearly wrong, given the fact that we never observe inﬂuencedBy(c,c) for any constant c in the training set, is placed at a top position by the KGE approach due to misleading similarity considerations.

Original: We applied the following strategy to remove erroneous self-referential predictions from the ranking.
Revised: We applied the following strategy to remove erroneous self-referential predictions from the ranking.

Original: For each relation r We checked in the training set, whether there is a triples r ( c, c ).
Revised: For each relation r We checked in the training set, whether there is a triples r ( c, c ).

Original: If such a triples does not exist, we marked the relation as irreﬂexive.
Revised: If such a triples does not exist, we marked the relation as irreﬂexive.

Original: For each irreﬂexive relation r , given the prediction task r ( c, ?) or r (? , c ) we removed c from the candidate ranking.
Revised: For each irreﬂexive relation r , given the prediction task r ( c, ?) or r (? , c ) we removed c from the candidate ranking.

Original: Note that this ﬁltering technique removes exactly one candidate or no candidates from a given ranking.
Revised: Note that this ﬁltering technique removes exactly one candidate or no candidates from a given ranking.

Original: The results of the approach are depicted in Table 6.
Revised: The results of the approach are depicted in Table 7.

Original: Rules/KGE Results
Revised: Rules/KGE Results

Original: [Table]
Revised: [Table]

Original: Approach h@ h@ MRR h@ h@
Revised: MRR h [Table] h

Revised: [Table]

Original: MRR h@ h@
Revised: MRR h@ h@

Original: MRR
Revised: MRR

Original: [Table]
Revised: [Table]

Original: ComplEx .
Revised: [Table] [Table] [Table]

Original: +.
Revised: +.

Original: +. +.
Revised: +. +

Original: ConvE .
Revised: [Table] [Table]

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: DistMult .
Revised: [Table] [Table] [Table]

Original: +.
Revised: +.

Original: +. +.
Revised: +. +

Original: HittER* .
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Original: +.
Revised: +

Original: +.
Revised: [Table] [Table]

Original: +.
Revised: +.

Original: RESCAL .
Revised: RESCAL .

Original: +.
Revised: +.

Original: +. +.
Revised: +. +.

Original: TransE .
Revised: TransE .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: W N
Revised: W N

Original: RR
Revised: RR

Original: ComplEx .
Revised: ComplEx .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: ConvE .
Revised: ConvE .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: DistMult .
Revised: DistMult .

Original: +.
Revised: +.

Original: +. +.
Revised: +. +.

Original: HittER* .
Revised: HittER* .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: RESCAL .
Revised: RESCAL .

Original: +.
Revised: +.

Original: +. +.
Revised: +. +.

Original: TransE .
Revised: TransE .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: C o
Revised: C o

Original: D E x - M
Revised: D E x - M

Original: ComplEx .
Revised: ComplEx .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: ConvE .
Revised: ConvE .

Original: +.
Revised: +.

Original: +. +.
Revised: +. +.

Original: HittER* .
Revised: HittER* .

Original: +. +.
Revised: +. +.

Original: +.
Revised: +.

Original: RESCAL .
Revised: RESCAL .

Original: +.
Revised: +.

Original: +. +.
Revised: +. +.

Original: TransE .
Revised: TransE .

Original: +.
Revised: +.

Original: +. +.
Revised: +. +

Original: TuckER

Original: [Table]
Revised: [Table]

Original: We can see that the geometric embedding technique used by TransE seems to be aﬀected much more compared to the other KGE models.
Revised: We can see that the geometric embedding technique used by TransE seems to be aﬀected much more compared to the other KGE models.

Original: If we focus on the other models, we see an improvement of .000 to 0.006 in hits@1.
Revised: If we focus on the other models, we see an improvement of .000 to 0.006 in hits@1.

Original: It is interesting to see that hits@10 scores is much less aﬀected.
Revised: It is interesting to see that hits@10 scores is much less aﬀected.

Original: This holds especially for CoDEx-M.
Revised: This holds especially for CoDEx-M.

Original: We can conclude that in most cases, where we observe a positive impact of this speciﬁc ﬁltering approach, the self-referential prediction is on #1 and the correct alternative is on #2.
Revised: We can conclude that in most cases, where we observe a positive impact of this speciﬁc ﬁltering approach, the self-referential prediction is on #1 and the correct alternative is on #2.

Original: A number as 0.005 means (for example, HittER* on CoDEx-M), that we observe this pattern in 1 out of 200 completion tasks.
Revised: A number as 0.005 means (for example, HittER* on CoDEx-M), that we observe this pattern in 1 out of 200 completion tasks.

Original: This speciﬁc ﬁltering technique is thus, much weaker, compared to the ﬁltering that is based on the top-100 AnyBURL ranking.
Revised: This speciﬁc ﬁltering technique is thus, much weaker, compared to the ﬁltering that is based on the top-100 AnyBURL ranking.

Original: Appendix C. Additional Rule Types
Revised: Appendix C. Additional Rule Types

Original: AnyBURL can mine more general rules than the rules shown in Section 3 by replacing the constant in the body by an unbound variable.
Revised: AnyBURL can mine more general rules than the rules shown in Section 3 by replacing the constant in the body by an unbound variable.

Original: Some examples are listed in the following.
Revised: Some examples are listed in the following.

Original: [Equation]
Revised: [Equation]

Original: Rule (8) speciﬁes the probability that something that has a profession (no matter what profession) is female.
Revised: Rule (8) speciﬁes the probability that something that has a profession (no matter what profession) is female.

Original: Note that there is also a rule with the same body and the head gender ( X, male ).
Revised: Note that there is also a rule with the same body and the head gender ( X, male ).

Original: Such rules specify a value distribution for a certain type of entities.
Revised: Such rules specify a value distribution for a certain type of entities.

Original: They have in most cases a rather low conﬁdence and become relevant if nothing else is known that can be understood as a stronger signal.
Revised: They have in most cases a rather low conﬁdence and become relevant if nothing else is known that can be understood as a stronger signal.

Original: The ﬁnal type of rules is a new type that has been added to the latest version of
Revised: The ﬁnal type of rules is a new type that has been added to the latest version of

Original: AnyBURL recently.
Revised: AnyBURL recently.

Original: This rule type is based on the fact that asking the knowledge base completion task r ( a, ?)
Revised: This rule type is based on the fact that asking the knowledge base completion task r ( a, ?)

Original: implies that there is a correct answer to ?.
Revised: implies that there is a correct answer to ?.

Original: The conﬁdence of such a rule speciﬁes the probability that a randomly chosen r -triple has a speciﬁc value in its subject or object position without considering any other information than the distribution of values.
Revised: The conﬁdence of such a rule speciﬁes the probability that a randomly chosen r -triple has a speciﬁc value in its subject or object position without considering any other information than the distribution of values.

Original: Here are some examples for these rules.
Revised: Here are some examples for these rules.

Original: [Equation]
Revised: [Equation]

Original: Without these rules, AnyBURL would sometimes create an empty candidate ranking.
Revised: Without these rules, AnyBURL would sometimes create an empty candidate ranking.

Original: These rules yield a kind of default answer to avoid such cases.
Revised: These rules yield a kind of default answer to avoid such cases.

Original: However, in some situations these rules have an inﬂuence that is too strong.
Revised: However, in some situations these rules have an inﬂuence that is too strong.

Original: Thus, their conﬁdence is multiplied by 0.
Revised: Thus, their conﬁdence is multiplied by 0.

Original: (for Rule (8) and (9)) and 0.01 (Rule (10) and (11)).
Revised: (for Rule (8) and (9)) and 0.01 (Rule (10) and (11)).

Original: Without this modiﬁcations important entities, e.g., USA, are too often ranked high without any speciﬁc reason.
Revised: Without this modiﬁcations important entities, e.g., USA, are too often ranked high without any speciﬁc reason.

Original: This setting is currently the default setting of AnyBURL, which is constantly used across all datasets.
Revised: This setting is currently the default setting of AnyBURL, which is constantly used across all datasets.

Original: Appendix D. Summary Statistics and Datasets
Revised: Appendix D. Summary Statistics and Datasets

Original: [Table]
Revised: [Table]

Original: Appendix E. Normalizing KGE Scores
Revised: Appendix E. Normalizing KGE Scores

Original: As KGE scores can be scattered over a wide value range, we need to normalize the score before we compute the aggregated score.
Revised: As KGE scores can be scattered over a wide value range, we need to normalize the score before we compute the aggregated score.

Original: We could, for example, map the score of the # rank to 1.0 and the score of the #k rank to 0.
Revised: We could, for example, map the score of the # rank to 1.0 and the score of the #k rank to 0.

Original: The problem of this approach becomes clear when looking at the completion tasks locatedIn(?,US) and locatedIn(?,Liechtenstein) .
Revised: The problem of this approach becomes clear when looking at the completion tasks locatedIn(?,US) and locatedIn(?,Liechtenstein) .

Original: If we set k to 100 (the parameter setting we choose in our experiments)
Revised: If we set k to 100 (the parameter setting we choose in our experiments)

Original: we will probably have a situation where the ﬁrst completion task might yield reasonable candidates within the top-k ranking.
Revised: we will probably have a situation where the ﬁrst completion task might yield reasonable candidates within the top-k ranking.

Original: For the second ranking this might not be the case, and only a few top candidates are correct.
Revised: For the second ranking this might not be the case, and only a few top candidates are correct.

Original: Nevertheless, we would map the candidate at position #100 in both cases on 0 and in the US case a signiﬁcantly higher value would be more appropriate.
Revised: Nevertheless, we would map the candidate at position #100 in both cases on 0 and in the US case a signiﬁcantly higher value would be more appropriate.

Original: Instead of that, we map the KGE score to [ min, max ] with max = conf
Revised: Instead of that, we map the KGE score to [ min, max ] with max = conf

Original: ( A [1] , r ( e, ?)) and min = conf ( A [ k ] , r ( e, ?)), i.e., we map to a range that is deﬁned by the conﬁdence of ﬁrst and last candidate in the AnyBURL ranking.
Revised: ( A [1] , r ( e, ?)) and min = conf ( A [ k ] , r ( e, ?)), i.e., we map to a range that is deﬁned by the conﬁdence of ﬁrst and last candidate in the AnyBURL ranking.

Original: We are aware that there might be more sophisticated techniques, however, for our purpose this approach turned out to work quite well.
Revised: We are aware that there might be more sophisticated techniques, however, for our purpose this approach turned out to work quite well.

Original: We abstained from other approaches that deﬁnes the mapping on a global level by looking at all possible scores that can be derived from the embedding space.
Revised: We abstained from other approaches that deﬁnes the mapping on a global level by looking at all possible scores that can be derived from the embedding space.

Original: Such an approach has to solve the inherent problem that there is probably a wide range of scores that belong to clearly wrong triples.
Revised: Such an approach has to solve the inherent problem that there is probably a wide range of scores that belong to clearly wrong triples.

Revised: Appendix F. Further Results Related to the Ablation Study

Revised: In the following Tables

Revised: [Table]

Revised: we present the best relation speciﬁc MRR results and β r,ht values in diﬀerent settings.

Revised: Tables 9 and 11 list the values learned with the default setting that we used to achieve the main results shown in Figure 1 for CoDEx-L for the models ComplEx and HittER*, respectively.

Revised: In general, the values for β r,ht vary between the diﬀerent values of the search space demonstrating the complementary nature of the two approaches (latent vs. symbolic).

Revised: In Tables 10 and 12 we report values that were learned in a setting where the search space for β is restricted to { 0 . 0 , 1 . 0 } .

Revised: This mimics a behaviour where a hard selection between the two approaches must be made for each direction/relation in the dataset.

Revised: For

Revised: ComplEx in Table

Revised: [Table]

Revised: the β values are almost equal distributed whereas for HittER* most of the time β is selected to be 0.

Revised: A possible reason is the fact that we show these results for the most frequent relations.

Revised: We found a more equally distributed pattern when considering all the relations of CoDEx-L.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: the 10 most frequent relations.

Revised: [Table]