Original: Rethinking Knowledge Graph Evaluation Under the Open-World Assumption
Revised: Rethinking Knowledge Graph Evaluation Under the Open-World Assumption

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Most knowledge graphs (KGs) are incomplete, which motivates one importantresearch topic on automatically complementing knowledge graphs.
Revised: Most knowledge graphs (KGs) are incomplete, which motivates one important research topic on automatically complementing knowledge graphs.

Original: However,evaluation of knowledge graph completion (KGC) models often ignores theincompleteness—facts in the test set are ranked against all unknown triplets whichmay contain a large number of missing facts not included in the KG yet.
Revised: However, evaluation of knowledge graph completion (KGC) models often ignores the incompleteness—facts in the test set are ranked against all unknown triplets which may contain a large number of missing facts not included in the KG yet.

Original: Treating allunknown triplets as false is called the closed-world assumption.
Revised: Treating all unknown triplets as false is called the closed-world assumption.

Original: This closed-worldassumption might negatively affect the fairness and consistency of the evaluationmetrics.
Revised: This closed-world assumption might negatively affect the fairness and consistency of the evaluation metrics.

Original: In this paper, we study KGC evaluation under a more realistic setting,namely the open-world assumption , where unknown triplets are considered toinclude many missing facts not included in the training or test sets.
Revised: In this paper, we study KGC evaluation under a more realistic setting, namely the open-world assumption , where unknown triplets are considered to include many missing facts not included in the training or test sets.

Original: For the currently most used metrics such as mean reciprocal rank (MRR) and Hits@K, wepoint out that their behavior may be unexpected under the open-world assumption.
Revised: For the currently most used metrics such as mean reciprocal rank (MRR) and Hits@K, we point out that their behavior may be unexpected under the open-world assumption.

Original: Speciﬁcally, with not many missing facts, their numbers show a logarithmic trendwith respect to the true strength of the model, and thus, the metric increase could beinsigniﬁcant in terms of reﬂecting the true model improvement.
Revised: Specifically, with not many missing facts, their numbers show a logarithmic trend with respect to the true strength of the model, and thus, the metric increase could be insignificant in terms of reflecting the true model improvement.

Original: Further, considering the variance, we show that the degradation in the reported numbers may resultin incorrect comparisons between different models, where stronger models mayhave lower metric numbers.
Revised: Further, considering the variance, we show that the degradation in the reported numbers may result in incorrect comparisons between different models, where stronger models may have lower metric numbers.

Original: We validate the phenomenon both theoretically andexperimentally.
Revised: We validate the phenomenon both theoretically and experimentally.

Original: Finally, we suggest possible causes and solutions for this problem.
Revised: Finally, we suggest possible causes and solutions for this problem.

Revised: Our code and data are available at https://github.com/GraphPKU/Open-World-KG.

Original:  Introduction
Revised:  Introduction

Original: Knowledge graph (KG) is a structural method to store facts about some ﬁeld or the world.
Revised: Knowledge graph (KG) is a structural method to store facts about some field or the world.

Original: Becausemost KGs are incomplete, the knowledge graph completion (KGC) task is proposed to automaticallycomplement the existing KG with missing facts.
Revised: Because most KGs are incomplete, the knowledge graph completion (KGC) task is proposed to automatically complement the existing KG with missing facts.

Original: However, when we do not know the missing factsin advance, we must manually evaluate whether each predicted completion is correct, which is animpossible task for modern KGs.
Revised: However, when we do not know the missing facts in advance, we must manually evaluate whether each predicted completion is correct, which is an impossible task for modern KGs.

Original: This problem is called the open-world problem and the assumptionthat KGs are incomplete is called the open-world assumption .
Revised: This problem is called the open-world problem and the assumption that KGs are incomplete is called the open-world assumption .

Original: A general solution is to extract thetraining, validation and test sets from the existing incomplete KG and then evaluate the trained modelson the test set.
Revised: A general solution is to extract the training, validation and test sets from the existing incomplete KG and then evaluate the trained models on the test set.

Original: Then, a natural question is whether the conclusion drawn from the incomplete test setis consistent with the true strength of the model, which should be measured on the complete KG.
Revised: Then, a natural question is whether the conclusion drawn from the incomplete test set is consistent with the true strength of the model, which should be measured on the complete KG.

Original: To answer this question, we need to investigate the metrics used to evaluate KGC models.
Revised: To answer this question, we need to investigate the metrics used to evaluate KGC models.

Original: KGCmodels are often evaluated by ranking-based metrics, such as mean reciprocal rank (MRR) and 
Revised: KGC models are often evaluated by ranking-based metrics, such as mean reciprocal rank (MRR) and [Table]

Original: Hits@K. Under the open-world assumption, when a missing fact that should have been included inthe test answers is predicted by the model, its ranking could be higher than some test answers,which makes the rankings of these test answers drop .
Revised: Hits@K. Under the open-world assumption, when a missing fact that should have been included in the test answers is predicted by the model, its ranking could be higher than some test answers, which makes the rankings of these test answers drop .

Original: In this situation, despite actually recognizingmore right answers, the metrics drop instead.
Revised: In this situation, despite actually recognizing more right answers, the metrics drop instead.

Original: [Table] To intuitively show the problem, we train BetaE [Ren and Leskovec, 2020], one state-of-the-artmulti-hop KGC model, on the FB15k-237 dataset [Toutanova and Chen, 2015].
Revised: To intuitively show the problem, we train BetaE [Ren and Leskovec, 2020], one state-of-the-art multi-hop KGC model, on the FB15k-237 dataset [Toutanova and Chen, 2015].

Original: One of the testqueries is “ What sports were included in the 1956 Summer Olympics? ”.
Revised: One of the test queries is “ What sports were included in the 1956 Summer Olympics? ”.

Original: The two test answers areswimming and sailing, both with the ﬁltered rankings (refer to Section 2) of 5, so the MRR on thisquery is 20% .
Revised: The two test answers are swimming and sailing, both with the filtered rankings (refer to Section 2) of 5, so the MRR on this query is 20% .

Original: However, when we manually check the ﬁrst 30 predictions , we found that many ofthem are in fact sports included in the 1956 Summer Olympics but not included in the answer set.
Revised: However, when we manually check the first 30 predictions , we found that many of them are in fact sports included in the 1956 Summer Olympics but not included in the answer set.

Original: We present these missing answers in Table 1.
Revised: We present these missing answers in Table 1.

Original: We can see that all the four sports previously rankinghigher than the two test answers turn out to be missing true answers.
Revised: We can see that all the four sports previously ranking higher than the two test answers turn out to be missing true answers.

Original: Thus, if we correct the answerset by adding these missing answers to the test set, the actual ﬁltered rankings of the two test answersare both 1, and the corrected MRR on the new test set becomes 82% which is much higher than thereported 20% , indicating that the model strength on this query is signiﬁcantly underestimated.
Revised: Thus, if we correct the answer set by adding these missing answers to the test set, the actual filtered rankings of the two test answers are both 1, and the corrected MRR on the new test set becomes 82% which is much higher than the reported 20% , indicating that the model strength on this query is significantly underestimated.

Original: In this paper, we study the odd behavior of the ranking-based metrics under the open-world assumption, and summarize two problems affecting the KGC evaluation: 1) Metric Degradation .
Revised: In this paper, we study the odd behavior of the ranking-based metrics under the open-world assumption, and summarize two problems affecting the KGC evaluation: 1) Metric Degradation .

Original: It meansthat with the increasing of the actual model strength, the increasing of the reported metric becomesslower and slower.
Revised: It means that with the increasing of the actual model strength, the increasing of the reported metric becomes slower and slower.

Original: Thus, the reported metric might not be able to reﬂect the true model improvement.2)
Revised: Thus, the reported metric might not be able to reflect the true model improvement.

Original:  Metric Inconsistency .
Revised: 2) Metric Inconsistency .

Original: It means that when comparing two models, the model with lower reportedmetric may actually have better performance if we evaluate them on the complete KG.
Revised: It means that when comparing two models, the model with lower reported metric may actually have better performance if we evaluate them on the complete KG.

Original: Our main contributions include that: For the ﬁrst time, we theoretically analyze the evaluation of
Revised: Our main contributions include that: For the first time, we theoretically analyze the evaluation of

Original: KGC under the open-world assumption and point out the degradation and inconsistency problems.
Revised: KGC under the open-world assumption and point out the degradation and inconsistency problems.

Original: Furthermore, we suggest that the degradation and inconsistency may be related to the focus-on-topbehavior of the metrics, and provide a solution to relieve the two problems.
Revised: Furthermore, we suggest that the degradation and inconsistency may be related to the focus-on-top behavior of the metrics, and provide a solution to relieve the two problems.

Original: Finally, we verify thetheoretical analysis through experiments on an artiﬁcial closed-world KG.
Revised: Finally, we verify the theoretical analysis through experiments on an artificial closed-world KG.

Original:  Background and related work
Revised:  Background and related work

Original: Knowledge graph completion Current KGC models can be mainly categorized into three classes:logic-based, embedding-based, and neural-based.
Revised: Knowledge graph completion Current KGC models can be mainly categorized into three classes: logic-based, embedding-based, and neural-based.

Original: Logic-based models [Joseph and Riley, 1998,
Revised: Logic-based models [Joseph and Riley, 1998,

Original: Richardson and Domingos, 2006] use some explicit rules for KGC, which are manually provided ormined by some rule-mining methods, such as [Galárraga et al., 2013, Yang et al., 2017, Sadeghianet al., 2019].
Revised: Richardson and Domingos, 2006] use some explicit rules for KGC, which are manually provided or mined by some rule-mining methods, such as [Galárraga et al., 2013, Yang et al., 2017, Sadeghian et al., 2019].

Original: These models search through the existing KG and deduce missing facts according to thegiven rules.
Revised: These models search through the existing KG and deduce missing facts according to the given rules.

Original: However, this process can be time-consuming and noise-sensitive.
Revised: However, this process can be time-consuming and noise-sensitive.

Original: At the same time, ifthe KGs are highly incomplete, the performance could be poor.
Revised: At the same time, if the KGs are highly incomplete, the performance could be poor.

Original: Embedding-based models [Bordeset al., 2013, Yang et al., 2015, Trouillon et al., 2016, Sun et al., 2019] represent entities and relationsby learned vectors or tensors, where the possibility of a fact is measured by a score function .
Revised: Embedding-based models [Bordes et al., 2013, Yang et al., 2015, Trouillon et al., 2016, Sun et al., 2019] represent entities and relations by learned vectors or tensors, where the possibility of a fact is measured by a score function .

Original: Thesemodels have good scalability and can be applied to large and sparse KGs.
Revised: These models have good scalability and can be applied to large and sparse KGs.

Original: Some works aim togeneralize embedding-based models to more patterns [Trouillon et al., 2016, Abboud et al. , 2020]and more assumptions (such as multiple answers) [Vilnis et al., 2018, Ren et al., 2020, Abboud et al.
Revised: Some works aim to generalize embedding-based models to more patterns [Trouillon et al., 2016, Abboud et al., 2020] and more assumptions (such as multiple answers) [Vilnis et al., 2018, Ren et al., 2020, Abboud et al., 2020].

Original: One of the interesting directions is to consider multi-hop reasoning [Ren et al., 2020, Ren and
Revised: One of the interesting directions is to consider multi-hop reasoning [Ren et al., 2020, Ren and

Original: Leskovec, 2020, Zhang et al., 2021], where a query can be composed by several conditions, suchas “ Who is the Canadian and won the Turing Award ? ” Note that the open-world problem could bemore severe in the multi-hop setting, because missing in any condition leads to missing in the ﬁnalresults.
Revised: Leskovec, 2020, Zhang et al., 2021], where a query can be composed by several conditions, such as “ Who is the Canadian and won the Turing Award ? ” Note that the open-world problem could be more severe in the multi-hop setting, because missing in any condition leads to missing in the final results.

Original: Neural-based models combine neural networks with embeddings.
Revised: Neural-based models combine neural networks with embeddings.

Original: Dettmers et al.
Revised: Dettmers et al.

Original: [2018]and Nguyen et al.
Revised: [2018] and Nguyen et al.

Original: [2018] use a convolution networks as the score function to enlarge the capacity ofthe models.
Revised: [2018] use a convolution networks as the score function to enlarge the capacity of the models.

Original: Nathani et al.
Revised: Nathani et al.

Original: [2019], Vashishth et al.
Revised: [2019], Vashishth et al.

Original: [2020] and Wang et al.
Revised: [2020] and Wang et al.

Original: [2021] use graph neuralnetworks on the KGs to learn the embeddings or directly predict the links.
Revised: [2021] use graph neural networks on the KGs to learn the embeddings or directly predict the links.

Original: KG evaluation Current KGC evaluation resorts to manually split training, validation and test setsfrom the incomplete KG.
Revised: KG evaluation Current KGC evaluation resorts to manually split training, validation and test sets from the incomplete KG.

Original: Given a test query r ( e h , ?) (which entities have the relation r with the head entity e h ?), a typical method is to predict a score for all entities as the tail entity, rank all theentities, and then measure the average of a ranking-based function h ( r ) on the test answers.
Revised: Given a test query r ( e h , ?) (which entities have the relation r with the head entity e h ?), a typical method is to predict a score for all entities as the tail entity, rank all the entities, and then measure the average of a ranking-based function h ( r ) on the test answers.

Original: Here, themost-used metrics are MRR h ( r ) = 1 / r and the Hits@K h ( r ) =
Revised: Here, the most-used metrics are MRR h ( r ) = 1 / r and the Hits@K h ( r ) =

Original: I ( r ≤ K ) .
Revised: I ( r ≤ K ) .

Original: Because there could bemultiple answers for a query, the metrics should be ﬁltered , which means the answers in the trainingand test sets do not occupy a position so that the number of training and test answers does not affectthe metrics.
Revised: Because there could be multiple answers for a query, the metrics should be filtered , which means the answers in the training and test sets do not occupy a position so that the number of training and test answers does not affect the metrics.

Original: The details of the ﬁltering can be found in [Bordes et al., 2013].
Revised: The details of the filtering can be found in [Bordes et al., 2013].

Original: Due to the nonlinearityof most ranking-based metrics, some works have theoretically investigated their behavior.
Revised: Due to the nonlinearity of most ranking-based metrics, some works have theoretically investigated their behavior.

Original: Wanget al.
Revised: Wang et al.

Original: [2013] point out some ranking-based metrics always converge to 1 on different models as thenumber of objects to rank goes to inﬁnity, so that the performance of models is indistinguishable.
Revised: [2013] point out some ranking-based metrics always converge to 1 on different models as the number of objects to rank goes to infinity, so that the performance of models is indistinguishable.

Original: Krichene and Rendle [2020] analyze the behavior of ranking-based metrics under negative sampling.
Revised: Krichene and Rendle [2020] analyze the behavior of ranking-based metrics under negative sampling.

Original: They point out the sampled metrics can be inconsistent with exact metrics and all metrics lose theirfocus-on-top feature and collapse to a linear one, AUC-ROC, in the small sample limit.
Revised: They point out the sampled metrics can be inconsistent with exact metrics and all metrics lose their focus-on-top feature and collapse to a linear one, AUC-ROC, in the small sample limit.

Original: Sun et al.[2020] focus on the unfair tie-breaking methods.
Revised: Sun et al. [2020] focus on the unfair tie-breaking methods.

Original: Akrami et al.
Revised: Akrami et al.

Original: ﬁnd some data argumentationsuch as adding inverse relations could be a kind of excessive data leakage during evaluation.
Revised: [2020] find some data argumentation such as adding inverse relations could be a kind of excessive data leakage during evaluation.

Original: Open-world problem
Revised: Open-world assumption Some recent works have noticed the gap between the actual openworld situation and the closed-world assumption. Cao et al. [2021] point out that the closed-world assumption leads to a trivial evaluation on the triple classification task. They offer their manuallylabeled positive-negative-unknown ternary triple classification datasets following the open-world assumption and point out the lack of capacity for current models to distinguish unknown from negative .

Revised: However, the unknown part in the dataset is only on the triple classification task, while we focus on the link prediction task here.

Revised: Additionally, Das et al.

Revised: [2020] analyze the open-world setting as an evolving world that continuously adds new entities and facts to KGs.

Revised: Under this setting, their work focuses on the inductive or case learning capacity, i.e., the capacity of models to generalize on unobserved entities.

Revised: Here, we aim to analyze the possible inconsistent comparison in evaluation with missing facts instead of a specific framework with a larger inductive capacity.

Original: In this section, we formally deﬁne the open-world problem that will be analyzed in our paper.
Revised: In this section, we formally define the open-world problem that will be analyzed in our paper.

Original: Deﬁnition 3.1 (Knowledge Graph) .
Revised: Definition 3.1 (Knowledge Graph) .

Original: A knowledge graph is a relational graph G =
Revised: A knowledge graph is a relational graph G =

Original: ( E, F, R )
Revised: ( E, F, R )

Original: where
Revised: where

Original: E is the vertex set containing entities , F is the edge set containing facts , and R is the relation set.
Revised: E is the vertex set containing entities , F is the edge set containing facts , and R is the relation set.

Original: Each edge f ∈ F is labeled by a relation.
Revised: Each edge f ∈ F is labeled by a relation.

Original: If an edge f between entities e h and e t is labeled byrelation r ∈ R , we denote the edge f as r ( e h , e t ) where e h is the head entity and e t is the tail entity.
Revised: If an edge f between entities e h and e t is labeled by relation r ∈ R , we denote the edge f as r ( e h , e t ) where e h is the head entity and e t is the tail entity.

Original: In this paper, we assume E and R are ﬁxed.
Revised: In this paper, we assume E and R are fixed.

Original: Therefore, we sometimes directly use G to denote thefact set F , and r ( e h , e t ) ∈ G means there is relation r between e h and e t in the KG G .
Revised: Therefore, we sometimes directly use G to denote the fact set F , and r ( e h , e t ) ∈ G means there is relation r between e h and e t in the KG G .

Original: A set of KGs with the same entities E and relations R but different facts F is called a world anddenoted by W ( E, R ) .
Revised: A set of KGs with the same entities E and relations R but different facts F is called a world and denoted by W ( E, R ) .

Original: An (open-world) KG can be considered as an observation or understanding ofthe world where there could be unobserved or unknown facts, while the closed-world KG contains allthe true facts of the world.
Revised: An (open-world) KG can be considered as an observation or understanding of the world where there could be unobserved or unknown facts, while the closed-world KG contains all the true facts of the world.

Original: Formally, the closed-world and open-world KGs are deﬁned as follows:
Revised: Formally, the closed-world and open-world KGs are defined as follows:

Original: Deﬁnition 3.2 (Closed-World KG and Open-World KG) .
Revised: Definition 3.2 (Closed-World KG and Open-World KG) .

Original: For a world W ( E, R ) , the closed-world
Revised: For a world W ( E, R ) , the closed-world

Original: KG G is the closure of the world.
Revised: KG G is the closure of the world.

Original: [Equation]
Revised: [Equation]

Original: where the union is deﬁned on the fact set.
Revised: where the union is defined on the fact set.

Original: And for a KG G (cid:48) ∈ W , if G (cid:48) (cid:54) =
Revised: And for a KG G ′ ∈ W , if G ′ ̸

Original:  G , G (cid:48) is open-world.
Revised: = G , G ′ is open-world.

Original: Some property of closed-world KGs: 1) There is a one-to-one correspondence between closed-world
Revised: Some property of closed-world KGs: 1) There is a one-to-one correspondence between closed-world

Original: KGs and worlds W ( E, R ) .
Revised: KGs and worlds W ( E, R ) .

Original: All the KGs in a world are subgraphs of the closed-world one. 3)
Revised: All the KGs in a world are subgraphs of the closed-world one. 3)

Original: If Gis the closed-world KG of the world W , we have f (cid:54)∈ G ⇒ ∀ G (cid:48) ∈ W, f (cid:54)∈ G (cid:48) .
Revised: If G is the closed-world KG of the world W , we have f ̸∈ G ⇒ ∀ G ′ ∈ W, f ̸∈ G ′ .

Original: The third property is critical.
Revised: The third property is critical.

Original: It means with the closed-world G , we know there is no such a relationr between e h and e t in the world when r ( e h , e t ) (cid:54)∈ G .
Revised: It means with the closed-world G , we know there is no such a relation r between e h and e t in the world when r ( e h , e t ) ̸∈ G .

Original: Given the closed-world KG, we have allknowledge of the world, including both the positive and negative one.
Revised: Given the closed-world KG, we have all knowledge of the world, including both the positive and negative one.

Original: Conversely, if a KG is openworld, we do not know whether the triplet is false or unknown when r ( e h , e t ) (cid:54)∈ G .
Revised: Conversely, if a KG is openworld, we do not know whether the triplet is false or unknown when r ( e h , e t ) ̸∈ G .

Original: In other words,an open-world KG only contains positive knowledge.
Revised: In other words, an open-world KG only contains positive knowledge.

Original: In the rest of the paper, we denote the closed-world KG as G full .
Revised: In the rest of the paper, we denote the closed-world KG as G full .

Original: Because we want to study theevaluation, we denote the existing open-world dataset as G test , and extract the training set G trainfrom G test .
Revised: Because we want to study the evaluation, we denote the existing open-world dataset as G test , and extract the training set G train from G test .

Original: Here, G train ⊆
Revised: Here, G train ⊆

Original: G test ⊆
Revised: G test ⊆

Original: G full and the facts in G train , G test \ G train , G full \ G testare training facts , test facts and missing facts respectively.
Revised: G full and the facts in G train , G test \ G train , G full \ G test are training facts , test facts and missing facts respectively.

Original: In addition, we also call the facts in
Revised: In addition, we also call the facts in

Original: G full \ G train full test facts and G test \ G train sparse test facts .
Revised: G full \ G train full test facts and G test \ G train sparse test facts .

Original: Now, we can formally deﬁne the open-world problem .
Revised: Now, we can formally define the open-world problem .

Original: We believe the actual strength of a modelshould be evaluated on the full test facts G full \ G train .
Revised: We believe the actual strength of a model should be evaluated on the full test facts G full \ G train .

Original: However, because the closed-world KG
Revised: However, because the closed-world KG

Original: G full is unavailable, the evaluation is often performed over G test \ G train .
Revised: G full is unavailable, the evaluation is often performed over G test \ G train .

Original: The question is:
Revised: The question is:

Original: Whether the conclusions from evaluation on the sparse test facts G test \ G trainlead to consistent conclusions from evaluation on the full test facts G full \ G train .
Revised: Whether the conclusions from evaluation on the sparse test facts G test \ G train lead to consistent conclusions from evaluation on the full test facts G full \ G train .

Original:  Theoretical analysis on metric degradation and inconsistency
Revised:  Theoretical analysis on metric degradation and inconsistency

Original: To study the open-world problem, we theoretically analyze the behavior of ranking-based metricswith missing facts.
Revised: To study the open-world problem, we theoretically analyze the behavior of ranking-based metrics with missing facts.

Original: All the proofs are in Appendix A.1.
Revised: All the proofs are in Appendix A.1.

Original: The randomness comes from two sources:the missing of facts and the predictions of the model.
Revised: The randomness comes from two sources: the missing of facts and the predictions of the model.

Original: We model them as two random events.
Revised: We model them as two random events.

Original: • Missing Fact Model: For a full test fact r ( e h ,
Revised: • Missing Fact Model: For a full test fact r ( e h ,

Original: e t ) ∈ G full \ G train , X means it is a missingfact with P ( X )
Revised: e t ) ∈ G full \ G train , X means it is a missing fact with P ( X )

Original: =
Revised: =

Original: β while X means it is included in the sparse test set G test \ G train withP ( X ) = 1 − β =
Revised: β while X means it is included in the sparse test set G test \ G train with P ( X ) = 1 − β =

Original: α .
Revised: α .

Original: β is called the sparsity of the KG.
Revised: β is called the sparsity of the KG.

Original: • Prediction Model: For simplicity of analysis, we model KGC as a classiﬁcation task.
Revised: • Prediction Model: For simplicity of analysis, we model KGC as a classification task.

Original: In fact,an ideal (oracle) KGC model is exactly a classiﬁcation model, which identiﬁes all the correct facts.
Revised: In fact, an ideal (oracle) KGC model is exactly a classification model, which identifies all the correct facts.

Original: Here, for a full test fact r ( e h , e t ) ∈ G full \ G train , Y means the answer e t is correctly classiﬁed as positive with P ( Y ) =
Revised: Here, for a full test fact r ( e h , e t ) ∈ G full \ G train , Y means the answer e t is correctly classified as positive with P ( Y ) =

Original: l . l is called the strength of a model.
Revised: ℓ . ℓ is called the strength of a model.

Original: We break ties uniformly at random for entities classiﬁed into the same class.
Revised: We break ties uniformly at random for entities classified into the same class.

Revised: Note that one of our basic assumptions is ‘set answer’ followed [Ren et al., 2020], where we believe the answer for a query should be a set or concept instead of exactly one entity.

Revised: For the open-world problems we care about, we assume that the number of elements in the answer set should be larger than one.

Original: 4.1 Expectation degradation
Revised: 4.1 Expectation degradation

Original: We ﬁrst assume the independence of the random events X and Y .
Revised: We first assume the independence of the random events X and Y .

Original: We show that the expectationof the metrics will degrade with missing facts.
Revised: We show that the expectation of the metrics will degrade with missing facts.

Original: Speciﬁcally, the increasing of the metrics shows alogarithmic trend, so that it could be too ﬂat to reﬂect the true increasing of the model strength.
Revised: Specifically, the increasing of the metrics shows a logarithmic trend, so that it could be too flat to reflect the true increasing of the model strength.

Original: Assume the number of entities is N entity in the KG.
Revised: Assume the number of entities is N entity in the KG.

Original: For a given query r ( e h , ?) , let N be the numberof full test answers.
Revised: For a given query r ( e h , ?) , let N be the number of full test answers.

Original: The random variant m is the number of missing answers G full \ G test , and itfollows the binomial distribution B ( N, β ) .
Revised: The random variant m is the number of missing answers G full \ G test , and it follows the binomial distribution B ( N, β ) .

Original: The other N − m answers are test answers.
Revised: The other N − m answers are test answers.

Original: We denotethe ﬁltered ranking of the entity e as r ( e ) .
Revised: We denote the filtered ranking of the entity e as r ( e ) .

Original: Then we have the lemma.
Revised: Then we have the lemma.

Original: Lemma 4.1 (Expectation of ranking-based metrics) .
Revised: Lemma 4.1 (Expectation of ranking-based metrics) .

Original: With the modeling of missing fact and predictionas above, the expectation of ranking-based metric M = 1 N − m (cid:80) N − m i =
Revised: With the modeling of missing fact and prediction as above, the expectation of ranking-based metric M = [Equation] N − m [Equation]

Revised: i

Revised: [Equation]

Original: 1 f
Revised: 1 f

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: e i )) can be expressed as
Revised: e i )) can be expressed as

Original: [Equation]
Revised: [Equation]

Original: where ˆΦ is the cumulative distribution function (cdf) of binomial distribution B ( N + 1 , lβ ) and0 < δ ≤ (1 − l ) ln( N entity − N ) N entity − N .
Revised: where ˆΦ is the cumulative distribution function (cdf) of binomial distribution B ( N + 1 , lβ ) and 0 < δ ≤ (1 − l ) ln ( N entity − N ) N entity − N .

Original: Generally, the N entity isa largenumber and the answer rate N/N entity < 10% in almost all queries,so the item δ is negligible.
Revised: Generally, the N entity is large and the answer rate N/N entity < 10% in almost all queries, so the item δ is negligible.

Original: In the rest of the paper, we denote ˆ E =
Revised: In the rest of the paper, we denote ˆ E =

Original: [Equation]
Revised: [Equation]

Original: which is a good approximation of E .
Revised: which is a good approximation of E .

Original: With this lemma, we get a closed-form expression of the expectation of the ranking-based metric M .
Revised: With this lemma, we get a closed-form expression of the expectation of the ranking-based metric M .

Original: However, this expression cannot explain why the metrics will degrade.
Revised: However, this expression cannot explain why the metrics will degrade.

Original: Next, we derive the form ofderivative of the expectation ˆ E w.r.t the model strength l to account for its degradation.
Revised: Next, we derive the form of derivative of the expectation ˆ E w.r.t the model strength l to account for its degradation.

Original: Corollary 4.1 (Derivative of Expectation w.r.t Model Strength) .
Revised: Corollary 4.1 (Derivative of Expectation w.r.t Model Strength) .

Original: Let g ( r ) =
Revised: Let g ( r ) =

Original: r /f ( r ) , r ∈ N + andg (0)
Revised: r /f ( r ) , r ∈ N + and g (0)

Original: = 0 .
Revised: = 0 .

Original: Under the condition of 4.1, the derivative of the expectation w.r.t the model strength l is
Revised: Under the condition of 4.1, the derivative of the expectation w.r.t the model strength ℓ is

Original: [Equation]
Revised: [Equation]

Original: And for the most-used metrics MRR and Hits@K, their derivative are expressed as follows.
Revised: And for the most-used metrics MRR and Hits@K, their derivative are expressed as follows.

Original: Corollary 4.2 (Derivative of MRR w.r.t strength l ) .
Revised: Corollary 4.2 (Derivative of MRR w.r.t strength ℓ ) .

Original: For MRR where f ( r ) =
Revised: For MRR where f ( r ) =

Original: r , ∀ r ∈ N + , itsderivative w.r.t.
Revised: r , ∀ r ∈ N + , its derivative w.r.t.

Original: l is
Revised: ℓ is

Original: [Equation]
Revised: [Equation]

Original: where ε = (1 − lβ ) N +1 .
Revised: where ε = (1 − ℓβ ) N +1 .

Original: When lβ and N are not too small, the term ε is negligible.
Revised: When ℓβ and N are not too small, the term ε is negligible.

Original: In this situation, the derivative of themetric MRR w.r.t the model strength is approximately of O (1 /Nl ) , which will result in insigniﬁcantchanges in the metric with the increasing of the model strength.
Revised: In this situation, the derivative of the metric MRR w.r.t the model strength is approximately of O (1 /Nℓ ) , which will result in insignificant changes in the metric with the increasing of the model strength.

Original: Corollary 4.3 (Derivative of Hits@K w.r.t strength l ) .
Revised: Corollary 4.3 (Derivative of Hits@K w.r.t strength ℓ ) .

Original: For Hits@K where f = 1 for r ≤ k andf = + ∞ otherwise, the derivative is
Revised: For Hits@K where f = 1 for r ≤ k and f = + ∞ otherwise, the derivative is

Original: [Equation]
Revised: [Equation]

Original: where Φ is the cdf of the binomial distribution B ( N, lβ ) .
Revised: where Φ is the cdf of the binomial distribution B ( N, ℓβ ) .

Original: The behavior of Hits@K is similar to MRR when K (cid:28) N .
Revised: The behavior of Hits@K is similar to MRR when K ≪ N .

Original: When lβ is not too small and K is nottoo large, the derivative Φ( K − 1) is so small that the increase could be insigniﬁcant.
Revised: When ℓβ is not too small and K is not too large, the derivative Φ( K − 1) is so small that the increase could be insignificant.

Original: Finally, we further approximate Equation (1) by a more intuitive expression with a tolerable error.
Revised: Finally, we further approximate Equation (1) by a more intuitive expression with a tolerable error.

Original: Theorem 4.1 (Expectation of MRR) .
Revised: Theorem 4.1 (Expectation of MRR) .

Original: For MRR, we can further approximate its expectation by
Revised: For MRR, we can further approximate its expectation by

Original: [Equation]
Revised: [Equation]

Original: where γ ≈ 0 .
Revised: where γ ≈ 0 .

Original: 577 is the Euler’s constant and therelative error e = | ˜ 
Revised: 577 is the Euler’s constant and the error e = | ˜ E − ˆ E | ≤ max {

Original: [Equation]
Revised: [Equation]

Original: Firstly, we explain the rationales of the approximation ˜
Revised: Firstly, we explain the rationales of the approximation ˜

Original: E as follows.
Revised: E as follows.

Original: • N is large enough.
Revised: • N is large enough.

Original: For many KGs, especially those commonsense KGs which are not limited to a certain ﬁeld, the number of answers is often quite large.
Revised: For many KGs, especially those commonsense KGs which are not limited to a certain field, the number of answers is often quite large.

Original: In the experiment conducted by Ren and Leskovec [2020], there are many tests queries with dozens or hundreds of answers.
Revised: In the experiment conducted by Ren and Leskovec [2020], there are many tests queries with dozens or hundreds of answers.

Original: • Sparsity β is not too small.
Revised: • Sparsity β is not too small.

Original: For most real-world KGs, although we do not exactly know their sparsity, we expect many of them could have a rather high sparsity β due to the incompleteness of knowledge extraction and the long-tail distribution of commonsense knowledge.
Revised: For most real-world KGs, although we do not exactly know their sparsity, we expect many of them could have a rather high sparsity β due to the incompleteness of knowledge extraction and the long-tail distribution of commonsense knowledge.

Original: • Model strength l is not too small.
Revised: • Model strength ℓ is not too small.

Original: Here we are more concerned with how to select and evaluate those models that perform well.[Figure]
Revised: Here we are more concerned with how to select and evaluate those models that perform well.

Original: The shadow shows the [ − σ is the numerical std.

Original: Under the above conditions, the relative error e is negligible.
Revised: Under the above conditions, the relative error e is negligible.

Original: To further show that our approximation is reliable,we do numerical simulations as shown in Figure 1.
Revised: To further show that our approximation is reliable, we do numerical simulations as shown in Figure 1.

Original: For1 − α ≥ 0 . 2 and l > 0 .
Revised: For 1 − α ≥ 0 . 2 and ℓ > 0 .

Original: 3 , the analytical and numericalcurves almost overlap, which means the log approximationis accurate.
Revised: 3 , the analytical and numerical curves almost overlap, which means the log approximation is accurate.

Original: At the same time, we point out the log trendis just the reason why the curve becomes ﬂatter and ﬂatter.
Revised: At the same time, we point out the log trend is just the reason why the curve becomes flatter and flatter.

Original:  The details of the simulation is in Appendix A.2.
Revised: [Figure] The shadow shows the [ − 2 σ, 2 σ ] interval, where σ is the numerical std.

Original: Theorem 4.1 illustrates the metric degradation intuitively.
Revised: Theorem 4.1 illustrates the metric degradation intuitively.

Original: There are some conclusions about the MRR under theopen-world assumption: 1)
Revised: There are some conclusions about the MRR under the open-world assumption: 1)

Original: Although the theoretical maximum of MRR is 1 , the expectation of the MRR of a perfectmodel l = 1 is still much lower than 1 and depends on thesparsity of the KG. 2)
Revised: Although the theoretical maximum of MRR is 1 , the expectation of the MRR of a perfect model ℓ = 1 is still much lower than 1 and depends on the sparsity of the KG. 2)

Original: With the sparsity β not very small,the MRR will be a log function of the strength l times theanswer number N which means that as the model gets stronger, the increase of the metric MRR willbe less and less signiﬁcant.
Revised: With the sparsity β not very small, the MRR will be a log function of the strength ℓ times the answer number N which means that as the model gets stronger, the increase of the metric MRR will be less and less significant.

Original: The sparser the KG is, the more severe the degradation problem is.
Revised: The sparser the KG is, the more severe the degradation problem is.

Original: Notethat in the closed-world KG, the curve should be very closed to y = x .
Revised: Note that in the closed-world KG, the curve should be very closed to y = x .

Original: 4.2 Inconsistency due to high variance
Revised: 4.2 Inconsistency due to high variance

Original: In Figure 1, another notable phenomenon is the vibrated curves, which suggests instability of themetric and relatively high variance.
Revised: In Figure 1, another notable phenomenon is the vibrated curves, which suggests instability of the metric and relatively high variance.

Original: This phenomenon combined with the ﬂattening of the expectationcan lead to inconsistency , which means higher MRR might not mean stronger models unless thedifference of the metric is large enough, because the increasing of expectation could be easilyoverwhelmed by the variance.
Revised: This phenomenon combined with the flattening of the expectation can lead to inconsistency , which means higher MRR might not mean stronger models unless the difference of the metric is large enough, because the increasing of expectation could be easily overwhelmed by the variance.

Original: One trivial method to solve the problem is to use more test queries.
Revised: One trivial method to solve the problem is to use more test queries.

Original: Here we show the number ofqueries required to ensure the reliability of conclusions can be very large.
Revised: Here we show the number of queries required to ensure the reliability of conclusions can be very large.

Original: Theorem 4.2 (Consistence with High Probability) .
Revised: Theorem 4.2 (Consistence with High Probability) .

Original: Assuming the number of test queries N q is largeenough ( N q > 50 ), we can approximate the average MRR M = 1 N q (cid:80) N q i =
Revised: Assuming the number of test queries N q is large enough ( N q > 50 ), we can approximate the average MRR M = 1 N q (cid:80) N q i =

Original: MRR
Revised: MRR

Original: ( q i ) to follow anormal distribution.
Revised: ( q i ) to follow a normal distribution.

Original: Given two independent models M 1 and M 2 whose strength is l and l + ∆ lrespectively.
Revised: Given two independent models M 1 and M 2 whose strength is ℓ and ℓ + ∆ ℓ respectively.

Original: The probability of inconsistency between two models can be approximated as follows.
Revised: The probability of inconsistency between two models can be approximated as follows.

Original: [Equation]
Revised: [Equation]

Original: where Ψ is the cdf of the standard normal distribution.
Revised: where Ψ is the cdf of the standard normal distribution.

Original: Note for a given KG, the sparsity β and the answer number N are ﬁxed.
Revised: Note for a given KG, the sparsity β and the answer number N are fixed.

Original: Assuming  ∆ ll (cid:28) 1 , we have 
Revised: Assuming 0 < ∆ ℓℓ ≪ 1 , we have V ( β, ℓ ) ≈

Original: V ( β, l ) ≈ V ( β, l + ∆ l ) := V and ln(1 + ∆ ll ) ≈ ∆ ll .
Revised: V ( β, ℓ + ∆ ℓ ) : = V and ln(1 + ∆ ℓℓ ) ≈ ∆ ℓℓ .

Original: Then we have the following corollary.
Revised: Then we have the following corollary.

Original: Corollary 4.4 (Lower Bound of the Number of Queries) .
Revised: Corollary 4.4 (Lower Bound of the Number of Queries) .

Original: Under the above assumption of ∆ ll (cid:28) 1 ,with the upper bound of inconsistency probability p , the number of test queries required N q has alower-bound as follows.
Revised: Under the above assumption of ∆ ℓℓ ≪ 1 , with the upper bound of inconsistency probability p , the number of test queries required N q has a lower-bound as follows.

Original: [Equation]
Revised: [Equation]

Original: where c ( β, l, N, p ) =
Revised: where c ( β, ℓ, N, p ) =

Original: 2( βl ( N + 1)Ψ − 1 ( p )) 2 V .
Revised: 2( βℓ ( N + 1)Ψ − 1 ( p )) 2 V .

Original: Note that the required number is of the second order
Revised: Note that the required number is of the second order

Original: O ((1 / ∆ l ) 2 ) , which means one should beparticularly careful when comparing two models with close strength.
Revised: O ((1 / ∆ ℓ ) 2 ) , which means one should be particularly careful when comparing two models with close strength.

Original: For example, when we setβ = 0 . 35 , l = 0 . 7 , N = 43 and p = 5% we have c ≈ 2 . 85 where we use the numerical variance
Revised: For example, when we set β = 0 . 35 , ℓ = 0 . 7 , N = 43 and p = 5% we have c ≈ 2 . 85 where we use the numerical variance

Original: V = 7 . 4 × 10 − 3 .
Revised: V = 7 . 4 × 10 − 3 .

Original: In this situation, when ∆ l = 0 . 05 , N q ≥ 1140 , while when ∆ l = 0 .
Revised: In this situation, when ∆ ℓ = 0 . 05 , N q ≥ 1140 , while when ∆ ℓ = 0 .

Original: N q ≥ 28500 which cannot be easily satisﬁed.
Revised: N q ≥ 28500 which cannot be easily satisfied.

Original: 4.3 Correlation between missing and misclassiﬁcation
Revised: 4.3 Correlation between missing and misclassification

Original: In the previous two subsections, we analyze the degradation and inconsistency with independenceassumption between missing facts and model predictions.
Revised: In the previous two subsections, we analyze the degradation and inconsistency with independence assumption between missing facts and model predictions.

Original: In some conditions, there could becorrelation between the missing data and the trained models.
Revised: In some conditions, there could be correlation between the missing data and the trained models.

Original: Let us give some examples:
Revised: Let us give some examples:

Original: • The missing facts in closed-world KG G full follow some non-uniform distribution.
Revised: • The missing facts in closed-world KG G full follow some non-uniform distribution.

Original: For example, in some KGs, the missing facts are more frequently related to some certain entities.
Revised: For example, in some KGs, the missing facts are more frequently related to some certain entities.

Original: The modelcould be under-trained on these entities because there are more missing facts related to themwhen training.
Revised: The model could be under-trained on these entities because there are more missing facts related to them when training.

Original: And when the model is tested, the queries with more missing test answerscorrespond to the lower predicting capacity.
Revised: And when the model is tested, the queries with more missing test answers correspond to the lower predicting capacity.

Original:  • The target KG has been preliminarily complemented by some models.
Revised: [Figure] • The target KG has been preliminarily complemented by some models.

Original: In this situation, the missing facts show a negative correlation with the predictive power of this type ofmodels.
Revised: In this situation, the missing facts show a negative correlation with the predictive power of this type of

Original: Next, we will extend the previous analysis to the situation without the independence assumption.
Revised: Next, we will extend the previous analysis to the situation without the independence assumption.

Original: Forthis goal, we use the correlation efﬁcient to model the correlation between random event X : a fact ismissing and Y : this fact is predicted by the model.
Revised: For this goal, we use the correlation efficient to model the correlation between random event X : a fact is missing and Y : this fact is predicted by the model.

Original: Deﬁnition 4.1 (Correlation between fact missing and model prediction) .
Revised: Definition 4.1 (Correlation between fact missing and model prediction) .

Original: The correlation coefﬁcientr between two random events X and Y can be deﬁned as follows.
Revised: The correlation coefficient r between two random events X and Y can be defined as follows.

Original: [Equation]
Revised: [Equation]

Original: With the correlation coefﬁcient ρ , the prediction accuracy on missing answers e ∈ G full \ G test andon test answers e ∈ G test \ G train can be calculated as the conditional probability as P ( Y | X ) 
Revised: With the correlation coefficient ρ , the prediction accuracy on missing answers e ∈ G full \ G test and on test answers e ∈ G test \ G train can be calculated as the conditional probability as P ( Y | X ) =

Original: = l + (cid:112) l (1 − l ) α/β · ρ and P ( Y | X ) =
Revised: ℓ + (cid:112) ℓ (1 − ℓ ) α/β · ρ and P ( Y | X ) =

Original: l − (cid:112) l (1 − l ) β/α · ρ .
Revised: ℓ − (cid:112) ℓ (1 − ℓ ) β/α · ρ .

Original: We denote them as l 1 and l 2 respectively.
Revised: We denote them as ℓ 1 and ℓ 2 respectively.

Original: Theorem 4.1 can then be generalized as follows:
Revised: Theorem 4.1 can then be generalized as follows:

Original: Theorem 4.3.
Revised: Theorem 4.3.

Original: We have the approximation for MRR
Revised: We have the approximation for MRR

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: The theoretical error analysis is in Appendix A.1.
Revised: The theoretical error analysis is in Appendix A.1.

Original: We also evaluate the approximation by numericalsimulation and the results are shown in Figure 2.
Revised: We also evaluate the approximation by numerical simulation and the results are shown in Figure 2.

Original: The approximation ﬁts the numerical simulationwell.
Revised: The approximation fits the numerical simulation well.

Original: Comparing different models with the same model strength l but different correlation coefﬁcientρ , the models with smaller ρ have the higher MRR.
Revised: Comparing different models with the same model strength ℓ but different correlation coefficient ρ , the models with smaller ρ have the higher MRR.

Original: The phenomenon is consistent with Corollary 4.5.
Revised: The phenomenon is consistent with Corollary 4.5.

Original: Corollary 4.5 (The derivative w.r.t ρ with correlation) .
Revised: Corollary 4.5 (The derivative w.r.t ρ with correlation) .

Original: If we have the inequality l 1 β ( N + 2) ≥exp( α + (cid:113) αβ (1 − l ) l − γ ) , then the derivative ∂  E (MRR) ∂ρ < 0 .
Revised: If we have the inequality ℓ 1 β ( N + 2) ≥ exp( α + (cid:113) αβ (1 − ℓ ) ℓ − γ ) , then the derivative ∂ ˜ E (MRR) ∂ρ < 0 .

Original: The condition in Corollary 4.5 has requirements for lower bound l 1 and ρ .
Revised: The condition in Corollary 4.5 has requirements for lower bound ℓ 1 and ρ .

Original: If l 1 is close to 0, thecondition can be violated.
Revised: If ℓ 1 is close to 0, the condition can be violated.

Original: This corollary suggests more severe inconsistency.
Revised: This corollary suggests more severe inconsistency.

Original: In a reasonable range,the expectation of MRR is monotonically decreasing w.r.t ρ .
Revised: In a reasonable range, the expectation of MRR is monotonically decreasing w.r.t ρ .

Original: This conclusion suggests that the metric
Revised: This conclusion suggests that the metric

Original: MRR may favor the models with smaller ρ instead of larger l .
Revised: MRR may favor the models with smaller ρ instead of larger ℓ .

Original: Note that the inconsistency is ofexpectation, which cannot be solved by more test queries.
Revised: Note that the inconsistency is of expectation, which cannot be solved by more test queries.

Original:  Relationship between focus-on-top and degradation
Revised:  Relationship between focus-on-top and degradation

Original: We have pointed out the degradation and inconsistency under the open-world assumption for somemost-used ranking-based metrics.
Revised: We have pointed out the degradation and inconsistency under the open-world assumption for some most-used ranking-based metrics.

Original: Speciﬁcally, the derivative of the metrics w.r.t l can be too smallto reﬂect the increasing of the true improvement of the model strength (degradation).
Revised: Specifically, the derivative of the metrics w.r.t ℓ can be too small to reflect the increasing of the true improvement of the model strength (degradation).

Original: Accordingto Corollary 4.1, the derivative is related to the expectation of g ( r ) =
Revised: According to Corollary 4.1, the derivative is related to the expectation of g ( r ) =

Original: r /f (( r )) where r follows abinomial distribution.
Revised: r /f ( r ) where r follows a binomial distribution.

Original: We point out the degradation is due to the too small expectation of g relative tothe denominator N , which is inherently caused by a property of the metrics called focus-on-top .
Revised: We point out the degradation is due to the too small expectation of g relative to the denominator N , which is inherently caused by a property of the metrics called focus-on-top .

Original: The focus-on-top property means that the metrics are more sensitive to ranking change in top places.
Revised: The focus-on-top property means that the metrics are more sensitive to ranking change in top places.

Original: For example, MRR changes from 1 to 0.5 when the ranking changes from 1 to 2, but only changesfrom 1e-2 to 0.99e-2 when the ranking changes from 100 to 101.
Revised: For example, MRR changes from 1 to 0.5 when the ranking changes from 1 to 2, but only changes from 1e-2 to 0.99e-2 when the ranking changes from 100 to 101.

Original: This property can simulate thehuman behavior that people pay more attention to the top answers.
Revised: This property can simulate the human behavior that people pay more attention to the top answers.

Original: However, under the open-worldassumption, the focus-on-top property causes negative impacts by making the function 1 /f ( r )decrease too fast so that the expectation of g is too small relative to N .
Revised: However, under the open-world assumption, the focus-on-top property causes negative impacts by making the function 1 /f ( r ) decrease too fast so that the expectation of g is too small relative to N .

Original: For example, according to
Revised: For example, according to

Original: Corollary 4.3, a smaller K means more focus-on-top and smaller derivative.
Revised: Corollary 4.3, a smaller K means more focus-on-top and smaller derivative.

Original: We can also understand the relationship intuitively.
Revised: We can also understand the relationship intuitively.

Original: Focusing-on-top means that a few missinganswers can have a large impact on the metric, especially when the model performance is alreadygood and the rankings of the rest answers fall into the sensitive range.
Revised: Focusing-on-top means that a few missing answers can have a large impact on the metric, especially when the model performance is already good and the rankings of the rest answers fall into the sensitive range.

Original: It is also consistent with ourobservation that the ﬂatting problem is more severe when the strength l increases.
Revised: It is also consistent with our observation that the flatting problem is more severe when the strength ℓ increases.

Original: There is a trade-off between focus-on-top and consistency.
Revised: There is a trade-off between focus-on-top and consistency.

Original: Therefore, one solution to the degradationand inconsistency is to add in some less focus-on-top metrics as a veriﬁcation when evaluating, whichhave a relatively slower descending rate.
Revised: Therefore, one solution to the degradation and inconsistency is to add in some less focus-on-top metrics as a verification when evaluating, which have a relatively slower descending rate.

Original: For example, the log -MRR where f ( r ) = log 2 ( r + 1) andp -MRR where f ( r ) =
Revised: For example, the log -MRR where f ( r ) = log 2 ( r + 1) and p -MRR where f ( r ) =

Original: r p , 0 < p < 1 are less focus-on-top than the standard MRR.
Revised: r p , 0 < p < 1 are less focus-on-top than the standard MRR.

Original: If the conclusionsof these less focus-on-top metrics are consistent with the MRR or Hits@K, the credibility of theconclusions will be greatly enhanced.
Revised: If the conclusions of these less focus-on-top metrics are consistent with the MRR or Hits@K, the credibility of the conclusions will be greatly enhanced.

Original:  Experiments on an artiﬁcial KG
Revised:  Experiments on an artificial KG

Original: In this section, we aim to conduct experiments with practical KGC models on a meaningful closedworld KG to further verify our conclusions.The reason we want a closed-world KG is for comparing [Figure]
Revised: In this section, we aim to conduct experiments with practical KGC models on a meaningful closedworld KG to further verify our conclusions.

Original: 0.8 1.0 MRR (full test set) 0.00 0.05 0.10 0.15 0.20 0.25 0.30 M RR MRR (density 65%)

Original: [Figure] [Figure] [Figure] the reported MRR with the true model strength which should be measured on the full test set.
Revised: The reason we want a closed-world KG is for comparing the reported MRR with the true model strength which should be measured on the full test set.

Original: To ﬁnda closed-world KG, however, it is impractical to resort to existing real-world ones since we have noguarantee that the KG has no missing facts.
Revised: To find a closed-world KG, however, it is impractical to resort to existing real-world ones since we have no guarantee that the KG has no missing facts.

Original: Therefore, we must resort to some artiﬁcial KGs.
Revised: Therefore, we must resort to some artificial KGs.

Original: For this purpose, we generate an artiﬁcial family tree KG , which contains 6,004 entities, 23 relations,and 192,532 facts.
Revised: For this purpose, we generate an artificial family tree KG , which contains 6,004 entities, 23 relations, and 192,532 facts.

Original: The relation set contains all common family relations such as parent, child,husband, wife, sister, and brother.
Revised: The relation set contains all common family relations such as parent, child, husband, wife, sister, and brother.

Original: The details are included in Appendix A.3.
Revised: The details are included in Appendix A.3.

Original: The generated KG isclosed-world since all the facts can be deduced by a symbolic reasoning tool called DLV system[Leone et al., 2006] (free for academic use).
Revised: The generated KG is closed-world since all the facts can be deduced by a symbolic reasoning tool called DLV system [Leone et al., 2006] (free for academic use).

Original: With the closed-world KG, we can simulate the practicalopen-world setting by artiﬁcially controlling the degree of random fact missing, which is measuredby density d = | G test | / | G full | .
Revised: With the closed-world KG, we can simulate the practical open-world setting by artificially controlling the degree of random fact missing, which is measured by density d = | G test | / | G full | .

Original: The relation between d and α is explained in Appendix A.3.
Revised: The relation between d and α is explained in Appendix A.3.

Revised: The generated KG may be representative of a class of real-world KGs with rich rules and simple relations.

Revised: However, we admit that for Wiki-KGs such as Freebase[Bollacker et al., 2008], there may be a certain interval, but the closed-form solution KG corresponding to the latter is impossible to obtain.

Original: With the closed-world KG, we aim to verify our previous conclusions restated below:
Revised: With the closed-world KG, we aim to verify our previous conclusions restated below:

Original: There is metric degradation which means the curves of metric increasing become ﬂatterand ﬂatter with the increasing of model strength l .
Revised: There is metric degradation which means the curves of metric increasing become flatter and flatter with the increasing of model strength ℓ .

Original: Further, the degradation may result ininconsistency, where stronger models report lower metric numbers.
Revised: Further, the degradation may result in inconsistency, where stronger models report lower metric numbers.

Original: Considering the correlation between fact missing and model prediction, if the correlationdegrees vary among different models, the inconsistency problem may become more severe.
Revised: Considering the correlation between fact missing and model prediction, if the correlation degrees vary among different models, the inconsistency problem may become more severe.

Original: The degrees of degradation and inconsistency are related to the focus-on-top property of themetrics.
Revised: The degrees of degradation and inconsistency are related to the focus-on-top property of the metrics.

Original: With less focus-on-top metrics, these problems could be relieved.
Revised: With less focus-on-top metrics, these problems could be relieved.

Original: All codes are provided in the supplement and will be released after publication.
Revised: Our code and data are available at https://github.com/GraphPKU/Open- World

Original:  The experimentswere run on two clusters with four NVIDIA A40 and six NVIDIA GeForce 3090 GPUs respectively.
Revised: KG. The experiments were run on two clusters with four NVIDIA A40 and six NVIDIA GeForce 3090 GPUs respectively.

Original: 6.1 Degradation and inconsistency under independence assumption
Revised: 6.1 Degradation and inconsistency under independence assumption

Original: We train four KGC models with different hyperparameter settings (which results in 18 different modelsin total) and test them on full test set G full \ G train and sparse test set G test \ G train respectively.
Revised: We train four KGC models with different hyperparameter settings (which results in 18 different models in total) and test them on full test set G full \ G train and sparse test set G test \ G train respectively.

Original: The full test metric can be considered as a measurement of model strength l which is what wereally want to measure, while the sparse test metric is what we can observe in practice.
Revised: The full test metric can be considered as a measurement of model strength ℓ which is what we really want to measure, while the sparse test metric is what we can observe in practice.

Original: We plotthe sparse-full test curve under different densities in Figure 3, where each curve represents a modelwhose label is shown in the right legend, and the details of the models are given in Appendix A.4.
Revised: We plot the sparse-full test curve under different densities in Figure 3, where each curve represents a model [Figure] 0.8 1.0 MRR (full test set) 0.00 0.05 0.10 0.15 0.20 0.25 0.30 M RR MRR (density 65%) [Figure]

Revised: [Figure]

Revised: The of model 4 lying on the red segment all have better full test MRR than the checkpoint M of model 7, while reporting worse sparse test MRR under the open-world setting.

Original: From the ﬁgure, we ﬁrst observe that these curves are indeed shaped like log curves.
Revised: From the figure, we first observe that these curves are indeed shaped like log curves.

Original: The increasingof the sparse test MRR is slower and slower with the increasing of the full test MRR.
Revised: The increasing of the sparse test MRR is slower and slower with the increasing of the full test MRR.

Original: Due to theﬂatting of the curves, the same sparse MRR has a rather broad interval of the corresponding fullmetric.
Revised: Due to the flatting of the curves, the same sparse MRR has a rather broad interval of the corresponding full metric.

Original: This phenomenon indicates the degradation of the metric MRR.
Revised: This phenomenon indicates the degradation of the metric MRR.

Original: And as the sparsity increases,the range of the y-axis shrinks (i.e., the curves become ﬂatter), which means the degradation is moresevere.
Revised: And as the sparsity increases, the range of the y-axis shrinks (i.e., the curves become flatter), which means the degradation is more severe.

Original: Further, these results demonstrate the inconsistency problem of MRR.
Revised: Further, these results demonstrate the inconsistency problem of MRR.

Original: To illustrate this pointmore clearly, we zoom in a part of the full ﬁgure with two curves as shown in Figure 4.
Revised: To illustrate this point more clearly, we zoom in a part of the full figure with two curves as shown in Figure 4.

Original: For the modelcheckpoint corresponding to point M on curve 7, any model checkpoint corresponding to a point onthe red segment of curve 4 is actually stronger than that model , but reports a lower sparse MRR .[Figure]
Revised: For the model checkpoint corresponding to point M on curve 7, any model checkpoint corresponding to a point on the red segment of curve 4 is actually stronger than that model , but reports a lower sparse MRR .

Original: In this part [Figure] we will simulate the third example we provided in Section 4.3 to check our theoryconsidering the correlation between fact missing and model prediction.
Revised: In this part, we will simulate the third example we provided in Section 4.3 to check our theory considering the correlation between fact missing and model prediction.

Original: Here we use one of the trained
Revised: Here we use one of the trained

Original: ComplEx model (labeled as 16)
Revised: ComplEx model (labeled as 16)

Original: [Trouillon et al., 2016] to predict on the full test set G full \ G trainand use its predictions to choose the test set G test \ G train and missing facts G full \ G test .
Revised: [Trouillon et al., 2016] to predict on the full test set G full \ G train and use its predictions to choose the test set G test \ G train and missing facts G full \ G test .

Original: Themissing facts are highly correlated with this ComplEx model and therefore could be correlated withother models according to the correlation between different frameworks and model settings.
Revised: The missing facts are highly correlated with this ComplEx model and therefore could be correlated with other models according to the correlation between different frameworks and model settings.

Original: Then wetest the other models except for this ComplEx model on the correlated test set.
Revised: Then we test the other models except for this ComplEx model on the correlated test set.

Original: The results of densityd = 75% are shown in Figure 5a and others are shown in Appendix A.5.
Revised: The results of density d = 75% are shown in Figure 5a and others are shown in Appendix A.5.

Original: Though the correlationcoefﬁcient is not available for these different models, we indeed observe the gaps between differentmodels become larger than the independent setting, which suggests the inconsistency is more severe.
Revised: Though the correlation coefficient is not available for these different models, we indeed observe the gaps between different models become larger than the independent setting, which suggests the inconsistency is more severe.

Original: 6.3 Less focus-on-top metrics
Revised: 6.3 Less focus

Revised: -on-top metrics

Original: The next conclusion is that with less focus-on-top metrics, the degradation and inconsistency can berelieved.
Revised: The next conclusion is that with less focus-on-top metrics, the degradation and inconsistency can be relieved.

Original: In Figure 5b, we show the sparse-full curves with some less focus-on-top metrics ( log -MRRand p -MRR) for the experiments from Section 6.1.
Revised: In Figure 5b, we show the sparse-full curves with some less focus-on-top metrics ( log -MRR and p -MRR) for the experiments from Section 6.1.

Original: Their curves are more close to y =
Revised: Their curves are more close to y =

Original: x instead ofthe log function.
Revised: x instead of the log function.

Original: The ﬂatting is less signiﬁcant due to the wider range of y-axis.
Revised: The flatting is less significant due to the wider range of y-axis.

Original: We also observe thatthe gaps between different models become smaller, which indicates inconsistency is also relieved.
Revised: We also observe that the gaps between different models become smaller, which indicates inconsistency is also relieved.

Original: Additional results with more metrics, density d and correlation are shown in Appendix A.6.
Revised: Additional results with more metrics, density d and correlation are shown in Appendix A.6.

Original:  Conclusion and future work
Revised:  Conclusion and future work

Original: In this paper, we study KGC evaluation under the open-world assumption.
Revised: In this paper, we study KGC evaluation under the open-world assumption.

Original: Theoretically, we model
Revised: Theoretically, we model

Original: KGC as a positive-negative classiﬁcation and then deduce an approximation of the expectation of theranking-based metrics with or without the independence assumption.
Revised: KGC as a positive-negative classification and then deduce an approximation of the expectation of the ranking-based metrics with or without the independence assumption.

Original: According to the approximation,we illustrate the degradation and inconsistency of these metrics under the open-world assumption.
Revised: According to the approximation, we illustrate the degradation and inconsistency of these metrics under the open-world assumption.

Original: Furthermore, we point out the focus-on-top property of ranking-based metrics worsen the degradationand inconsistency.
Revised: Furthermore, we point out the focus-on-top property of ranking-based metrics worsen the degradation

Original: Finally, we generate a closed-world family tree KG and do experiments to verifyour theoretical conclusions.
Revised: [Figure]

Original: There is still some future work.
Revised: There is still some future work.

Original: First, our analysis is based on thepositive-negative classiﬁcation model, which may be too idealistic.
Revised: First, our analysis is based on the positive-negative classification model, which may be too idealistic.

Original: In practice, the ranking in positiveand negative parts may not be uniformly at random.
Revised: In practice, the ranking in positive and negative parts may not be uniformly at random.

Original: A more realistic modeling of the KGC task is adirection for our future research.
Revised: A more realistic modeling of the KGC task is a direction for our future research.

Original: In addition, the correlation between missing facts and predictioncould be more complex than our analysis.
Revised: In addition, the correlation between missing facts and prediction could be more complex than our analysis.

Original: Finally, we are curious about the possibility to ﬁnd a morefundamental solution to the open-world problem, which we leave for future work. 
Revised: Finally, we are curious about the possibility to find a more fundamental solution to the open-world problem, which we leave for future work. Acknowledgments

Revised: Z. Lin was supported by the major key project of PCL (grant no. PCL2021A12), the NSF China (No.s 62276004 and 61731018), and Project 2020BD006 supported by PKU-Baidu Fund.

Revised: M. Zhang is supported by the NSF China (No. 62276003) and CCF-Baidu Open Fund (NO.2021PP15002000).

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] We use some assumptions and approximations, for which we have stated in Section 7.
Revised: [Yes] We use some assumptions and approximations, for which we have stated in Section 7.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: Our work does not involve speciﬁc application scenarios and has no negative social impact.
Revised: Our work does not involve specific application scenarios and has no negative social impact.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] Our work conform to the ethics review guidelines.
Revised: [Yes] Our work conform to the ethics review guidelines.

Original: Our work does notinvolve ethical risks.
Revised: Our work does not involve ethical risks.

Original: All datasets are commonly used and public, expect the ones which we generated ourselves.
Revised: All datasets are commonly used and public, expect the ones which we generated ourselves.

Original: And all people are virtual in the artiﬁcial KG.
Revised: And all people are virtual in the artificial KG.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] The complete proofs of all theoretical results are shown in Appendix.
Revised: [Yes] The complete proofs of all theoretical results are shown in Appendix.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We includethecode in the supplemental material.
Revised: [Yes] We include the code as a URL.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] All the details of our experiments, including data generation andsplits, hyperparameters are speciﬁed in Section 6 and Appendix A.

Revised: [Yes] All the details of our experiments, including data generation and splits, hyperparameters are specified in Section 6 and Appendix A.

Original: and A.4.
Revised: and A.4.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] The simulation and the experiments on artiﬁcial family tree, we both report the results after multiple times.
Revised: [Yes] The simulation and the experiments on artificial family tree, we both report the results after multiple times.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 6. 
Revised: [Yes] See Section 6. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] See Section 6 and A.3.
Revised: [Yes] See Section 6 and A.3.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]We generate an artiﬁcial family tree KG
Revised: [Yes] We generate an artificial family tree KG

Original: and we include it in the supplemental material.
Revised: and we include it in the supplemental material.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re[Table] using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A]
Revised: [N/A]

Original: The data is generated ourselves.
Revised: The data is generated ourselves.

Original: (e) Did you discuss whether the data you are using/curating contains personally identi- ﬁable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: The data does not contain personally identiﬁable information or offensive content.
Revised: The data does not contain personally identifiable information or offensive content.

Original: It is a virtual KG.
Revised: It is a virtual KG.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if 
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original:  applicable? 
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: Proof.

Original: Board (IRB) approvals, if applicable?
Revised: First, according to the linearity of expectation, we have

Original: [N/A]
Revised: [Equation]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: where E ( [Equation] =

Revised: [Equation]

Revised: 1 f ( r ( e N − m )) | m ) , and here { e 1 , e 2 , . . .

Revised: , e N − m } is the test answer set.

Revised: We denote the final item as E ( 1 f ( r ( e )) ) .

Revised: Then, using the conditional expectation, we have

Revised: [Equation]

Revised: For the first item, because the ranking of each positive entity is uniformly at random and note the ranking is filtered, we have P ( r = k | Y )

Revised: =

Revised: 1 / ( m + 1) , ∀ k = 1 , 2 , . . .

Revised: , m + 1 .

Revised: Note m is a random variant following the binomial distribution B ( N, βℓ ) , we have

Revised: [Equation]

Revised: where ˆΦ is the cdf of binomial distribution B ( N +1 , ℓβ ) .

Revised: To prove this lemma, we only need to prove 0 <

Revised: [Equation]

Revised: 1 −

Revised: [Equation]

Revised: ( 1 f

Revised: [Equation]

Revised: e ))

Revised: |

Revised: [Equation]

Revised: ln

Revised: [Equation]

Revised: N entity − N ) N entity − N .

Revised: The left side is obvious, and the right side can be proved as follow.

Revised: Here we use N e to denote the number of entity instead of N entity .

Revised: Condition on m , note that the minimal ranking of negative entities is m + 1 because there have been m missing answers with higher rankings, and the maximal ranking of them is N e − ( N − m ) which is the number of entities except for the filtered ones.

Revised: So we have

Revised: [Equation]

Revised: The first inequality is because 1 /k ≤ ln( k ) − ln

Revised: ( k − 1) .

Revised: This proves the lemma.

Revised: A.1.2 Corollary

Revised: Proof.

Revised: We need the derivative of the cdf of binomial distribution.

Revised: Assuming Φ is the cdf of B ( N, p ) , we have

Revised: [Equation]

Revised: So for ˆΦ( k ) is the cdf of B ( N + 1 , ℓβ ) , we have ∂ ˆΦ( k ) ∂ℓ = − β (cid:18) N +k +(cid:19)

Revised: ( k + 1)( ℓβ ) k (1 − ℓβ ) N − k and dˆ E d ℓ = ℓβ ( N + 1)

Revised: N (cid:88) k

Revised: = k

Revised: +f (

Revised: [Equation]

Revised: The final equation is because g (0) = 0 .

Revised: A.1.3 Corollary 4.

Revised: Proof.

Revised: For MRR, g ( r ) = 1 , ∀ r ∈ N + and g (0)

Revised: = 0 .

Revised: Just replace g into Corollary 4.1, we can get this corollary.

Revised: A.1.4 Corollary 4.

Revised: Proof.

Revised: According to the Corollary 4.1, we have

Revised: [Equation]

Revised: where Φ is the cdf of the binomial distribution B ( N, ℓβ ) .

Revised: A.1.5 Theorem

Revised: Proof.

Revised: Let E ′ =

Revised: β ˆ E

Revised: = 1 N +1 (cid:80) Nk

Original: {+=
+}
Revised: {+=
+}

Revised: 1 k +1 (1 − ˆΦ( k )) and t = ℓβ .

Revised: In the same way in Corollary 4.1, we have

Revised: [Equation]

Revised: [Equation]

Revised: 1 − (1 − t ) N + t ( N + 1) .

Revised: For 0 < t 0 < t < 1 , we have

Revised: [Equation]

Revised: Then we integrate them from t 0 to 1.

Revised: [Equation]

Revised: Because t 0 is arbitrary, we replace t 0 as general ℓβ .

Revised: [Equation]

Revised: Note that

Revised: [Equation]

Original: {+We denote it as E 1 , where ε N +
+}
Revised: {+We denote it as E 1 , where ε N +
+}

Revised: =

Revised: ln

Revised: ( N + 2) + γ − (cid:80)

Revised: N +1 k =1 1 k is the residual of the sum of harmonic series and 0 < ε N +1 ≤ 1 2( N +1) .

Revised: Then, we have

Revised: [Equation]

Revised: For the second inequality, it is equivalent to

Revised: [Equation]

Revised: For the first inequality, we have

Revised: [Equation]

Revised: 1 −

Revised: [Equation]

Revised: N

Revised: [Equation]

Revised: N

Revised: [Equation]

Revised: [Equation]

Revised: N

Revised: [Equation]

Revised: Therefore, we have the error bound:

Revised: [Equation]

Revised: A.1.6 Theorem 4.

Revised: Proof.

Revised: Given all the independence assumption, M ( M 2 ) − M ( M 1 ) follows normal distribution

Revised: N ( ln(1+ ∆ ℓℓ ) β ( N +1) , V ( β,ℓ )

Revised: + V ( β, ∆ ℓ + ℓ ) N q ) .

Revised: So

Revised: [Equation]

Revised: Then M ( M 2 ) ≤ M ( M 1 ) is equivalent to

Revised: [Equation]

Revised: So the probability is as shown in the theorem.

Revised: A.1.7 Corollary 4.

Revised: Proof.

Revised: Just solve N q from the Theorem 4.2.

Revised: A.1.8 Theorem 4.

Revised: Proof.

Revised: We can generalize the Lemma 4.1 as follows.

Revised: Lemma A.1 (Expectation with Correlation) .

Revised: Under the same assumptions as the lemma 4.1 and the correlation coefficient is ρ , the expectation of the metric M :

Revised: [Equation]

Revised: where ˜Φ is the cdf of binomial distribution B ( N + 1 , ℓ 1 β ) and 0 ≤ δ ′ ≤ (1 − ℓ 2 )

Revised: ln( N entity − N ) N entity − N .

Revised: The proof of the lemma is similar to what we have shown in A.1.1.

Revised: Given the lemma, the ˆ E can be similarly expressed as ℓ 2 ℓ 1 ·

Revised: [Equation]

Revised: In the similar way in A.1.5, let E ′ =

Revised: ℓ 1 β ˆ E and t =

Revised: ℓ 1 β we have

Revised: [Equation]

Revised: where E 1 = ℓ 2 (ln( N +2)+ γ − ε N +1 )

Revised: N + =

Revised: ˜ E | ℓ 1 =

Revised: β =1 =

Revised: ˆ E | ℓ 1 = β =1 .

Revised: Also using the same technique, the error bound is

Revised: [Equation]

Revised: and

Revised: [Equation]

Revised: The error bound is that

Revised: [Equation]

Revised: A.1.9 Corollary

Revised: Proof.

Revised: Let β ( N + 1) =

Revised: c and ln β + ln

Revised: ( N + 2) + γ =

Revised: d , we have ∂ ˜ E ∂ℓ

Revised: [Equation]

Revised: ℓ

Revised: [Equation]

Revised: 1 − ln ℓ 1 − d ) cℓ, ∂ ∂ℓ 2 = ln

Revised: ( ℓ 1 ) + d cℓ, and ∂ℓ∂ρ =

Revised: (cid:115) αℓ (1 − ℓ ) β , ∂ℓ∂ρ = − (cid:114) βℓ

Revised: (1 − ℓ ) α .

Revised: So the derivative w.r.t ρ ∂ E ∂ρ = (cid:112) ℓ (1 − ℓ ) cℓ(cid:32) ℓ 2 (1 − ln ℓ 1 − d ) (cid:114) α β − ℓ 1 (ln( ℓ 1 ) + d ) (cid:114) β α (cid:33) =

Revised: (cid:112) ℓ (1 − ℓ ) cℓ(cid:114)

Revised: α β (cid:18) ℓ 2 − (ln( ℓ 1 ) + d ) ℓ α (cid:19)

Revised: Note that α + β = 1 .

Revised: Because of the conditions ℓ 1 β ( N + 2) ≥ exp( α + (cid:113) αβ

Revised: [Equation]

Revised: − γ ) , we have

Revised: [Equation]

Revised: and then

Revised: [Equation]

Revised: Combining this inequality with the derivative expression, we have ∂ E ∂ρ < 0 .

Revised: A.2 Details of the simulation

Revised: In the Figures 1 and 2, we choose N as 43 = 14505 × 30% × 1% , where we assume N entity =as the same as FB15k-237, the total answers accounts for one percent of all entities and the test set accounts for thirty percent of the total answers.

Revised: For each ℓ and α we repeat the simulation withtimes to calculate the average MRR and the standard derivation.

Revised: A.3 Details of artificial family tree KG

Revised: Our codes are modified from [Hohenecker and Lukasiewicz, 2020] (BSD license) to generate the

Revised: KG.

Revised: Firstly, it generates all the parent-child relations and then deduced other relations by a symbolic reasoning systems called DLV system [Leone et al., 2006].

Revised: We generate 20 family trees then merge them into a whole.

Revised: Each family tree has three layer depth and 300 entities, with maximal branching width 20 at each internal node.

Revised: The final artificial KG has 6,004 entities, 23 relations and 192, facts.

Revised: The relations are listed as follow:

Revised: • parentOf • sisterOf • brotherOf • siblingOf • motherOf • fatherOf • wifeOf • husbandOf • grandmotherOf • grandfatherOf • auntOf • uncleOf • girlCousinOf • boyCousinOf • cousinOf • daughterOf • sonOf • childOf • granddaughterOf • grandsonOf • grandchildOf • nieceOf • nephewOf

Revised: Note that G full ⊇ G test ⊇ G train .

Revised: We use density d to denote the ratio | G test | / | G full | and then in the open-world KG G test we split the training set and test set with ratio η = | G train | / | G test | .

Revised: For each facts, it is a missing fact with probability 1 − d , a test fact with probability d (1 − η ) and a training fact with probability dη .

Revised: We set η = 0 .

Revised: 7 and d = 95% , 85% , 75% , 65% which corresponds to α = | G test \ G train | | G full \ G train | =

Revised: d (1 − η ) 1 − dη = 85% , 63% , 47% , 35% .

Revised: As the same as [Ren et al., 2020, Ren and Leskovec, 2020], we organize the test by queries which means we firstly randomly sample the test queries r ( e h , ?) and then search answers e ∈ G full \ G test as missing answers and e ∈ G test \ G train as test answers.

Revised: In order to simulate the real situation of common KGs, we filter out the queries with less than 10 answers in the closed-world graph G full .

Revised: Finally, for each sparsity d we choose 500 test queries.

Revised: For training, we use all facts in G train .

Revised: A.4 Details of models

Revised: Different models are trained on artificial family tree KG.

Revised: During training, we test them on full test set and sparse test set to plot the sparse-full curve to show the inconsistency.

Revised: We choose different framework, including RotatE, pRotatE [Sun et al., 2019], ComplEx [Trouillon et al., 2016] and BetaE [Ren and Leskovec, 2020].

Revised: We also test Q2B

Revised: [Ren et al., 2020] and TransE [Bordes et al., 2013] models, both of which cannot fit the KG well.

Revised: For each framework, we use several setting, where their label in Figure 3 and the hyper-parameters of the models are shown in Table 2.

Revised: Here, We have filtered some models which maximal strength ℓ < 0 . 1 .

Revised: A.5 Experiments with correlation

Revised: Here we show more results of the experiments on the correlated family tree KG in Figure 6.

Revised: [Table]

Revised: [Figure]

Revised: r

Revised: [Figure]

Revised: t

Revised: [Figure]

Revised: t

Revised: [Figure]

Revised: t

Original: [N/A]
Revised: [Figure]

Revised: r s e t e s t s e t )

Revised: [Figure]

Revised: [Figure]

Revised: A.6 Family tree experiments with MRR, Hits@K and more less focus-on-top metrics

Revised: The results for other density d and more metrics in the independent situation are shown in Figures 7-10.

Revised: And the results in the correlated situation are shown in Figures 11-14.

Revised: [Figure]

Revised: [Figure]

Revised: HITS@

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]