Original: Behavioral Testing of Knowledge Graph Embedding Models
Revised: Behavioral Testing of Knowledge Graph Embedding Models

Original: Abstract
Revised: Abstract

Original: Knowledge graph embedding (KGE) models are often used to encode knowledge graphs in order to predict new links inside the graph.
Revised: Knowledge graph embedding (KGE) models are often used to encode knowledge graphs in order to predict new links inside the graph.

Original: The accuracy of these methods is typically evaluated by computing an averaged accuracy metric on a held-out test set.
Revised: The accuracy of these methods is typically evaluated by computing an averaged accuracy metric on a held-out test set.

Original: This approach, however, does not allow the identiﬁcation of where the models might systematically fail or succeed.
Revised: This approach, however, does not allow the identiﬁcation of where the models might systematically fail or succeed.

Original: To address this challenge, we propose a new evaluation framework that builds on the idea of (black-box) behavioral testing, a software engineering principle that enables users to detect system failures before deployment.
Revised: To address this challenge, we propose a new evaluation framework that builds on the idea of (black-box) behavioral testing, a software engineering principle that enables users to detect system failures before deployment.

Original: With behavioral tests, we can speciﬁcally target and evaluate the behavior of KGE models on speciﬁc capabilities deemed important in the context of a particular use case.
Revised: With behavioral tests, we can speciﬁcally target and evaluate the behavior of KGE models on speciﬁc capabilities deemed important in the context of a particular use case.

Original: To this end, we leverage existing knowledge graph schemas to design behavioral tests for the link prediction task.
Revised: To this end, we leverage existing knowledge graph schemas to design behavioral tests for the link prediction task.

Original: With an extensive set of experiments, we perform and analyze these tests for several KGE models.
Revised: With an extensive set of experiments, we perform and analyze these tests for several KGE models.

Original: Crucially, we for example ﬁnd that a model ranked second to last on the original test set actually performs best when tested for a speciﬁc capability.
Revised: Crucially, we for example ﬁnd that a model ranked second to last on the original test set actually performs best when tested for a speciﬁc capability.

Original: Such insights allow users to better choose which KGE model might be most suitable for a particular task.
Revised: Such insights allow users to better choose which KGE model might be most suitable for a particular task.

Original: The framework is extendable to additional behavioral tests and we hope to inspire fellow researchers to join us in collaboratively growing this framework.
Revised: The framework is extendable to additional behavioral tests and we hope to inspire fellow researchers to join us in collaboratively growing this framework.

Original: The framework is available at https: //github.com/nec-research/KGEval .
Revised: The framework is available at https: //github.com/nec-research/KGEval .

Original: 1. Introduction
Revised: 1. Introduction

Original: Knowledge Graphs (KGs) are graph databases that represent information about entities and their relationships in the form of canonical (head, relation, tail)-triples.
Revised: Knowledge Graphs (KGs) are graph databases that represent information about entities and their relationships in the form of canonical (head, relation, tail)-triples.

Original: KGs are used in many downstream tasks such as question answering [Huang et al., 2019], recommender systems [Guo et al., 2020], information extraction [Gashteovski et al., 2020], and named entity linking [Shen et al., 2012].
Revised: KGs are used in many downstream tasks such as question answering [Huang et al., 2019], recommender systems [Guo et al., 2020], information extraction [Gashteovski et al., 2020], and named entity linking [Shen et al., 2012].

Original: A common challenge when working with KGs is that they suﬀer from incompleteness [West et al., 2014].
Revised: A common challenge when working with KGs is that they suﬀer from incompleteness [West et al., 2014].

Original: For example, the popular KG DBpedia [Lehmann et al., 2015] contains information about the entities Joe Biden and United
Revised: For example, the popular KG DBpedia [Lehmann et al., 2015] contains information about the entities Joe Biden and United

Original: States , but it does not contain the fact expressing the presidentOf relationship between the two entities.
Revised: States , but it does not contain the fact expressing the presidentOf relationship between the two entities.

Original: Motivated by the incompleteness of most KGs, there is a large body of work on link prediction in knowledge graphs, that is, on deriving missing triples from the set of existing triples.
Revised: Motivated by the incompleteness of most KGs, there is a large body of work on link prediction in knowledge graphs, that is, on deriving missing triples from the set of existing triples.

Original: KG Embeddings (KGE) have been shown to be eﬀective at predicting missing links [Minervini et al., 2015, Ruﬃnelli et al., 2020].
Revised: KG Embeddings (KGE) have been shown to be eﬀective at predicting missing links [Minervini et al., 2015, Ruﬃnelli et al., 2020].

Original: KGE methods learn low-dimensional representations of entities and relation types in a vector space (see also the more detailed deﬁnition in Section 2.1).
Revised: KGE methods learn low-dimensional representations of entities and relation types in a vector space (see also the more detailed deﬁnition in Section 2.1).

Original: KGE models are typically evaluated by measuring their ability to rank candidate entities averaged over triples in a held out test set.
Revised: KGE models are typically evaluated by measuring their ability to rank candidate entities averaged over triples in a held out test set.

Original: This average-based evaluation, however, poses several problems (Bianchi et al.
Revised: This average-based evaluation, however, poses several problems (Bianchi et al.

Original: [2020], Kadlec et al.
Revised: [2020], Kadlec et al.

Original: [2017], Sun et al.
Revised: [2017], Sun et al.

Original: [2020b], Mohamed et al.
Revised: [2020b], Mohamed et al.

Original: [2020]; for a closer discussion see also Section 2.2) and leaves open the question of what models fail to learn.
Revised: [2020]; for a closer discussion see also Section 2.2) and leaves open the question of what models fail to learn.

Original: As a step towards a better evaluation of KGE models, we propose the use of behavioral tests.
Revised: As a step towards a better evaluation of KGE models, we propose the use of behavioral tests.

Original: Behavioral testing, a standard-practice in software engineering, is concerned with testing behaviors of a (black-box) software system by feeding it various inputs and observing and analyzing the system’s behavior.
Revised: Behavioral testing, a standard-practice in software engineering, is concerned with testing behaviors of a (black-box) software system by feeding it various inputs and observing and analyzing the system’s behavior.

Original: Such behavioral tests have several advantages.
Revised: Such behavioral tests have several advantages.

Original: First, it is possible to ﬁnd out if a model makes systematic mistakes for a certain capability of interest.
Revised: First, it is possible to ﬁnd out if a model makes systematic mistakes for a certain capability of interest.

Original: Second, one can compare diﬀerent models in a more detailed and ﬁne-grained manner, allowing us to understand under which circumstances diﬀerent models oﬀer an advantage.
Revised: Second, one can compare diﬀerent models in a more detailed and ﬁne-grained manner, allowing us to understand under which circumstances diﬀerent models oﬀer an advantage.

Original: Third, the tests could be used to uncover particular issues in the training data, which could then be corrected (e.g. in the context of KGs by adding more entities or relations of a certain type).
Revised: Third, the tests could be used to uncover particular issues in the training data, which could then be corrected (e.g. in the context of KGs by adding more entities or relations of a certain type).

Original: Fourth, by using a more rigorous testing approach, we can increase the trust of stakeholders in production settings.
Revised: Fourth, by using a more rigorous testing approach, we can increase the trust of stakeholders in production settings.

Original: In this work, we kick-start a new evaluation framework by exploring possible testable capabilities and by deﬁning detailed tests for two of these capabilities.
Revised: In this work, we kick-start a new evaluation framework by exploring possible testable capabilities and by deﬁning detailed tests for two of these capabilities.

Original: First, we explore how well KGE models handle symmetric relations (e.g. the relation spouseOf is symmetric), which has been a much discussed property (Sun et al.
Revised: First, we explore how well KGE models handle symmetric relations (e.g. the relation spouseOf is symmetric), which has been a much discussed property (Sun et al.

Original: [2019], Peng and Zhang [2020],
Revised: [2019], Peng and Zhang [2020],

Original: Zhang et al.
Revised: Zhang et al.

Original: [2020], Trouillon et al.
Revised: [2020], Trouillon et al.

Original: [2016], Wang et al.
Revised: [2016], Wang et al.

Original: [2014]; inter alia ) but so far lacks a systematic evaluation.
Revised: [2014]; inter alia ) but so far lacks a systematic evaluation.

Original: Second, entities in a KG are often associated with an entity type (e.g. Diane Sawyer is of type Actor ) and we evaluate how well KGE models have learnt to respect entity types.
Revised: Second, entities in a KG are often associated with an entity type (e.g. Diane Sawyer is of type Actor ) and we evaluate how well KGE models have learnt to respect entity types.

Original: We run these tests on six KGE models and ﬁnd that the model ranking on the original test set does not reﬂect the same ranking when testing for speciﬁc capabilities.
Revised: We run these tests on six KGE models and ﬁnd that the model ranking on the original test set does not reﬂect the same ranking when testing for speciﬁc capabilities.

Original: For instance, we ﬁnd that the model ComplEx [Trouillon et al., 2016] ranks second to last on the original test set but is the best at predicting unseen triples for symmetric relations.
Revised: For instance, we ﬁnd that the model ComplEx [Trouillon et al., 2016] ranks second to last on the original test set but is the best at predicting unseen triples for symmetric relations.

Original: 2. Link Prediction for Knowledge Graphs
Revised: 2. Link Prediction for Knowledge Graphs

Original: A knowledge graph K consists of a set of entities E , a set of relation types R , and a set of triples d
Revised: A knowledge graph K consists of a set of entities E , a set of relation types R , and a set of triples d

Original: = ( h, r, t ), with head h ∈ E , relation type r ∈ R and tail t ∈ E .
Revised: = ( h, r, t ), with head h ∈ E , relation type r ∈ R and tail t ∈ E .

Original: For example, the triple ( Porto , locatedIn , Portugal ) represents the information that the entity Porto is located in the entity Portugal .
Revised: For example, the triple ( Porto , locatedIn , Portugal ) represents the information that the entity Porto is located in the entity Portugal .

Original: Knowledge graphs are typically incomplete.
Revised: Knowledge graphs are typically incomplete.

Original: Link prediction is the task of inferring new triples based on the triples contained in the knowledge graph.
Revised: Link prediction is the task of inferring new triples based on the triples contained in the knowledge graph.

Original: This problem can be framed as a tail and head prediction query of the form ( h, r, ?) and (? , r, t ), where one seeks to ﬁnd substitutions for ?
Revised: This problem can be framed as a tail and head prediction query of the form ( h, r, ?) and (? , r, t ), where one seeks to ﬁnd substitutions for ?

Original: that result in a new correct triple.
Revised: that result in a new correct triple.

Original: 2.1 Knowledge Graph Embedding Models
Revised: 2.1 Knowledge Graph Embedding Models

Original: A Knowledge Graph Embedding (KGE) model consists of three main components.
Revised: A Knowledge Graph Embedding (KGE) model consists of three main components.

Original: First, the KGE model’s parameters w .
Revised: First, the KGE model’s parameters w .

Original: Typically, the parameters are vectors associated with each entity and relation type.
Revised: Typically, the parameters are vectors associated with each entity and relation type.

Original: Second, given the model’s parameters w we have a scoring function φ ( d ; w ) which maps the parameters of the head, tail, and relation type occurring in a triple d to a real-valued number.
Revised: Second, given the model’s parameters w we have a scoring function φ ( d ; w ) which maps the parameters of the head, tail, and relation type occurring in a triple d to a real-valued number.

Original: KGE models diﬀer mainly in the particular choice of scoring functions.
Revised: KGE models diﬀer mainly in the particular choice of scoring functions.

Original: Third, the model’s parameters are learnt based on a training set D train of known triples, where a commonly used loss function aims to maximize the score of known triples while minimizing the score of randomly sampled (much more likely to be incorrect) triples.
Revised: Third, the model’s parameters are learnt based on a training set D train of known triples, where a commonly used loss function aims to maximize the score of known triples while minimizing the score of randomly sampled (much more likely to be incorrect) triples.

Original: Once a model is trained, it can predict a score for an unseen triple that indicates the likelihood of this being true.
Revised: Once a model is trained, it can predict a score for an unseen triple that indicates the likelihood of this being true.

Original: 2.2 Standard Evaluation Metrics and their Shortcomings
Revised: 2.2 Standard Evaluation Metrics and their Shortcomings

Original: The standard approach for evaluating link prediction methods uses a (held-out) test set
Revised: The standard approach for evaluating link prediction methods uses a (held-out) test set

Original: D test of correct triples.
Revised: D test of correct triples.

Original: For each of these ( h, r, t )-triples, the model scores all possible substitutions of the queries ( h, r, ?) and (? , r, t ).
Revised: For each of these ( h, r, t )-triples, the model scores all possible substitutions of the queries ( h, r, ?) and (? , r, t ).

Original: Then, the entities are ranked in descending order according to their predicted scores.
Revised: Then, the entities are ranked in descending order according to their predicted scores.

Original: There are two commonly used metrics for evaluating the resulting rankings—MRR and Hits@k ( t may be replaced with h for the reverse direction):
Revised: There are two commonly used metrics for evaluating the resulting rankings—MRR and Hits@k ( t may be replaced with h for the reverse direction):

Original: [Equation]
Revised: [Equation]

Original: The MRR (mean reciprocal rank; lower is better) metric identiﬁes the gold tail t ’s rank rank ( t ) according to the model’s scores and computes the mean of the reciprocal ranks over all test triples.
Revised: The MRR (mean reciprocal rank; lower is better) metric identiﬁes the gold tail t ’s rank rank ( t ) according to the model’s scores and computes the mean of the reciprocal ranks over all test triples.

Original: The Hits@k metric computes the mean, over all test triples, of the event that the gold tail occurs in the top k ranked entities.
Revised: The Hits@k metric computes the mean, over all test triples, of the event that the gold tail occurs in the top k ranked entities.

Original: Using the above metrics as averages over a test set is a global and coarse-grained measure of accuracy.
Revised: Using the above metrics as averages over a test set is a global and coarse-grained measure of accuracy.

Original: It conﬂates diﬀerent relation and entity types and does not allow the user to understand the types of errors the KGE model makes on particular groups of triples.
Revised: It conﬂates diﬀerent relation and entity types and does not allow the user to understand the types of errors the KGE model makes on particular groups of triples.

Original: For example, the model could perform poorly on speciﬁc relation types, either due to a weakness of the model or misspeciﬁed training data.
Revised: For example, the model could perform poorly on speciﬁc relation types, either due to a weakness of the model or misspeciﬁed training data.

Original: The need for an evaluation approach that can analyze and explore the behavior of KGE methods in a more ﬁne-grained manner was also expressed in prior work [Bianchi et al., 2020].
Revised: The need for an evaluation approach that can analyze and explore the behavior of KGE methods in a more ﬁne-grained manner was also expressed in prior work [Bianchi et al., 2020].

Original: To move towards a better evaluation paradigm for KGE models, we propose a new, extendable evaluation framework which can compare
Revised: To move towards a better evaluation paradigm for KGE models, we propose a new, extendable evaluation framework which can compare

Original: KGE models in a more systematic manner.
Revised: KGE models in a more systematic manner.

Original: In addition to the coarse-grained nature of standard evaluation measures, prior work has identiﬁed additional shortcomings of common evaluation procedures.
Revised: In addition to the coarse-grained nature of standard evaluation measures, prior work has identiﬁed additional shortcomings of common evaluation procedures.

Original: First, Kadlec et al.
Revised: First, Kadlec et al.

Original: [2017] found that hyperparameter tuning can be more important than model architecture changes, which caused the authors to raise doubts about the traditional accuracy based evaluation.
Revised: [2017] found that hyperparameter tuning can be more important than model architecture changes, which caused the authors to raise doubts about the traditional accuracy based evaluation.

Original: By using and extending our framework of more ﬁne-grained behavioral testing, it will be possible to determine if and under which circumstances diﬀerent model architecture perform better.
Revised: By using and extending our framework of more ﬁne-grained behavioral testing, it will be possible to determine if and under which circumstances diﬀerent model architecture perform better.

Original: Second, it has been found that recently published (and seemingly superior)
Revised: Second, it has been found that recently published (and seemingly superior)

Original: KGE models have been evaluated incorrectly [Sun et al., 2020b].
Revised: KGE models have been evaluated incorrectly [Sun et al., 2020b].

Original: By introducing a common test framework, such inconsistency could be avoided.
Revised: By introducing a common test framework, such inconsistency could be avoided.

Original: Third, Mohamed et al.
Revised: Third, Mohamed et al.

Original: [2020] has shown that the traditional accuracy-based metrics overestimate the performance of KGE models because they magnify the accuracy on frequently occurring entities and/or relations.
Revised: [2020] has shown that the traditional accuracy-based metrics overestimate the performance of KGE models because they magnify the accuracy on frequently occurring entities and/or relations.

Original: The less frequent, “long-tail” triples, however, are often the triples that one cares the most about [Ilievski et al., 2020].
Revised: The less frequent, “long-tail” triples, however, are often the triples that one cares the most about [Ilievski et al., 2020].

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Test 1 tests (h, r, ?)
Revised: Test 1 tests (h, r, ?)

Original: For each e 2 ee m . .
Revised: For each e 2 ee m . .

Original: e
Revised: e

Original: Capability
Revised: Capability

Original: Capability n
Revised: Capability n

Original: Detailed Report . . .
Revised: Detailed Report . . .

Original: 3. Behavioral Testing
Revised: 3. Behavioral Testing

Original: Overview.
Revised: Overview.

Original: We propose to use behavioral tests as a new evaluation framework that allows one to better understand what KGE models succeed or fail to learn.
Revised: We propose to use behavioral tests as a new evaluation framework that allows one to better understand what KGE models succeed or fail to learn.

Original: Behavioral testing is a software engineering principle where capabilities of software systems are tested by treating them as black boxes and analyzing their behavior for speciﬁcally designed inputs.
Revised: Behavioral testing is a software engineering principle where capabilities of software systems are tested by treating them as black boxes and analyzing their behavior for speciﬁcally designed inputs.

Original: For example, a software engineer might write a function that expects positive integers inputs.
Revised: For example, a software engineer might write a function that expects positive integers inputs.

Original: One behavioral test for this function could be a test that checks what happens if the function is given a negative integer instead - does the function handle this disallowed case appropriately?
Revised: One behavioral test for this function could be a test that checks what happens if the function is given a negative integer instead - does the function handle this disallowed case appropriately?

Original: The idea of behavioral testing has also recently been applied to NLP [Ribeiro et al., 2020].
Revised: The idea of behavioral testing has also recently been applied to NLP [Ribeiro et al., 2020].

Original: For example, Ribeiro et al.
Revised: For example, Ribeiro et al.

Original: [2020] analyzed the robustness of named entity recognition systems by replacing the city name in speciﬁc sentences with diﬀerent city names, and observing the resulting change in behavior of the system.
Revised: [2020] analyzed the robustness of named entity recognition systems by replacing the city name in speciﬁc sentences with diﬀerent city names, and observing the resulting change in behavior of the system.

Original: Here, we transfer the idea of behavioral testing to analyze the behavior of KGE models in a more ﬁne-grained manner.
Revised: Here, we transfer the idea of behavioral testing to analyze the behavior of KGE models in a more ﬁne-grained manner.

Original: Knowledge graphs are typically associated with a predeﬁned schema [Auer et al., 2007], which imposes constraints on the set of possible triples.
Revised: Knowledge graphs are typically associated with a predeﬁned schema [Auer et al., 2007], which imposes constraints on the set of possible triples.

Original: For instance, a given relation type might have entity type constraints for its domain and range (e.g., the DBpedia relation type birthPlace must have an entity of type person as head and an entity of type location as tail).
Revised: For instance, a given relation type might have entity type constraints for its domain and range (e.g., the DBpedia relation type birthPlace must have an entity of type person as head and an entity of type location as tail).

Original: We propose to use such schemas to design behavioral tests and to check the consistency of KGE models with regards to the schema constraints.
Revised: We propose to use such schemas to design behavioral tests and to check the consistency of KGE models with regards to the schema constraints.

Original: Based on a given KG schema, ﬁne-grained tests can be designed to assess KGE models with respect to their behavior for particular capabilities.
Revised: Based on a given KG schema, ﬁne-grained tests can be designed to assess KGE models with respect to their behavior for particular capabilities.

Original: As a result, this type of evaluation can oﬀer more detailed insights into where KGE models perform well and where they might make systematic mistakes.
Revised: As a result, this type of evaluation can oﬀer more detailed insights into where KGE models perform well and where they might make systematic mistakes.

Original: A graphical overview of our proposed approach is depicted in
Revised: A graphical overview of our proposed approach is depicted in

Original: Figure 1.
Revised: Figure 1.

Original: Possible Capabilities.
Revised: Possible Capabilities.

Original: Our framework can be easily extended by adding new capabilities and corresponding tests.
Revised: Our framework can be easily extended by adding new capabilities and corresponding tests.

Original: Adding various such capabilities and tests to one joint evaluation framework would ensure better comparison between diﬀerent KGE models in the future.
Revised: Adding various such capabilities and tests to one joint evaluation framework would ensure better comparison between diﬀerent KGE models in the future.

Original: Some possible capabilities are:
Revised: Some possible capabilities are:

Original: • Relation symmetry: KGs often contain symmetric relations, where a relation r ∈ R is symmetric if ∀ x, y ∈ E , ( x, r, y )
Revised: • Relation symmetry: KGs often contain symmetric relations, where a relation r ∈ R is symmetric if ∀ x, y ∈ E , ( x, r, y )

Original: =
Revised: =

Original: ⇒ ( y, r, x ).
Revised: ⇒ ( y, r, x ).

Original: For example, if a KGE model was trained on the symmetric relation ( x , spouseOf , y ), a test could check if it correctly predicts ( y , spouseOf , x ).
Revised: For example, if a KGE model was trained on the symmetric relation ( x , spouseOf , y ), a test could check if it correctly predicts ( y , spouseOf , x ).

Original: Ensuring that a KGE model can handle symmetric relations has been the focus of several recently proposed KGE models (Sun et al.
Revised: Ensuring that a KGE model can handle symmetric relations has been the focus of several recently proposed KGE models (Sun et al.

Original: [2019], Peng and Zhang [2020], Zhang et al.
Revised: [2019], Peng and Zhang [2020], Zhang et al.

Original: [2020], Trouillon et al.
Revised: [2020], Trouillon et al.

Original: [2016], Wang et al.
Revised: [2016], Wang et al.

Original: [2014]; inter alia ).
Revised: [2014]; inter alia ).

Original: The idea can also be extended to other properties of relation types, such as antisymmetry, inversion, and composition.
Revised: The idea can also be extended to other properties of relation types, such as antisymmetry, inversion, and composition.

Original: • Entity hierarchy: Entities can be associated with an entity type.
Revised: • Entity hierarchy: Entities can be associated with an entity type.

Original: For example, the FB15k-237 entity Diane Sawyer is of type Actor .
Revised: For example, the FB15k-237 entity Diane Sawyer is of type Actor .

Original: The entity types themselves are organized in a hierarchical (hypernym) taxonomy; e.g. Actor → Artist → Person .
Revised: The entity types themselves are organized in a hierarchical (hypernym) taxonomy; e.g. Actor → Artist → Person .

Original: Various KGE models have speciﬁcally been designed to learn entity hierarchies well [Zhang et al., 2020, Kolyvakis et al., 2020, Balazevic et al., 2019, Chami et al., 2020, Sun et al., 2020a].
Revised: Various KGE models have speciﬁcally been designed to learn entity hierarchies well [Zhang et al., 2020, Kolyvakis et al., 2020, Balazevic et al., 2019, Chami et al., 2020, Sun et al., 2020a].

Original: Tests could for example explore how well KGE models work at diﬀerent hierarchy levels.
Revised: Tests could for example explore how well KGE models work at diﬀerent hierarchy levels.

Original: • Entity distributions: Relations can be grouped into 4 categories with regards to how many correct heads/tails they may have: 1-to-1 , 1-to-many , many-to-1 and many-to-many .
Revised: • Entity distributions: Relations can be grouped into 4 categories with regards to how many correct heads/tails they may have: 1-to-1 , 1-to-many , many-to-1 and many-to-many .

Original: For example, some relations by deﬁnition can only have one correct tail as an answer (e.g. birthPlace ).
Revised: For example, some relations by deﬁnition can only have one correct tail as an answer (e.g. birthPlace ).

Original: Prior work studied how well KGE models perform in such diﬀerent scenarios (e.g.[Bordes et al., 2013a, Peng and Zhang, 2020]) and this type of analysis could easily be added to the evaluation framework.
Revised: Prior work studied how well KGE models perform in such diﬀerent scenarios (e.g.[Bordes et al., 2013a, Peng and Zhang, 2020]) and this type of analysis could easily be added to the evaluation framework.

Original: • Robustness to adversarial attacks: Pezeshkpour et al.
Revised: • Robustness to adversarial attacks: Pezeshkpour et al.

Original: [2019] study how KGE models are eﬀected by adversarial modiﬁcations.
Revised: [2019] study how KGE models are eﬀected by adversarial modiﬁcations.

Original: This idea could be transformed into a capability that checks how prone KGE models are to adversarial attacks.
Revised: This idea could be transformed into a capability that checks how prone KGE models are to adversarial attacks.

Original: • Relation/entity frequency:
Revised: • Relation/entity frequency:

Original: The observation that KGE models perform better on frequently occurring entities/relations [Mohamed et al., 2020] could be systematically tested by creating diﬀerent test sets where this frequency is varied.
Revised: The observation that KGE models perform better on frequently occurring entities/relations [Mohamed et al., 2020] could be systematically tested by creating diﬀerent test sets where this frequency is varied.

Original: The tests deﬁned for a capability explore the behavior of a model under a particular setting or condition.
Revised: The tests deﬁned for a capability explore the behavior of a model under a particular setting or condition.

Original: They can be utilised in two ways.
Revised: They can be utilised in two ways.

Original: First, they can serve as evaluation benchmarks with which diﬀerent systems can be compared to each other.
Revised: First, they can serve as evaluation benchmarks with which diﬀerent systems can be compared to each other.

Original: Based on this, we can choose the best model for a particular capability of interest.
Revised: Based on this, we can choose the best model for a particular capability of interest.

Original: Second, they can be used to determine the failure rate of a system by making a binary decision for each triple, e.g. by deﬁning a cut-oﬀ point for each tested triple.
Revised: Second, they can be used to determine the failure rate of a system by making a binary decision for each triple, e.g. by deﬁning a cut-oﬀ point for each tested triple.

Original: 1 With this view, we can determine if a model is good enough to be deployed.
Revised: 1 With this view, we can determine if a model is good enough to be deployed.

Original: While the latter is ultimately more important in a production setting, we focus here on the former, with which we explore how known KGE models can be compared against each other.
Revised: While the latter is ultimately more important in a production setting, we focus here on the former, with which we explore how known KGE models can be compared against each other.

Original: Evaluation Setup.
Revised: Evaluation Setup.

Original: To kick-start the new evaluation framework, we deﬁne several tests for two capabilities and then use the tests to evaluate diﬀerent KGE models.
Revised: To kick-start the new evaluation framework, we deﬁne several tests for two capabilities and then use the tests to evaluate diﬀerent KGE models.

Original: For the evaluation, we employ six KGE models: DistMult [Yang et al., 2015], ComplEx
Revised: For the evaluation, we employ six KGE models: DistMult [Yang et al., 2015], ComplEx

Original: [Trouillon et al., 2016], RotatE
Revised: [Trouillon et al., 2016], RotatE

Original: [Sun et al., 2019], HyperKG [Kolyvakis et al., 2020], LinearRE
Revised: [Sun et al., 2019], HyperKG [Kolyvakis et al., 2020], LinearRE

Original: [Peng and
Revised: [Peng and

Original: Zhang, 2020], HAKE [Zhang et al., 2020].
Revised: Zhang, 2020], HAKE [Zhang et al., 2020].

Original: For an overview of their scoring functions, see
Revised: For an overview of their scoring functions, see

Original: Table 1 in the appendix.
Revised: Table 1 in the appendix.

Original: The ﬁrst capability we explore is relation symmetry.
Revised: The ﬁrst capability we explore is relation symmetry.

Original: All proposed methods can, in principle, learn that a relation type is symmetric.
Revised: All proposed methods can, in principle, learn that a relation type is symmetric.

Original: DistMult in particular treats all relation types as such.
Revised: DistMult in particular treats all relation types as such.

Original: However, the main evaluation for these models was conducted on the standard test sets with averaged metrics.
Revised: However, the main evaluation for these models was conducted on the standard test sets with averaged metrics.

Original: While ablation studies regarding symmetric capability are sometimes included [Sun et al., 2019, Peng and Zhang, 2020], 2 they are neither systematic nor comparable across papers.
Revised: While ablation studies regarding symmetric capability are sometimes included [Sun et al., 2019, Peng and Zhang, 2020], 2 they are neither systematic nor comparable across papers.

Original: Therefore, it is diﬃcult to judge to what degree they can handle symmetry and which model might be the best with regards to this capability.
Revised: Therefore, it is diﬃcult to judge to what degree they can handle symmetry and which model might be the best with regards to this capability.

Original: To be able to answer this question, we design targeted tests which allow us to measure the performance speciﬁcally for symmetric capability.
Revised: To be able to answer this question, we design targeted tests which allow us to measure the performance speciﬁcally for symmetric capability.

Original: In turn, we can analyse in detail under which settings which model performs best.
Revised: In turn, we can analyse in detail under which settings which model performs best.

Original: The second capability we investigate is entity hierarchy.
Revised: The second capability we investigate is entity hierarchy.

Original: Many KGs are hierarchical by nature; as an example see Figure 3 for entity types in DBpedia.
Revised: Many KGs are hierarchical by nature; as an example see Figure 3 for entity types in DBpedia.

Original: This encodes important additional knowledge; for instance, certain relations only apply to certain entity types.
Revised: This encodes important additional knowledge; for instance, certain relations only apply to certain entity types.

Original: For example, the relation starsIn is typically associated with tail entities of type Actor , which is at level 4 in the hierarchy (see Figure 3).
Revised: For example, the relation starsIn is typically associated with tail entities of type Actor , which is at level 4 in the hierarchy (see Figure 3).

Original: In contrast, the relation birthPlace can have tail entities of the higher ranked level 1 type Place (e.g. City or Town , because both entity types are a type of Place ).
Revised: In contrast, the relation birthPlace can have tail entities of the higher ranked level 1 type Place (e.g. City or Town , because both entity types are a type of Place ).

Original: For this capability, we design tests that explore how well
Revised: For this capability, we design tests that explore how well

Original: KGE models have learnt to handle diﬀerent entity types at varying levels in the entity type hierarchy.
Revised: KGE models have learnt to handle diﬀerent entity types at varying levels in the entity type hierarchy.

Original: For this capability, we can test whether KGE models that have speciﬁcally been designed to learn hierarchical structure ( HyperKG , HAKE ) perform better than other models.
Revised: For this capability, we can test whether KGE models that have speciﬁcally been designed to learn hierarchical structure ( HyperKG , HAKE ) perform better than other models.

Original: The models are trained on FB15k-

Revised: The models are trained on FB15k-

Original: [Toutanova et al., 2015]
Revised: [Toutanova et al., 2015]

Original: and we evaluate the models which achieved highest performance on the original FB15k-273 test set.
Revised: and we evaluate the models which achieved highest performance on the original FB15k-273 test set.

Original: All hyperparameters are included in Section C of the appendix.
Revised: All hyperparameters are included in Section C of the appendix.

Original: 3.1 Capability: Relation Symmetry
Revised: 3.1 Capability: Relation Symmetry

Original: Some relations in a knowledge graph are symmetric: given x, y ∈ E and r ∈ R , if ( x, r, y ) is true, then ( y, r, x ) is also true and vice versa.
Revised: Some relations in a knowledge graph are symmetric: given x, y ∈ E and r ∈ R , if ( x, r, y ) is true, then ( y, r, x ) is also true and vice versa.

Original: We deﬁne four tests to better understand the extent to which KGE models have the ability to handle symmetry.
Revised: We deﬁne four tests to better understand the extent to which KGE models have the ability to handle symmetry.

Original: The ﬁrst three tests target symmetric relations and become progressively more diﬃcult.
Revised: The ﬁrst three tests target symmetric relations and become progressively more diﬃcult.

Original: The last test examines how well models recognize if a relation is not symmetric (or asymmetric).
Revised: The last test examines how well models recognize if a relation is not symmetric (or asymmetric).

Original: To set up these tests, the relations of a dataset need to be split into symmetric and asymmetric relations.
Revised: To set up these tests, the relations of a dataset need to be split into symmetric and asymmetric relations.

Original: For FB15k-237 we report the set of symmetric relations in Table 2 in the appendix.
Revised: For FB15k-237 we report the set of symmetric relations in Table 2 in the appendix.

Original: All other relations of the dataset are asymmetric.
Revised: All other relations of the dataset are asymmetric.

Original: Based on the set of symmetric relations, we ﬁnd 6,220 symmetric triples in the training set and 2,520 triples
Revised: Based on the set of symmetric relations, we ﬁnd 6,220 symmetric triples in the training set and 2,520 triples

Original: [Figure]
Revised: [Figure]

Original: O r i g i n a l
Revised: O r i g i n a l

Original: T e s t 1 :
Revised: T e s t 1 :

Original: M e m o r iz a t i o n
Revised: M e m o r iz a t i o n

Original: T e s t 2 : O n e
Revised: T e s t 2 : O n e

Original: D i r e c t i o n U n s e e n
Revised: D i r e c t i o n U n s e e n

Original: T e s t 3 :
Revised: T e s t 3 :

Original: B o t h
Revised: B o t h

Original: D i r e c t i o n s U n s e e n
Revised: D i r e c t i o n s U n s e e n

Original: T e s t 4 :
Revised: T e s t 4 :

Original: A s y m - m e t r y
Revised: A s y m - m e t r y

Original: [Figure]
Revised: [Figure]

Original: Test 1: Memorization.
Revised: Test 1: Memorization.

Original: The ﬁrst and easiest test measures the extent to which a model memorizes training set triples with a symmetric relation type.
Revised: The ﬁrst and easiest test measures the extent to which a model memorizes training set triples with a symmetric relation type.

Original: For each training triple ( x, r, y ), if r is in the set of symmetric relations, we add ( x, r, y ) to the test set for Testand ask the model to predict the tail entity.
Revised: For each training triple ( x, r, y ), if r is in the set of symmetric relations, we add ( x, r, y ) to the test set for Testand ask the model to predict the tail entity.

Original: This test can be considered an upper bound for the other tests since the data has been seen during training; thus this test constitutes the easiest scenario.
Revised: This test can be considered an upper bound for the other tests since the data has been seen during training; thus this test constitutes the easiest scenario.

Original: Test 2: One Direction Unseen.
Revised: Test 2: One Direction Unseen.

Original: For the second test, we create a test set where a triple with a symmetric relation was seen in one direction during training but the reverse direction was not seen: if ( x, r, y ) is in the training set, but ( y, r, x ) is not, then ( y, r, x ) is added to the test set for Test 2.
Revised: For the second test, we create a test set where a triple with a symmetric relation was seen in one direction during training but the reverse direction was not seen: if ( x, r, y ) is in the training set, but ( y, r, x ) is not, then ( y, r, x ) is added to the test set for Test 2.

Original: As these triples are unseen, the test is harder than the previous one and it will allow us to investigate how well a model has recognized that a relation is symmetric.
Revised: As these triples are unseen, the test is harder than the previous one and it will allow us to investigate how well a model has recognized that a relation is symmetric.

Original: Test 3: Both Directions Unseen.
Revised: Test 3: Both Directions Unseen.

Original: The third test consists of triples that have a symmetric relation but were never seen in either direction during training.
Revised: The third test consists of triples that have a symmetric relation but were never seen in either direction during training.

Original: For this test we collect unseen triples from the validation and test set and if a relation r is symmetric, we add both ( x, r, y ) and ( y, r, x ) to the test set for Test 3.
Revised: For this test we collect unseen triples from the validation and test set and if a relation r is symmetric, we add both ( x, r, y ) and ( y, r, x ) to the test set for Test 3.

Original: Since neither direction was seen by the model, this test is more diﬃcult than the previous two and measures how well symmetric relations generalize to unseen triples.
Revised: Since neither direction was seen by the model, this test is more diﬃcult than the previous two and measures how well symmetric relations generalize to unseen triples.

Original: Test 4: Asymmetry.
Revised: Test 4: Asymmetry.

Original: The last test aims to analyze whether a model mistakenly considers a relation symmetric.
Revised: The last test aims to analyze whether a model mistakenly considers a relation symmetric.

Original: For instance, for the triple (George W. Bush, fatherOf, George
Revised: For instance, for the triple (George W. Bush, fatherOf, George

Original: H. W. Bush) , if the model is given the instance with the head and tail inverted: (George
Revised: H. W. Bush) , if the model is given the instance with the head and tail inverted: (George

Original: H. W. Bush, fatherOf, ? ) , then George W. Bush should not be among the top predictions.
Revised: H. W. Bush, fatherOf, ? ) , then George W. Bush should not be among the top predictions.

Original: This implies that a higher MRR is worse for this test.
Revised: This implies that a higher MRR is worse for this test.

Original: To probe the behavior of
Revised: To probe the behavior of

Original: KGE models with respect such asymmetry, we randomly sample triples from the training set that do not contain a symmetric relation.
Revised: KGE models with respect such asymmetry, we randomly sample triples from the training set that do not contain a symmetric relation.

Original: Results
Revised: Results.

Original: Results on the original test set of FB15k-237 and our behavioral tests are shown in Figure 2 (a).
Revised: Results on the original test set of FB15k-237 and our behavioral tests are shown in Figure 2 (a).

Original: On the original test set LineaRE and HAKE perform best and obtain comparable MRR.
Revised: On the original test set LineaRE and HAKE perform best and obtain comparable MRR.

Original: However, the behavioral tests expose interesting diﬀerences between the two models.
Revised: However, the behavioral tests expose interesting diﬀerences between the two models.

Original: LineaRE performs far better than HAKE on Test 1 (“ Memorization ”) and
Revised: LineaRE performs far better than HAKE on Test 1 (“ Memorization ”) and

Original: Test 2 (“ One Direction Unseen ”).
Revised: Test 2 (“ One Direction Unseen ”).

Original: This indicates that LineaRE is better at memorizing the training set (Test 1) and recognizing symmetric relations (Test 2).
Revised: This indicates that LineaRE is better at memorizing the training set (Test 1) and recognizing symmetric relations (Test 2).

Original: However, on the the test that measures generalization to unseen triples (Test 3, “ Both Direction Unseen ”), we ﬁnd that HAKE outperforms LineaRE .
Revised: However, on the the test that measures generalization to unseen triples (Test 3, “ Both Direction Unseen ”), we ﬁnd that HAKE outperforms LineaRE .

Original: On Test 4 (“ Asymmetry ”) higher MRR indicates a worse model.
Revised: On Test 4 (“ Asymmetry ”) higher MRR indicates a worse model.

Original: We ﬁnd that LineaRE and HAKE are in fact both bad at this test, which together with the relatively low results of Test 3 compared to the original test set could indicate that the models have not suﬃciently learnt to distinguish symmetric relations from asymmetric ones.
Revised: We ﬁnd that LineaRE and HAKE , along with the other models, perform well on this test, indicating that they are not likely to mistake an asymmetric relation for a symmetric one.

Original: Even more surprising are the results of ComplEx .
Revised: Even more surprising are the results of ComplEx .

Original: On the original test set it is ranked second to last (i.e., only DistMult is worse), however, on Test 3 it achieves the best performance.
Revised: On the original test set it is ranked second to last (i.e., only DistMult is worse), however, on Test 3 it achieves the best performance.

Original: It also performs better than expected (based on the original test set) for Testsand 2.
Revised: It also performs better than expected (based on the original test set) for Testsand 2.

Original: This indicates that if generalizing well on symmetric relations is important, ComplEx would be a promising model to try out.
Revised: This indicates that if generalizing well on symmetric relations is important, ComplEx would be a promising model to try out.

Original: Based on these results, additional future investigation into why ComplEx performs better than expected on Test 3 could also lead to new insights on how to achieve better symmetric capability.
Revised: Based on these results, additional future investigation into why ComplEx performs better than expected on Test 3 could also lead to new insights on how to achieve better symmetric capability.

Original: There are two further interesting results.
Revised: There are two further interesting results.

Original: First, DistMult , which treats all relations as symmetric, performs worst.
Revised: First, DistMult , which treats all relations as symmetric, performs worst.

Original: Second, RotatE performs very well on Test 1 but not on the other tests, which indicates that RotatE can memorize well, but does not generalize to more diﬃcult setups.
Revised: Second, RotatE performs very well on Test 1 but not on the other tests, which indicates that RotatE can memorize well, but does not generalize to more diﬃcult setups.

Original: We investigate this further in Figure 2 (b), where we plot the results of DistMult and RotatE and we evaluate the models for diﬀerent dimension sizes.
Revised: We investigate this further in Figure 2 (b), where we plot the results of DistMult and RotatE and we evaluate the models for diﬀerent dimension sizes.

Original: For the ﬁrst three behavioral tests DistMult is better at lower dimension sizes, whereas for
Revised: For the ﬁrst three behavioral tests DistMult is better at lower dimension sizes, whereas for

Original: RotatE higher dimensions sizes bring a small performance increase.
Revised: RotatE higher dimensions sizes bring a small performance increase.

Original: For Test 4, we ﬁnd that all models in Figure 2(a) perform well (the lower the MRR, the better), showing that models do not easily mistake asymmetric relations for symmetric ones.
Revised: For Test 4, we ﬁnd that all models in Figure 2(a) perform well (the lower the MRR, the better), showing that models do not easily mistake asymmetric relations for symmetric ones.

Original: 3.2 Capability: Entity Hierarchy
Revised: 3.2 Capability: Entity Hierarchy

Original: Entities in a KG may be associated with entity types, which can be expressed in a hierarchical structure.
Revised: Entities in a KG may be associated with entity types, which can be expressed in a hierarchical structure.

Original: As shown in Figure 3, entity types vary from general and ambiguous concepts, such as Thing or Agent , to more speciﬁc ones such as Actor and Ballet
Revised: As shown in Figure 3, entity types vary from general and ambiguous concepts, such as Thing or Agent , to more speciﬁc ones such as Actor and Ballet

Original: Dancer .
Revised: Dancer .

Original: Therefore, entities are associated with types belonging to diﬀerent hierarchy levels.
Revised: Therefore, entities are associated with types belonging to diﬀerent hierarchy lev- els.

Original: Likewise, KG relations sometimes impose constraints on the entity types about their domain and range (e.g., birthPlace accepts as head and tail an entity of type Person and
Revised: Likewise, KG relations sometimes impose constraints on the entity types about their domain and range (e.g., birthPlace accepts as head and tail an entity of type Person and

Original: Place respectively).
Revised: Place respectively).

Original: We deﬁne two tests to analyze to what extent KGE models have learnt to respect entity type constraints when making predictions.
Revised: We deﬁne two tests to analyze to what extent KGE models have learnt to respect entity type constraints when making predictions.

Original: The ﬁrst test explores model performance based on the entity level of gold tails to investigate how well the models handle diﬀerent speciﬁcity levels.
Revised: The ﬁrst test explores model performance based on the entity level of gold tails to investigate how well the models handle diﬀerent speciﬁcity levels.

Original: In the second test we explore how much model performance could be improved if the model had learnt to associate a relation with the correct entity type.
Revised: In the second test we explore how much model performance could be improved if the model had learnt to associate a relation with the correct entity type.

Original: To set up the tests, we map each entity of a triple from Fb15k-237 to its counterpart in DBpedia , which deﬁnes entity types and arranges them in a hierarchical manner (see Figure 3).
Revised: To set up the tests, we map each entity of a triple from Fb15k-237 to its counterpart in DBpedia , which deﬁnes entity types and arranges them in a hierarchical manner (see Figure 3).

Original: A few entities have no direct mapping and are thus ﬁltered out, we also ﬁlter out level 5 entities as they only occur rarely (21 triples in total).
Revised: A few entities have no direct mapping and are thus ﬁltered out, we also ﬁlter out level 5 entities as they only occur rarely (21 triples in total).

Original: Test 1: Gold Tail.
Revised: Test 1: Gold Tail.

Original: To test how well KGE models perform at diﬀerent entity type levels, we create a test set for each level.
Revised: To test how well KGE models perform at diﬀerent entity type levels, we create a test set for each level.

Original: For this, we iterate over each triple in the original test set and look up the level of the gold tail’s entity type.
Revised: For this, we iterate over each triple in the original test set and look up the level of the gold tail’s entity type.

Original: The triple is then added to the test set of the found level.
Revised: The triple is then added to the test set of the found level.

Original: With this, we can test how performance varies as we move from very general entity types (Level 0) to more speciﬁc ones (Level 4).
Revised: With this, we can test how performance varies as we move from very general entity types (Level 0) to more speciﬁc ones (Level 4).

Original: Test 2: Type Constraints.
Revised: Test 2: Type Constraints.

Original: Next, we would like to test how model performance changes if we explicitly restrict the set of possible tail entities based on entity type.
Revised: Next, we would like to test how model performance changes if we explicitly restrict the set of possible tail entities based on entity type.

Original: If this improves the results, then this indicates that the models have not yet suﬃciently learnt to associate a relation with the correct entity type.
Revised: If this improves the results, then this indicates that the models have not yet suﬃciently learnt to associate a relation with the correct entity type.

Original: To test this, we use the training data to compile for every relation the most likely type at each level.
Revised: To test this, we use the training data to compile for every relation the most likely type at each level.

Original: For each test set (Level 0 - 4), we apply entity type restrictions in the following way: (1) At prediction time, given a relation, we restrict the entity set to have the entity type that occurred most often for this relation in the training set.
Revised: For each test set (Level 0 - 4), we apply entity type restrictions in the following way: (1) At prediction time, given a relation, we restrict the entity set to have the entity type that occurred most often for this relation in the training set.

Original: For example, for the relation BirthPlace at level 2, we restrict the entity set to be of type Place or a subtype of Place .
Revised: For example, for the relation BirthPlace at level 2, we restrict the entity set to be of type Place or a subtype of Place .

Original: (2) We move up in the hierarchy tree, which causes the restriction to relax with each step up.
Revised: (2) We move up in the hierarchy tree, which causes the restriction to relax with each step up.

Original: Moving up to level 0 would then be equal to not placing any restrictions.
Revised: Moving up to level 0 would then be equal to not placing any restrictions.

Original: Test 3: Gold Type
Revised: Test 3: Gold Type

Original: Finally, we would like to test if a model has learnt to recognize that the entity to be predicted should be of a certain type.
Revised: Finally, we would like to test if a model has learnt to recognize that the entity to be predicted should be of a certain type.

Original: For instance, for the triple (George
Revised: For instance, for the triple (George

Original: W. Bush, FatherOf, ?) , the predicted tail should be of type Person .
Revised: W. Bush, FatherOf, ?) , the predicted tail should be of type Person .

Original: To investigate this, we mark the ﬁrst entity that shares the same type as the gold tail as correct, even if it is not necessarily the gold tail itself.
Revised: To investigate this, we mark the ﬁrst entity that shares the same type as the gold tail as correct, even if it is not necessarily the gold tail itself.

Original: In the previous example, if the correct answer is ranked 300, meaning that the model only obtains a tail of type Person at rank 300, then the model clearly struggles with entity types.
Revised: In the previous example, if the correct answer is ranked 300, meaning that the model only obtains a tail of type Person at rank 300, then the model clearly struggles with entity types.

Original: However, if the model always predicts the correct type as the ﬁrst prediction, then the model has successfully learned which type of entity to expect
Revised: However, if the model always predicts the correct type as the ﬁrst prediction, then the model has successfully learned which type of entity to expect

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: a tail diﬀers by their level in the hierarchy.
Revised: a tail diﬀers by their level in the hierarchy.

Original: Figure (b) shows that models beneﬁt from type constraints.
Revised: Figure (b) shows that models beneﬁt from type constraints.

Original: Figure (c) shows that it is easier to predict more general entity types.
Revised: Figure (c) shows that it is easier to predict more general entity types.

Original: Results.
Revised: Results.

Original: For the Test 1 (“ Gold Tail ”), Figure 4 (a) shows the performance for the different entity type levels.
Revised: For the Test 1 (“ Gold Tail ”), Figure 4 (a) shows the performance for the different entity type levels.

Original: The model rankings are consistent across the diﬀerent levels with
Revised: The model rankings are consistent across the diﬀerent levels with

Original: LineaRE and HAKE performing best.
Revised: LineaRE and HAKE performing best.

Original: Regarding the diﬀerent levels, we ﬁnd that levelentity types are the easiest to predict, while more speciﬁc levels seem to be more diﬃcult.
Revised: Regarding the diﬀerent levels, we ﬁnd that levelentity types are the easiest to predict, while more speciﬁc levels seem to be more diﬃcult.

Original: In particular, all models perform by far the worst at level 2.
Revised: In particular, all models perform by far the worst at level 2.

Original: The drop of performance in level 2 compared to the other levels likely stems from the type of relation.
Revised: The drop of performance in level 2 compared to the other levels likely stems from the type of relation.

Original: The relations that occur in level 2 rarely occur in other levels and most of the relations expect a tail of the level 2 type “ Person ”.
Revised: The relations that occur in level 2 rarely occur in other levels and most of the relations expect a tail of the level 2 type “ Person ”.

Original: We conjecture that the models are particularly bad at predicting in this scenario because of the large number of entities that have the type “ Person ” (4,950; which constitutes 34% of all the entities in the set).
Revised: We conjecture that the models are particularly bad at predicting in this scenario because of the large number of entities that have the type “ Person ” (4,950; which constitutes 34% of all the entities in the set).

Original: The results for Test 2 (“ Type Constraints ”) can be found in Figure 4 (b).
Revised: The results for Test 2 (“ Type Constraints ”) can be found in Figure 4 (b).

Original: We see an improvement in MRR across the board when using type constraints during prediction.
Revised: We see an improvement in MRR across the board when using type constraints during prediction.

Original: For example, for the level 1 test set, we ﬁnd an improvement of about 0.05 MRR when restricting the entity set to belong to a relation’s most common level 1 entity type.
Revised: For example, for the level 1 test set, we ﬁnd an improvement of about 0.05 MRR when restricting the entity set to belong to a relation’s most common level 1 entity type.

Original: Additionally, at all levels, we ﬁnd that applying the restriction at the same level helps the most.
Revised: Additionally, at all levels, we ﬁnd that applying the restriction at the same level helps the most.

Original: This shows that models have not yet fully learnt to associate entity types with relations.
Revised: This shows that models have not yet fully learnt to associate entity types with relations.

Original: One future direction could be to to investigate how entity type information can be used to further improve models.
Revised: One future direction could be to to investigate how entity type information can be used to further improve models.

Original: Test 3 results are depicted in Figure 4 (c).
Revised: Test 3 results are depicted in Figure 4 (c).

Original: We ﬁnd that (1) all models perform similarly across the diﬀerent levels; (2) at all levels and for all models we observe relatively high MRR ( ≥ 0 . 6) showing that all models are quite good at correctly predicting the type of the tail entity; (3) performance slightly drops when moving to deeper levels, which is understandable as entities at higher levels are more general and more common; therefore easier to predict (for example, level 1 contains entities of types, e.g. Agent , Deity , while deeper levels are more speciﬁc, e.g. Artist or Musician ).
Revised: We ﬁnd that (1) all models perform similarly across the diﬀerent levels; (2) at all levels and for all models we observe relatively high MRR ( ≥ 0 . 6) showing that all models are quite good at correctly predicting the type of the tail entity; (3) performance slightly drops when moving to deeper levels, which is understandable as entities at higher levels are more general and more common; therefore easier to predict (for example, level 1 contains entities of types, e.g. Agent , Deity , while deeper levels are more speciﬁc, e.g. Artist or Musician ).

Original: These results of Test 3 contrast with Test 2, where we ﬁnd that despite correctly predicting the correct type at level 2 and level 4, it is much more diﬃcult to predict the correct tail.
Revised: These results of Test 3 contrast with Test 2, where we ﬁnd that despite correctly predicting the correct type at level 2 and level 4, it is much more diﬃcult to predict the correct tail.

Original: In the future, we would like to investigate the reason for this discrepancy.
Revised: In the future, we would like to investigate the reason for this discrepancy.

Original: 4. Conclusion
Revised: 4. Conclusion

Original: Knowledge graphs and knowledge graph embedding (KGE) models can be used for link prediction to infer new triples.
Revised: Knowledge graphs and knowledge graph embedding (KGE) models can be used for link prediction to infer new triples.

Original: However, they are typically evaluated using averaged accuracy metrics computed over a test set.
Revised: However, they are typically evaluated using averaged accuracy metrics computed over a test set.

Original: As a result, it remains unclear what exactly these models have learnt and which model might be the most suitable for a particular task.
Revised: As a result, it remains unclear what exactly these models have learnt and which model might be the most suitable for a particular task.

Original: We presented a framework to systematically test the capabilities of KGE models using behavioral tests.
Revised: We presented a framework to systematically test the capabilities of KGE models using behavioral tests.

Original: For two initial capabilities we deﬁned several tests and ran them for six KGE models.
Revised: For two initial capabilities we deﬁned several tests and ran them for six KGE models.

Original: Crucially, we ﬁnd that the model performance on the original test set does not necessarily mirror the same performance when testing models for a speciﬁc capability.
Revised: Crucially, we ﬁnd that the model performance on the original test set does not necessarily mirror the same performance when testing models for a speciﬁc capability.

Original: For instance,
Revised: For instance,

Original: ComplEx ranked second to last on the original dataset but is the best at predicting unseen triples for symmetric relations.
Revised: ComplEx ranked second to last on the original dataset but is the best at predicting unseen triples for symmetric relations.

Original: We hope that this initial framework will also inspire fellow researcher to contribute by adding more tests, models and datasets.
Revised: We hope that this initial framework will also inspire fellow researcher to contribute by adding more tests, models and datasets.

Original: Appendix A. Scoring Functions
Revised: Appendix A. Scoring Functions

Original: The scoring functions for various KGE models can be found in Table 1.
Revised: The scoring functions for various KGE models can be found in Table 1.

Original: [Table]
Revised: [Table]

Original: Appendix B. FB15k-237 Symmetric Relations
Revised: Appendix B. FB15k-237 Symmetric Relations

Original: We report the manually extracted symmetric relations for FB15k-237 in Table
Revised: We report the manually extracted symmetric relations for FB15k-237 in Table

Original: [Table]
Revised: [Table]

Original: /celebrity/sexual relationships.
Revised: /celebrity/sexual relationships.

Original: /celebrities/romantic relationship/celebrity /inﬂuence/inﬂuence node/peers.
Revised: /celebrities/romantic relationship/celebrity /inﬂuence/inﬂuence node/peers.

Original: /inﬂuence/peer relationship/peers /location/location/adjoin s. /location/adjoining relationship/adjoins /people/person/spouse s./people/marriage/spouse /people/person/sibling s./people/sibling relationship/sibling
Revised: /inﬂuence/peer relationship/peers /location/location/adjoin s. /location/adjoining relationship/adjoins /people/person/spouse s./people/marriage/spouse /people/person/sibling s./people/sibling relationship/sibling

Original: Appendix C. Hyperparameters of Trained Models
Revised: Appendix C. Hyperparameters of Trained Models

Original: We train models for DistMult
Revised: We train models for DistMult

Original: [Yang et al., 2015], RotatE [Sun et al., 2019], LinearRE [Peng and Zhang, 2020] and HAKE [Zhang et al., 2020].
Revised: [Yang et al., 2015], RotatE [Sun et al., 2019], LinearRE [Peng and Zhang, 2020] and HAKE [Zhang et al., 2020].

Original: These models diﬀer in their deﬁnition of the scoring function φ ( w ; d ), which may be found in Table 1.
Revised: These models diﬀer in their deﬁnition of the scoring function φ ( w ; d ), which may be found in Table 1.

Original: We use the python framework Pykeen [Ali et al., 2021] for all our experiments, it provides ready-to-use implementations of both the DistMult and the Rotate models.
Revised: We use the python framework Pykeen [Ali et al., 2021] for all our experiments, it provides ready-to-use implementations of both the DistMult and the Rotate models.

Original: For the DistMult experiments, we ﬁx the following hyperparameters: 500 negative per positive samples for the negative sampler, a learning rate of 1 e − 3 .
Revised: For the DistMult experiments, we ﬁx the following hyperparameters: 500 negative per positive samples for the negative sampler, a learning rate of 1 e − 3 .

Original: We train the ﬁrst group of models for 100 epochs while varying the size of embedding dimensions.
Revised: We train the ﬁrst group of models for 100 epochs while varying the size of embedding dimensions.

Original: For the second group, we train the models for diﬀerent epochs while keeping the hidden dimensions at 100.
Revised: For the second group, we train the models for diﬀerent epochs while keeping the hidden dimensions at 100.

Original: For the RotatE model, we use 256 negative per positive samples for the negative sampler, a learning rate of 5 e − 5 with the adam optimizer and a training and evaluation batch sizes of 1,024 and 16, respectively.
Revised: For the RotatE model, we use 256 negative per positive samples for the negative sampler, a learning rate of 5 e − 5 with the adam optimizer and a training and evaluation batch sizes of 1,024 and 16, respectively.

Original: We either ﬁx the dimensions at 1,000 or the epochs at 500, similarly to the DistMult trials.
Revised: We either ﬁx the dimensions at 1,000 or the epochs at 500, similarly to the DistMult trials.

Original: For both KGE models, we run 5 independent experiments for every set of hyper-parameters, we then report the average performance.
Revised: For both KGE models, we run 5 independent experiments for every set of hyper-parameters, we then report the average performance.

Original: For the experiments of ComplEx , we use the oﬃcial source code of the Rotate model, where the best model hyper parameters are reported as follows: batch size 1024, negative sample size 256, hidden dimensions of 1000, γ 200, α 1.0, learning rate 0.001, maximum steps of 100000 and a test batch size of 16.
Revised: For the experiments of ComplEx , we use the oﬃcial source code of the Rotate model, where the best model hyper parameters are reported as follows: batch size 1024, negative sample size 256, hidden dimensions of 1000, γ 200, α 1.0, learning rate 0.001, maximum steps of 100000 and a test batch size of 16.

Original: For the HAKE model, we use the recommended conﬁgurations in the paper, which are also provided with the source code, with a batch size of 1024, negative sample size 256, hidden dimensions 1000, γ 9.0, α 1.0, learning rate 0.00005, maximum steps 100000, test batch size 16, and their custom parameters of modulus and phase weights of 3.5 and 1. respectively.
Revised: For the HAKE model, we use the recommended conﬁgurations in the paper, which are also provided with the source code, with a batch size of 1024, negative sample size 256, hidden dimensions 1000, γ 9.0, α 1.0, learning rate 0.00005, maximum steps 100000, test batch size 16, and their custom parameters of modulus and phase weights of 3.5 and 1. respectively.

Original: For HyperKG , the negative sample size is 5, λ is 0.8, γ is 0.5 and β is n/2, with n=100.
Revised: For HyperKG , the negative sample size is 5, λ is 0.8, γ is 0.5 and β is n/2, with n=100.

Original: For LineaRE , α is 0.5, β is 1.0, γ is set to 12, the embedding dimensions are 1000, a batch size of 2048 and a negative sample size of 128.
Revised: For LineaRE , α is 0.5, β is 1.0, γ is set to 12, the embedding dimensions are 1000, a batch size of 2048 and a negative sample size of 128.

Original: Appendix D. Behavioral Tests D.1 Capability 1: Relation Symmetry
Revised: Appendix D. Behavioral Tests D.1 Capability 1: Relation Symmetry

Original: Examples in Figure 5 demonstrate the type of triples that are considered for the relation symmetry tests.
Revised: Examples in Figure 5 demonstrate the type of triples that are considered for the relation symmetry tests.

Original: D.1.1 Test Set Statistics
Revised: D.1.1 Test Set Statistics

Original: The symmetry tests have the following number of triples:
Revised: The symmetry tests have the following number of triples:

Original: Test
Revised: Test

Original: Test Test
Revised: Test Test

Original: Test5, 1,  3,
Revised: Test5, 1,  3,

Original: The original Test Set contains 20,466 triples that represent a variety of relations, both symmetric and asymmetric.
Revised: The original Test Set contains 20,466 triples that represent a variety of relations, both symmetric and asymmetric.

Original: D.1.2 Additional Results
Revised: D.1.2 Additional Results

Original: In the main paper, in Figure 2, we provide the results on the symmetry tests for varying the embedding dimensions for DistMult and RotatE .
Revised: In the main paper, in Figure 2, we provide the results on the symmetry tests for varying the embedding dimensions for DistMult and RotatE .

Original: In Figure 6a, we provide the results for varying the epochs.
Revised: In Figure 6a, we provide the results for varying the epochs.

Original: D.2 Capability 2: Entity Hierarchy D.2.1 Test Set Statistics
Revised: D.2 Capability 2: Entity Hierarchy D.2.1 Test Set Statistics

Original: The hierarchy tests have the following number of triples:
Revised: The hierarchy tests have the following number of triples:

Original: Level
Revised: Level

Original: Level Level
Revised: Level Level

Original: Level Level
Revised: Level Level

Original: Level3, 7, 3,
Revised: Level3, 7, 3,

Original: Triples from Level 5 and Level 6 were ﬁltered out because of the small data set size.
Revised: Triples from Level 5 and Level 6 were ﬁltered out because of the small data set size.

Original: The original Test Set contains 20,466 triples that represent a variety of relations with tails of diﬀerent levels in the entity type hierarchy.
Revised: The original Test Set contains 20,466 triples that represent a variety of relations with tails of diﬀerent levels in the entity type hierarchy.

Original: D.2.2 Additional Results
Revised: D.2.2 Additional Results

Original: In the main paper, in Figure 4, we provide the results on the hierarchy tests for varying the Epochs for DistMult .
Revised: In the main paper, in Figure 4, we provide the results on the hierarchy tests for varying the Epochs for DistMult .

Original: In Figure 6b, we provide the results for varying the embedding dimensions, and in Figure 7, we provide the same results for RotatE
Revised: In Figure 6b, we provide the results for varying the embedding dimensions, and in Figure 7, we provide the same results for RotatE

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]