O N U NDERSTANDING K NOWLEDGE G RAPH
R EPRESENTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Many methods have been developed to represent knowledge graph data, which implicitly exploit low-rank latent structure in the data to encode known information and enable unknown facts to be inferred.
To predict whether a relationship holds between entities, their embeddings are typically compared in the latent space following a relation-speciﬁc mapping.
Whilst link prediction has steadily improved, the latent structure, and hence why such models capture semantic information, remains unexplained.
We build on recent theoretical interpretation of word embeddings to derive an explicit structure for the representations of relations between entities.
From this we explain properties and justify the relative performance of leading knowledge graph representation methods for identiﬁable relation types, including their often overlooked ability to make independent predictions.
I NTRODUCTION
Knowledge graphs are large repositories of binary relations between words (or entities) in the form of fact triples (subject, relation, object) .
Many models have been developed for learning representations of entities and relations in knowledge graphs, such that known facts can be recalled and previously unknown facts can be inferred, a task known as link prediction .
Recent link prediction models (e.g.
Bordes et al., 2013; Trouillon et al., 2016;
Balaževi ´ c et al., 2019b) learn entity representations, or embeddings , of far lower dimensionality than the number of entities, by capturing latent structure in the data.
Relations are typically represented as a mapping from the embedding of a subject entity to its related object entity embedding(s).
Although the performance of knowledge graphlink prediction models has steadily improved for nearly a decade, relatively little is understood of the low-rank latent structure that underpins these models, which we address in this work.
We start by drawing a parallel between entity embeddings in knowledge graphs and unsupervised word embeddings, as learned by algorithms such as Word2Vec (W2V)
(Mikolov et al., 2013) and
GloVe (Pennington et al., 2014).
We assume that words have latent features, e.g. meaning(s), tense, grammatical type, that are innate and ﬁxed, irrespective of what an embedding may capture (which may be only a part, subject to the embedding method and/or the data source); and that this same latent structure gives rise to patterns observed in the data, e.g. in word co-occurrence statistics and in which words are related to which.
As such, an understanding of the latent structure from one embedding task (e.g. word embedding) might be useful to another (e.g. knowledge graph entity embedding).
Recent work (Allen & Hospedales, 2019; Allen et al., 2019) theoretically explains how semantic properties are encoded in word embeddings that (approximately) factorise a matrix of word cooccurrence pointwise mutual information (PMI), e.g. as is known for W2V (Levy & Goldberg, 2014).
Semantic relationships between words (speciﬁcally similarity, relatedness, paraphrase and analogy) are proven to manifest as linear relationships between rows of the PMI matrix (subject to known error terms), of which word embeddings can be considered low-rank projections.
This explains why similar words (e.g. synonyms) have similar embeddings; and embeddings of analogous word pairs share a common “vector offset”.
Importantly, this insight allows us to identify geometric relationships between such word embeddings necessary for other semantic relations to hold, such as those of knowledge graphs.
These relation conditions describe relation-speciﬁc mappings between entity embeddings, i.e. relation representations, providing a “blue-print” against which to consider knowledge graph representation models.
We ﬁnd that various properties of knowledge graph representation models, including the relative
[Table]
performance of leading link prediction models, accord with predictions based on these relation conditions, suggesting a commonality to the latent structure learned in word embedding models and knowledge graph representation models, despite the signiﬁcant differences between their training data and methodology.
In summary, the key contributions of this work are:
• to use recent understanding of PMI-based word embeddings to derive what a relation representation must achieve to map a subject word embedding to all related object word embeddings (relation conditions), based on which relations can be categorised into three types ; • to show that properties of knowledge graph models ﬁt predictions made from relation conditions, e.g. strength of a relation’s relatedness aspect is reﬂected in the eigenvalues of its relation matrix; • to show that the performance per relation of leading link prediction models corresponds to the ability of the model’s architecture to meet the relation conditions of the relation’s type, i.e. the better the architecture of a knowledge graph representation model aligns with the form theoretically derived for PMI-based word embeddings, the better the model performs; and • noting how ranking metrics can be ﬂawed, to provide novel insight into the prediction
accuracy per relation of recent knowledge graph models, an evaluation metric we recommend in future.
B ACKGROUND
Our work draws on knowledge graph representation and word embedding.
Whilst related, these tasks differ materially in their training data.
The former is restricted to datasets crafted by hand or automatically generated, the latter has the vast abundance of natural language text (e.g. Wikipedia).
2. K NOWLEDGE GRAPH REPRESENTATION
Almost all recent knowledge graph models represent entities e s , e o as vectors e s , e o ∈ R d e of low dimension (e.g. d e =200 ) relative to the number of entities n e (typically of order 10 4 ), and relations as transformations in the latent space from subject entity embedding to object.
These models are distinguished by their score function , which deﬁnes (i) the form of the relation transformation, e.g. matrix multiplication, vector addition; and (ii) how “closeness” between the transformed subject embedding and an object embedding is evaluated, e.g. dot product, Euclidean distance.
Score functions can be non-linear (e.g. Dettmers et al.
(2018)), or linear and sub-categorised as additive, multiplicative or both.
We focus on linear models due to their simplicity and strong performance at link prediction (including state-of-the-art).
Table 1 shows the score functions of competitive linear models that span all linear sub-categories:
TransE (Bordes et al., 2013), DistMult (Yang et al., 2015),
TuckER (Balaževi´c et al., 2019b) and MuRE (Balaževi´c et al., 2019a).
Additive models typically use Euclidean distance and contain a relation-speciﬁc translation from a (possibly transformed) subject to a (possibly transformed) object entity embedding.
A generic additive score function is given by φ
[Equation]
The simplest example is TransE for which R s =
R
o
=
I and b s =
b o =0 .
The score function of MuRE has R o
=
I and so combines multiplicative ( R s = R ) and additive ( r ) components.
Multiplicative models have the generic score function φ ( e s , r, e o ) =
e (cid:62) s Re o = (cid:104) e
( r ) s , e o (cid:105) , i.e. a bilinear product of the entity embeddings and a relation-speciﬁc matrix R .
DistMult is a simple example with diagonal R and so cannot model asymmetric relations (Trouillon et al., 2016).
In
TuckER, each relation-speciﬁc R = W × 3 r is a linear combination of d r “prototype” relation matrices in a core tensor W ∈ R d e × d r × d e (where × n denotes tensor product along mode n ), facilitating multi-task learning across relations.
2. W ORD EMBEDDING
Algorithms such as Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) generate succinct low-rank word embeddings that perform well on downstream tasks (Baroni et al., 2014).
Such models predict the context words ( c j ) observed around each target word ( w i ) in a text corpus using shallow neural networks.
Whilst recent language models (e.g. Devlin et al.
; Peters et al.
(2018)) create impressive context-speciﬁc word embeddings, we focus on the former embeddings since knowledge graph entities have no obvious context and, more importantly, they are interpretable .
Levy & Goldberg (2014) show that, for a dictionary of unique words D and embedding dimension d
(cid:28)| D | , W2V’s loss function is minimised when its weight matrices W , C ∈ R d ×| D | (whose columns are word embeddings w i , c j ) factorise a word co-occurrence pointwise mutual information (PMI) matrix, subject to a shift term (PMI ( w i , c j )=
log P ( w i ,c j ) P ( w i ) P ( c j ) ).
This connects W2V to earlier countbased embeddings and speciﬁcally to PMI, which has a long history in linguistic analysis (Turney &
Pantel, 2010).
From its loss function, GloVe can be seen to perform a comparable factorisation.
Recent work shows why word embeddings that factorise such PMI matrix encode semantic word relationships (Allen & Hospedales, 2019; Allen et al., 2019).
The authors show that word embeddings can be seen as low-rank projections of high dimensional PMI vectors (rows of the PMI matrix), between which the semantic relationships of similarity, relatedness, paraphrase and analogy provably manifest as linear geometric relationships (subject to deﬁned error terms), which are then preserved, under a sufﬁciently linear projection, between word embeddings.
Thus similar words have similar embeddings, and the embeddings of analogous word pairs share a common vector offset.
Speciﬁcally, the PMI vectors ( p x ) of an analogy “ man is to king as woman is to queen ” satisfy p Q − p W
≈ p K − p M because the difference between words associated with king and man (e.g. reign , crown ) mirrors that between queen and woman .
This leads to a common difference between their co-occurrence distributions (over all words), giving a common difference between their PMI vectors, which projects to a common difference between embeddings.
Any discrepancy in the mirroring of word associations is shown to introduce error, weakening the analogy, as does a lack of statistical independence within certain word pairs (see (Allen & Hospedales, 2019)).
The common difference in word co-occurrence distributions, e.g. the increased association with words { reign , crown , etc.}, can be interpreted semantically as a common change in context ( context-shift ) that transforms man to king and woman to queen by adding a royal context.
Under this interpretation, context can also be subtracted, e.g. “ king is to man as queen is to woman ” (minus royal ); or both, e.g. “ boy is to king as girl is to queen ” (minus youth plus royal ).
Adding context can also be interpreted as specialisation , and subtracting context as generalisation .
This establishes a correspondence between common word embedding vector offsets and semantic context-shifts.
Although the projection from PMI vectors to word embeddings preserves the relative relationships, and thus the above semantic interpretability of common embedding differences, a direct interpretation of dimensions themselves is obscured, not least because any embedding matrix can be arbitrarily scaled/rotated if the other is inversely transformed.
R ELATIONSHIPS BETWEEN EMBEDDINGS OF RELATED WORDS
Our aim is to build on the understanding of PMI-based word embeddings (henceforth word embeddings ), to identify what a knowledge graph relation representation needs to achieve to map all subject word embeddings to all related object word embeddings.
We note that if a semantic relation between two words implies a particular geometric relationship between their embeddings, then the latter serves as a necessary quantitative condition for the former to hold (a relation condition ).
Relation conditions implicitly deﬁne a relation-speciﬁc mapping by which all subject embeddings are mapped to all related object embedding(s), allowing related entities to be identiﬁed by a proximity measure (e.g.
Euclidean distance or dot product).
Since this is the approach of many knowledge graph representation models, their performance can be contrasted with their ability to express mappings that satisfy required relation conditions.
For example, similarity and context-shift relations respectively imply closeness and a relation-speciﬁc vector offset between embeddings (S2.2).
Such relation conditions can be tested for by respectively making no change to the subject entity or adding the relation-speciﬁc offset, before measuring proximity with the object.
We note that since relation conditions are not necessarily sufﬁcient, they do not guarantee a relation holds, i.e. false positives may arise.
[Figure]
In general, the data from which knowledge graph embeddings are derived differs greatly to the co-occurrence data used for word embeddings, and the latter would not be anticipated to be learned by knowledge graph models.
However, word embeddings provide a known solution (i.e. minimise the loss function) of any knowledge graph model able to express the required mapping(s) derived from relation conditions, where the loss function measures proximity between mapped entities.
The relation conditions for certain relation types (underlined) follow readily from S2.2:
• Similarity: Semantically similar words induce similar distributions over the words they co-occur with.
Thus their PMI vectors (Fig 1a) and word embeddings are similar.
• Relatedness: The relatedness of two words can be deﬁned in terms of the words with which both co-occur similarly ( S ∈ D ), which deﬁne the nature of relatedness, e
g. milk and cheese are related by S = { dairy , breakfast , ... } ; and | S | reﬂects the strength of relatedness.
Since PMI vector components corresponding to S are similar (Fig 1b), embeddings of “ S -related ” words have similar components in the subspace V S that spans the projected PMI vector dimensions corresponding to S .
The rank of V S might be expected to reﬂect relatedness strength.
In general, relatedness is a weaker, more variable relation than similarity, its limiting case with S = D and rank( V S )=
d . • Context-shift:
In the context of word embeddings, analogy typically refers to relational similarity (Turney, 2006; Gladkova et al., 2016).
More speciﬁcally, the relations within analogies that give a common vector offset between word embeddings require related words to have a common difference between their distributions of co-occurring words, deﬁned as a context-shifts (see S2.2).
These relations are strictly 1-to-1 and include an aspect of relatedness due to the word associations in common (Fig 1d).
A specialisation relation is a context-shift in which context is only added (Fig 1c).
• Generalised context-shift: Context-shift relations are generalised to 1-to-many, many-to-1 and many-to-many relations by letting the fully-speciﬁed added or subtracted context be one of a (relationspeciﬁc) context set
(Fig 1e), e
g. allowing an entity to be any colour or anything blue.
The potential scope and size of each context set means these relations can vary greatly.
The limiting case for small context sets has a single context in each, whereby the relation is an explicit context-shift (as above), and the difference between embeddings is a known vector offset.
In the limiting case where context sets are large, the added/subtracted context is so loosely deﬁned that, in effect, only the relatedness aspect of the relation and thus only the common subspace component of embeddings is known.
Link to set theory: Viewing PMI vectors as sets of word associations and taking intuition from
Fig 1, the above relations can be seen to reﬂect set operations: similarity as set equality; relatedness as equality of a subset; and context-shift as the set difference equalling a relation-speciﬁc set.
This highlights how the relatedness aspect of a relation reﬂects features that must be common, and contextshift reﬂects features that must differ.
Whilst this mirrors an intuitive notion of “feature vectors”, we emphasise that this is grounded in the co-occurrence statistics of PMI-based word embeddings.
3. C ATEGORISING KNOWLEDGE GRAPH RELATIONS
Analysing the relations of popular knowledge graph datasets with the above perspective, we ﬁnd that they comprise (i) a relatedness aspect that reﬂects a common theme (e.g. both entities are animals or geographic terms); and (ii) speciﬁc word associations of the subject and/or object entities.
Speciﬁcally, relations appear to fall under a hierarchy of three relation types : highly related ( R ); (generalised) specialisation ( S ); and (generalised) context-shift ( C ).
As above, “generalised” indicates that any added/subtracted contexts can be from a set.
From Fig 1, type R relations can be seen as a special case of S, which, in turn, is a special case of C. Type C is therefore a generalised case of all considered relations.
Whilst there are several other ways to classify relations (e.g. by their hierarchy,
[Table]
transitivity), by considering relation conditions, we delineate by the required mathematical form (and complexity) of their representation.
Table 2 shows a categorisation of the relations of the WN18RR dataset (Dettmers et al., 2018), containing 11 relations between 40,943 entities.
1 An explanation for this assignment is in Appx.
A and that for NELL-995 (Xiong et al., 2017) is in Appx. B. A review of the FB15k-237 dataset (Toutanova et al., 2015) shows the vast majority of relations to be of type C preventing a contrast between relation types being drawn, hence we do not consider that dataset.
3. R ELATIONS AS MAPPINGS OF EMBEDDINGS
Given the relation conditions of a particular relation type, we can recognise mappings that meet them and thus loss functions (that evaluate the proximity of mapped entity embeddings by dot product or Euclidean distance) able to identify relations of that type between PMI-based word embeddings .
We then contrast these theoretically inspired loss functions (one per relation type) with those of knowledge graph models (Table 1) and, on the outline assumption that a common low-rank latent structure is exploited by both word embeddings and knowledge graph models, predict properties and the relative performance of different knowledge graphs models for different relation types.
R:
To evidence S -relatedness, both entity embeddings e s , e o must be projected onto a subspace V
S , where their images are compared.
Projection requires multiplication by a matrix
P r ∈ R d × d and cannot be achieved additively, except in the limiting case of similarity, when P r = I or vector r ≈ 0 is added.
Comparison by dot product gives ( P r e s ) (cid:62) ( P r e o ) =
e (cid:62) s P (cid:62) r P r e o = e (cid:62) s M r e o (for a relation-speciﬁc symmetric M r = P (cid:62) r P r ).
Euclidean distance gives (cid:107) P r e s − P r e o (cid:107) 2 = ( e s − e o ) (cid:62) M r ( e s − e o )
= (cid:107) P r e s (cid:107) 2 − 2 e (cid:62) s M
r e o
+ (cid:107) P r e o (cid:107) 2 .
S/C:
Evidencing these relations requires a test both for S -relatedness and for relation-entity-speciﬁc embeddings component(s) ( v sr , v or ).
This can be achieved by (i) multiplying both entity embeddings by a relation-speciﬁc projection matrix P r that projects onto the subspace that spans the low-rank projection of dimensions corresponding to S , v sr and v or , (which tests for S -relatedness whilst preserving any entity-speciﬁc embedding components); and (ii) adding a relation-speciﬁc vector r = v or − v sr to the transformed subject entity embeddings.
Comparison of the ﬁnal transformed entity embeddings by dot product equates to ( P r e s + r ) (cid:62) P r e o ; and by Euclidean distance to (cid:107) P
r e
[Equation]
Contrasting the above loss functions with those of knowledge graph models (Table 1), we make the following predictions: ( P1 ) the ability to learn the representation of a particular relation is expected to reﬂect the complexity of its type (R > S > C), and whether all relation conditions (e.g. additive or multiplicative interactions) can be met under a given model; ( P2 ) relation matrices for relatedness (type R) relations are highly symmetric; ( P3 ) offset vectors for relatedness relations have low norm; and ( P4 ) as a proxy to the rank of V S , the eigenvalues of a relation matrix reﬂect a relation’s strength of relatedness.
To elaborate: P1 anticipates that additive-only models (e.g. TransE) are not suited to identifying the relatedness aspect of relations (except in limiting cases of similarity, requiring a zero vector); and multiplicative-only models (e.g. DistMult) should perform well on type R but are not suited to identifying entity-speciﬁc features of type S/C, for which an asymmetric relation matrix in TuckER may help compensate.
Further, the loss function of MuRE closely resembles that derived for type C relations (which generalise all others) and is thus expected to perform best overall.
[Table]
[Table]
C OMPARING KNOWLEDGE GRAPH MODELS
We test the predictions made on the basis of word embeddings by comparing the performance of competitive knowledge graph models, TransE, DistMult, TuckER and MuRE (see S2), which entail different forms of relation representation, on all WN18RR relations and a similar number of NELL995 relations (selected to represent each relation type).
Since applying the logistic sigmoid to the score function of TransE does not give a probabilistic interpretation comparable to other models, we include MuRE I , a constrained variant of MuRE with R s = R
o =
I , as a proxy to TransE for a fairer comparison.
Implementation details are included in Appx.
D. For evaluation, we generate 2 n e evaluation triples for each test triple (for the number of entities n e ) by ﬁxing the subject entity e s and relation r and replacing the object entity e o with all possible entities and then keeping e o and r ﬁxed and varying e s .
The obtained scores are ranked to give the standard metric hits@10 (Bordes et al., 2013), i.e. the fraction of times a true triple appears in the top 10 ranked evaluation triples.
4. P ERFORMANCE PER RELATION TYPE
Tables 3 and 4 report results (hits@10) for each relation and include the relation type and known confounding inﬂuences: percentage of relation instances in the training and test sets (approximately equal), number of instances in the test set, Krackhardt hierarchy score (see Appx.
E)
(Krackhardt, 2014; Balaževi´c et al., 2019a) and maximum and average shortest path between any two related nodes.
A further confounding effect is dependence between relations.
Balaževi´c et al.
(2019b) and Lacroix et al.
(2018) show that constraining the rank of relation representations beneﬁts datasets with many relations (particularly when the number of instances per relation is low) due to multi-task learning , which is expected to beneﬁt TuckER on the NELL-995 dataset (200 relations).
Note that all models have a comparable number of free parameters.
P1: As predicted, all models tend to perform best at type R relations, with a clear performance gap to other relation types.
Also, performance on type S relations appears higher in general than type
C. Additive-only models (TransE, MuRE I ) perform most poorly on average, in line with prediction since all relation types involve a relatedness component.
They achieve their best results on type R relations, where the relation vector can be zero/small.
Multiplicative-only DistMult performs well, sometimes best, on type R relations, ﬁtting expectation as it can fully represent those relations and has no additional parameters that may overﬁt to noise (which may explain where MuRE performs slightly worse).
As expected, MuRE, performs best on average (particularly on WN18RR), and most strongly on S and C type relations that require both multiplicative and additive components.
The comparable performance of TuckER on NELL-995 is explained by its ability for multi-task learning.
Other unexpected results also closely align with confounding factors, e.g. that all models perform poorly on the hypernym relation, despite it having type S and a relative abundance of training data (40% of all instances), might be explained by its hierarchical nature (Khs ≈ 1 and long paths).
The same may explain the reduced performance on type R relations also_see and agentcollaborateswithagent .
As found previously, none of the models considered are well suited to modelling hierarchical structures (Balaževi´c et al., 2019a).
We also note that the percentage of training instances of a relation does not seem to correlate with its performance, as might typically be expected.
P2/P3: Table 5 shows the symmetry score ( ∈ [-1, 1] indicating perfect anti-symmetry to symmetry; see Appx.
F) for the relation matrix of TuckER and the norm of relation vectors of TransE,
MuRE I and MuRE on the WN18RR dataset.
As expected, type R relations have high symmetry, whereas both other relation types have lower scores, ﬁtting the expectation that TuckER compensates for having no additive component.
All additive models learn relation vectors of a noticeably lower norm for type R relations (where, in the extreme, no additive component is required) than for types S and C.
[Table]
1.88 member_of_domain_usage
C 0.
1.88 member_of_domain_region
C 0.
2.11 synset_domain_topic_of
C 0.
1.52 member_meronym
C 0.
1.97 has_part
C 0.
1.25 hypernym
S 0.
[Figure]
P4: Fig 2 shows eigenvalue magnitudes (scaled relative to their largest and ordered) of the relationspeciﬁc matrices R of MuRE, labelled by relation type.
Predicted to reﬂect strength of a relation’s relatedness, they should be highest for type R relations, as observed.
For relation types S and C the proﬁles are more varied, ﬁtting the expectation that the relatedness of those types has greater variability in both choice and size of S , i.e. in the nature and strength of relatedness.
In summary, the results support all predictions made based on the assumption that knowledge graph models beneﬁt from the same latent semantic structure as word embeddings and the relation conditions theoretically derived from them.
Our analysis identiﬁes the best performing model per relation type: multiplicative-only DistMult for type R, additive-multiplicative MuRE for types S/C; providing a basis for dataset-dependent model selection .
The per-relation insight into where models perform poorly, e.g. on hierarchical or type C relations, can also be used to aid and direct future model design.
4. K NOWLEDGE GRAPH MODEL PREDICTIONS
Even though in practice we want to know whether a particular triple is true or false, such independent predictions are not commonly reported or evaluated.
Despite many recent link prediction models being able to independently predict the truth of each triple, it is common practice to report rankingbased metrics, e.g. mean reciprocal rank, hits@ k , which compare the prediction of a test triple to those of all evaluation triples (see S4).
Not only is this computationally costly, the evaluation is ﬂawed if entities are related to more than k others and does not evaluate a model’s ability to independently predict whether “ a is related to b ”.
We address this by considering actual model predictions.
Since for each relation there are n 2 e possible entity-entity relationships, we sub-sample by computing predictions for all ( e s , r, e o ) triples only for each e s , r pair seen in the test set.
We split positive predictions ( σ ( φ ( e s , r, e o )) > 0 . 5 ) between (i) training and test/validation instances (known truths); and (ii) other , the truth of which is not known.
Per relation, we then compute accuracy over the training instances (train) and the test/validation instances (test); and the average number of other truths predicted per e s , r pair.
Table 6 shows results for MuRE I , DistMult, TuckER and MuRE.
All
[Table]
[Figure]
models achieve almost perfect training accuracy.
The additive-multiplicative MuRE gives best test set performance, followed (surprisingly) closely by MuRE I , with multiplicative models (DistMult and TuckER) performing poorly on all but type R relations.
Analysing a sample of “other” positive predictions for a relation of each type (see Appx.
G), we estimate that TuckER is relatively accurate but pessimistic ( ∼ 0.3 correct of the 0.5 predictions ≈ 60% ), MuRE
I is optimistic but inaccurate ( ∼ 2.3 of 7.5 ≈ 31% ), whereas MuRE is both optimistic and accurate ( ∼ 1.1 of 1.5 ≈ 73% ).
Fig 3 shows histograms of MuRE prediction probabilities for the same sample relations, split by known truths (training and test/validation) and other instances.
There is a clear distinction between relation types: for type R, most train and test triples are classiﬁed correctly with high conﬁdence; for types S and C, an increasing majority of incorrect test predictions are far below the decision boundary, i.e. the model is conﬁdently incorrect.
For relation types where the model is less accurate, fewer positive predictions are made overall and the prediction distribution is more peaked towards zero.
This analysis probes further into the difﬁculty models have representing type S/C relations.
Given the additional insight provided and the beneﬁt of standalone predictions, we recommend the inclusion of predictive performance in future link prediction work.
C ONCLUSION
Many models learn low-rank representations for knowledge graph link prediction, yet little is known about the latent structure they learn.
We build on recent understanding of PMI-based word embeddings to theoretically establish what a relation representation must achieve to map a word embedding to those it is related to for the relations of knowledge graphs (relation conditions). Such conditions partition relations into three types and also provide a framework to assess loss functions of knowledge graph models.
Any model that satisﬁes a relation’s conditions can represent it if its entity embeddings are set to PMI-based word embeddings, i.e. a solution is known to exist. Whilst knowledge graph models do not learn the parameters of word embeddings, we show that the better a model’s architecture satisﬁes a relation’s conditions, the better its link prediction performance, ﬁtting the premise that similar latent structure is exploited.
Overall, we extend previous understanding of how semantic relations are encoded in relationships between PMI-based word embeddings – generalising from a limited set, e.g. similarity and analogy; we demonstrate commonality between the latent structure learned by PMI-based word embeddings (e.g. W2V) and knowledge graph representation models; and we provide novel insight into knowledge graph models by evaluating their predictive performance.
A C ATEGORISING W ORD N ET RELATIONS
Table 7 describes how each WN18RR relation was assigned to its respective category.
[Table]
B A NALYSING NELL-995 RELATIONS
NELL-995: We categorise a random subsample of 12 relations from the NELL-995 dataset (Xiong et al., 2017) containing 75,492 entities and 200 relations (a subset of NELL (Carlson et al., 2010)), which span our identiﬁed relation types (see Table 8).
Explanation for the relation category assignment is shown in Table
[Table] [Table]
[Table]
C S PLITTING THE NELL-995 DATASET
The test set of NELL-995 created by Xiong et al.
(2017) contains only 10 out of 200 relations present in the training set.
To ensure a fair representation of all training set relations in the validation and test sets, we create new validation and test set splits by combining the initial validation and test sets with the training set and randomly selecting 20,000 triples each from the combined dataset.
D I MPLEMENTATION DETAILS
All algorithms are re-implemented in PyTorch with the Adam optimizer (Kingma & Ba, 2015) that minimises binary cross-entropy loss, using hyper-parameters that work well for all models (learning rate: 0.001, batch size: 128, number of negative samples: 50).
Entity and relation embedding dimensionality is set to d e = d r =200 for all models except TuckER, for which d r =30 (Balaževi´c et al., 2019b).
E K RACKHARDT HIERARCHY SCORE
The Krackhardt hierarchy score measures the proportion of node pairs ( x, y ) where there exists a directed path x
→ y , but not y
→ x ; and it takes a value of one for all directed acyclic graphs, and zero for cycles and cliques (Krackhardt, 2014; Balaževi´c et al., 2019a).
Let M ∈ R n × n be the binary reachability matrix of a directed graph G with n nodes, with M i,j =
if there exists a directed path from node i to node j and 0 otherwise.
The Krackhardt hierarchy score of G is deﬁned as:
[Equation]
F S YMMETRY SCORE
The symmetry score ∈ [ − 1 , 1] (Hubert & Baker, 1979) for a relation matrix
R ∈ R d e × d e is deﬁned as:
[Equation]
where 1 indicates a symmetric and -1 an anti-symmetric matrix.
G “O THER ” PREDICTED FACTS
Tables 10 to 13 shows a sample of the unknown triples (i.e. those formed using the WN18RR entities and relations, but not present in the dataset) for the derivationally_related form (R), instance_hypernym (S) and synset_domain_topic_of (C) relations at a range of probabilities ( σ ( φ ( e s , r, e o ))
≈ { 0 . 4 , 0 .
8 , 1 } ), as predicted by each model.
True triples are indicated in bold; instances where a model predicts an entity is related to itself are indicated in blue.
[Table]
[Table] [Table]
” f ac
[Table]
M
[Table] [Table]
on (
[Table]
yp e )
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
s ,
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
o )
[Table]
f
[Table]
m
[Table] [Table]
qua
[Table]
NN
[Table] [Table]
VB
[Table] [Table]
apo
[Table]
NN
[Table] [Table]
t
[Table]
NN
[Table] [Table]
NN
[Table] [Table]
NN
[Table]
[Table] [Table]
NN
[Table] [Table]
llit e
[Table]
NN _
[Table] [Table]
ad
[Table] [Table]
i
[Table] [Table]
[Table] [Table]
[Table] [Table]
k
[Table]
VB
[Table] [Table]
NN
[Table]
) ( ﬂaun t
[Table]
NN _
[Table]
, s ho
[Table]
NN
[Table] [Table]
nu
[Table]
l
[Table]
NN
[Table] [Table]
r a w _ c l o s e _ V
B _3 ) ( s t ee r _ VB _1 , s t ee r _
VB _1 ) ( c on ti nu e _ VB _10 , p r o cee d_
VB _1 ) ( e x t r apo l a t e _
VB _3 , s y n t
h e s i s _ NN _ ) ( t r a i n _ VB _3 , t r a i n i n g_ NN _1 ) ( s o r t
ou t _ VB _1 , s o r t _ou t _
VB _1 ) ( s uppo r t _ VB _6 , d e f e nd_
VB _5 ) ( s t r a t e g
i s t _
NN _1 , m a c h i na ti on _ NN _1 ) ( s c r a t c h _ VB _3 , s k i n _ s e n s a ti o n _
NN _1 ) ( m a ke
f u ll _
VB _1 , m a ke _ f u ll _ VB _1 ) ( s hu tt e r _ NN _1 , ﬁ ll _up_
VB _3 ) ( c r u s h _ VB _4 , g r i nd _ VB _2 ) ( s c h e m e _ NN _5 , s c
h
e m a ti z a ti on _
NN _1 ) ( u tili z
e _ VB _1 , u tili z e _
VB _1 ) ( y a w n i ng_ NN _1 , pa t e ll a r _ r e ﬂ ex _ NN _1 ) ( s p i ke _ VB _5 , s t e ad y _ VB _2 ) ( o r da i n_ VB _3 , ve s t _ VB _1 )
( g e o l og y _ NN _1 , g e o l og y _
NN _1 ) ( y a w _
NN _1 , s p
i r a l _
VB _1 ) ( li ck i ng_ NN _1 , v anqu i s h_
VB _1 ) ( li e _
VB _1 , f r on t _ e nd _ NN _1 ) ( z oo l og y _ NN _2 , z oo l og y _
NN _2 ) ( s t r a t u m _ NN _2 , s o c
i a l _g r oup
_ NN _1 ) ( s y n t
h e ti c a l _
JJ_1 , s y n t
h e s i
z
e r _ NN _2 ) ( t r e ad _ NN _1 , s t e p_
NN _9 ) ( u r ano l og y _
NN _1 , u r ano l og y _ NN _1 ) ( du e l _
VB _1 , s c u f ﬂ e _
NN _3 ) ( r e a li z a ti on_
NN _2 , e m bod
i m e n t _
NN _3 ) ( r e g i s t e r _ NN _3 , ﬁ l e _a w a y _ VB _ 1 ) ( t r a ve l _ VB _1 , t
r a ve l _ VB _1 ) i n s t a n ce _ hyp e
r ny m ( S ) ( t ho m a s _aqu i na s _
NN _1 , m a r t y r _
NN _2 ) ( t a i w a n _ NN _1 , a s i a n _ n a ti o n _
NN _1 ) ( p r op h
e t s _
NN _1 , go s p e l s _
NN _ 1 ) ( h e l s i n k i _ NN _1 , u r ba n _ ce n t e r _
NN _1 ) ( v o l c ano_
i s l and s _
NN _1 , v o l c ano _ NN _2 ) ( s t .
_g r e go r y _
o f _ n .
NN _1 , c a n o n i z a ti o n _
NN _1 ) ( m a l c o l m _ x _
NN _1 , pa ss i ve _ r e
s i s t e r _ NN _1 ) ( m annh e i m _
NN _1 , s t o ck ad e _ NN _2 ) ( c ap e _ ho r n_
NN _1 , u r ban_ ce n t e r _
NN _1 ) ( s t .
_g r e go r y _
o f _n .
NN _1 , s a i n t _
VB _2 ) ( t a i w an_ NN _1 , na ti ona l _ c ap it a l _
NN _1 ) ( n i ppon_
NN _1 , n i ppon _ NN _1 ) ( b e r g e n_ NN _1 , na ti ona l _ c ap it a l _
NN _1 ) ( m cc o r
m
i ck _ NN _1 , ﬁnd_ VB _8 )
( t r u t h_
NN _5 , abo liti on i s m _
NN _1 ) ( v i c t
o r _hugo _ NN _1 , no ve l _
NN _1 ) ( m a rs ha ll _
NN _2 , g e n e r a l s h i p_
NN _1 ) ( s t .
_g r e go r y _ i _ NN _1 , b i s h op _ NN _ )
( t h o m a s _aq u i n a s _
NN _1 , s a i n t _
VB _2 ) ( r e g i o m on t anu s _
NN _1 , u r ano l og y _ NN _1 ) ( n a n s e n _ NN _1 , ve n t u r e r _ NN _ )
( r i c ha r d_bu ck
m i n s t e r _ f .
NN _1 , t ec hno l og i c a l _
JJ _2 ) ( ce n t r a l _a m e r i c a_
NN _1 , s .
_a m .
_ na ti on _ NN _1 ) ( p r op h
e t s _
NN _1 , book_
NN _ ) ( w i s c on s i n_
NN _2 , s t a t e _
c
ap it a l _
NN _1 ) ( t ho m a s _aqu i na s _
NN _1 , a r c hb i s hop
_ NN _1 ) ( d e _ m ill e _ NN _1 , dan ce _
VB _1 ) ( t ho m a s _aqu i na s _
NN _1 , c hu r c h_ f a t
h e r _
NN _1 ) ( p r u ss i a_ NN _1 , s t o ck ad e _
NN _2 ) ( m a rs
h
a ll _
NN _2 , g e n e r a l _ o f ﬁ ce r _ NN _1 ) ( s t .
_g r e go r y _ i _ NN _1 , apo s tl e _
NN _3 ) ( w ood y _gu t h r i e _
NN _1 , m i n s t r e l _ VB _1 ) ( d e _ m ill e _
NN _1 , da n c i n g - m a s t
e r _ NN _ ) ( n e w m an_ NN _2 , p r
i m a t
e s h i p
_ NN _1 ) ( f e r til e _ c r e
s ce n t _
NN _1 , a s i an_ na ti on _ NN _1 ) ( ce n t r a l _a m
e
r
i c a_
NN _1 , c .
_a m .
_ na ti on _ NN _1 ) ( a e g e an_ s e a_
NN _1 , a e g e an _
i s l and _ NN _1 ) ( t ho m a s _ t h e _apo s tl e _ NN _1 , s an c tif y _
VB _1 ) ( r ob e r t _o w e n _ NN _1 , i ndu s t r y _ NN _1 ) ( a e g e an_ s e a_
NN _1 , i s l and_ NN _1 ) s yn s e t _ do m a i n_ t op i c
_o f ( C ) ( w r it e _
VB _8 , t ap e _ VB _3 ) ( d r a w _ VB _6 , c
r e a ti ve _p e rs on_ NN _1 ) ( li b e l _
NN _1 , s u ll y _
VB _3 ) ( li b e l _
NN _1 , d i s pa r ag e m e n t _ NN _1 ) ( i n t r o ve r t _
NN _1 , s c
i e n ti ﬁ c _d
i s c
i p li n e _
NN _1 ) ( s u bo r d e r _
NN _1 , t axo n o m i
c _
g r o u p_ NN _1 ) ( r e l a ti on s h
i p_
NN _4 , r e l a ti on s h
i p_
NN _4 ) ( r o ll - o n _
r
o llo ff _
NN _1 , t r a n s po r t _ NN _1 ) ( li b e l _
NN _1 , s l u r _ NN _2 ) ( d r a w _ VB _6 , d r a w _
VB _6 ) ( e t y m o l
og
i
z i n g_ NN _1 , li n g u i s ti
c s _ NN _1 ) ( p r a n ce _ VB _4 , e q u e s t
r
i a n _ s po r t _ NN _1 ) ( e t y m o
l
og i
z
i ng_ NN _1 , l a w _ NN _1 ) ( ﬁ rs t _ s a c k e r _
NN _1 , ba ll ga m e _
NN _2 ) ( t u r n _
VB _12 , c u lti v a ti o n _
NN _ 2 ) ( li b e l _
NN _1 , t r ad u
ce m e n t _
NN _1 ) ( t e m p l e _ NN _4 , p l a ce _o f _ w o rs h
i p_ NN _1 ) ( a l c h e m i z e _
VB _1 , m od if y _
VB _3 ) ( b r y nh il d_ NN _1 , m y t
ho l og i z
e _
VB _2 ) ( s e ll _
VB _1 , s e lli n g_
NN _1 ) ( t r i a l _ i m p r e ss i on_
NN _1 , p r oo f _
VB _1 ) ( s e r m on_ NN _1 , s e r m on_
NN _1 ) ( b r y nh il d_ NN _1 , m y t h _
NN _1 ) ( t r o t _
VB _2 , r i d e _ho rs e ba
ck
_ VB _1 ) ( f r i e nd _o f _ t h e
_ c ou r t _
NN _1 , w a r _ m a c h i n e _
N N _1 ) ( s a i n t _
VB _2 , c a t h o li c _ c hu r c h _
NN _1 ) ( a ss i s t _
NN _2 , a m .
_ f oo t ba ll _ ga m e _
NN _1 ) ( p r an ce _ VB _4 , r i d e _ho
rs e ba ck
_ VB _1 )
( m u lti v .
_a n a l
y s i s _
NN _1 , app li e d_ m a t h _
NN _ 1 ) ( m a l e _
JJ _1 , m a s c u li n e _ JJ _2 ) ( m it z v ah_ NN _2 , hu
m an_a
c ti v it y _
NN _1 ) ( ga ll op_ VB _1 , r i d e _ho rs
e ba ck
_ VB _1 )
( s e ll _
VB _1 , t r a n s a c ti o n _
NN _1 ) ( ﬁ r e _
VB _3 , z oo l og y _ NN _2 ) ( d r i ve _ NN _ 12 , d r i ve _ VB _8 ) ( b r y nh il d_ NN _1 , m y t
h
o l og y _
NN _2 ) ( d r a w _ VB _6 , r e p r e s e n t _ VB _9 ) ( s e ll _
VB _1 , s e ll _ VB _1 ) ( r e l a ti o n s h i p _ NN _4 , b i o l og y _ NN _1 ) ( d r i ve _ NN _ 12 , bad m i n t o n _ NN _ )
T a b l e 11 : “
O t h e r ” f ac t s a s p
r
e
d i c t e d by D i s t
M u lt .
R e l a ti on ( T yp e ) σ ( φ ( e s , r , e o ) )
≈ 0 .σ
( φ ( e s , r , e o ) )
≈ 0 .σ
( φ ( e s , r , e o ) )
≈ 0 .σ
( φ ( e s , r , e o ) )
≈d e r i v a ti on a ll y_ r e l a t e d _ f o r m ( R ) ( s t ag
_ VB _3 ,
un d e r c o ve r _ w o r k_ NN _1 ) ( d i s h_
NN _2 , s t e w _
NN _2 ) ( s h r
i n k _ NN _1 , p e do l og y _
NN _1 ) ( a llit e r a t e _
VB _ 1 , ve rs
i
ﬁ
e r _ NN _1 ) ( p r i n t _
VB _4 , p u b li s h e r _ NN _2 ) ( e xpo s e _ VB _3 , s h o w _
NN _1 ) ( ﬁn i s h _ VB _6 ,
ﬁn
i s h
i n g_ NN _2 ) ( g e o l og y _
NN _ 1 , s t r u c t u r a l _ J J_ 5 ) ( c r i e r _
NN _3 , p it c
h m an_ NN _2 ) ( s y s t
e m _ NN _9 , o r d e r li n e ss _ NN _1 ) ( p l a y _
VB _26 , p l a y i n g_ NN _3 ) ( r e s ec t _ VB _ 1 , a m p u t a ti o n _
NN _ 2 ) ( p l a y _
VB _26 , t u r n _
NN _ ) ( s p r e ad_ NN _4 , s t
r e w _ VB _1 ) ( ce n t r a li z a ti o n _ NN _1 , un it e _ VB _6 ) ( n u t r iti o n_ NN _ 3 , m an _ NN _4 ) ( c oun t _ VB _4 , r ec it e _ VB _2 ) ( t a ke _do w n _ VB _2 , pu t _ VB _2 ) ( ex i s t
e n ce _ NN _1 , li v i ng_ NN _3 ) ( s a i n t _
NN _3 , s a n c ti f y _ VB _ ) ( v i v i d n e ss _
NN _2 , i m b u e _
VB _3 ) ( w r e s tl e _ VB _4 , w r e s tl
e r _
NN _1 ) ( m ou t h
_ VB _3 , s a ss i ng_ NN _1 ) ( r i g h t _ ﬁ e l d e r _ NN _1 , l e ft ﬁ e l d_ NN _ 1 ) ( s e a_ m e w _
NN _1 , l a r u s _
NN _1 ) ( a u t o t r .
_o r ga
n i s m _
NN _1 , e p i p
h
y ti c _JJ_ 1 ) ( c on s t
e
ll a ti on _ NN _2 , s t a r _
NN _1 ) ( li s t _
VB _ 4 , s l op e _
NN _ 2 ) ( a l k a li _ NN _2 , a c i d if y _ VB _2 )
( d u e l _
VB _1 , s l u g f e s t _
NN _1 ) ( p r i n t _
VB _4 , p u b li s h i n g_
h
o u s e _
NN _ 1 ) ( li e u t e n an cy _ NN _1 , c ap t a i n _ NN _) ( s e e _
VB _17 , und e rs t and_ VB _2 ) ( v o c a l _ NN _2 , r o ck _ s t a r _ NN _1 ) ( p u zz l e _ VB _2 , s ec r e t _
NN _3 ) ( t r e a d _ NN _ 1 , s t e p _ VB _ 7 ) ( s hun
_ VB _1 , h e dg i ng_ NN _2 ) ( s m e lli n g_
NN _1 , s ce n t _
VB _1 ) ( u r ano l og y _
NN _1 , tt _ NN _ ) ( e x e n t e r a ti o n _ NN _ 1 , e nu c l e a t e _ VB _ ) i n s t a n ce _ hyp e r ny m ( S ) ( w i s c on s i n_
NN _2 , u r ban _ ce n t e r _ NN _ ) ( m i ss i ss i
pp i
_ r i ve r _
NN _1 , a m e r i c an_ s t a t e _ NN _ ) ( d ee p _ s ou t h
_ NN _1 , s ou t h_
NN _1 ) ( h e l s i n k i _ NN _ 1 , u r b a n _ ce n t e r _
NN _1 ) ( m a rs
h
a ll _
NN _2 , li e u t e n a n t _g e n e r a l _
NN _1 ) ( r . _ e .
_ b y r d_
NN _1 , c
o mm i ss i on e d_ o f
ﬁ
ce
r _ NN _ ) ( c ap it a l _o f _ ga m b i a _ NN _1 , u r ba n _ ce n t e r _
NN _1 ) ( t h e _ na z a r e n e _ NN _1 , s a ve _ VB _7 ) ( ab i d j an_ NN _1 , c
o t e _d ’ i v o i r e _
NN _1 ) ( kob e nh a v n _
NN _1 , u r ba n _ ce n t e r _
NN _1 ) ( s ou t h _ w e
s t _ a f r i c a _ NN _1 , a f r i c a _ NN _ ) ( i r i s h
_ c a p it
a l _ NN _1 , u r ba n _ ce n t e r _
NN _ 1 ) ( w o r l d _ w a r _ i _ NN _1 , u r ban _ ce n t e r _ NN _ ) ( t h e _ ga m b i a _ NN _1 , a f r i c a _ NN _ ) ( b r and e nbu r g _ NN _1 , u r ban _ ce n t e r _ N N _ ) ( r .
_ e .
_ b y r d_
NN _ 1 , i n s p ec t
o r _g e n e r a l _
NN _1 ) ( s t .
_ pau l _
NN _2 , ev ang e li s t _
NN _2 ) ( ti r i
c h
m i r _
NN _1 , u r ban _ ce n t e r _ NN _ ) ( s i e rr a_n ev ada_ NN _1 , u r ban_ ce n t e r _
NN _ 1 ) ( r . _ e .
_ b y r d_
NN _ 1 , c h i e f _
o f _ s t a ff _ NN _ 1 ) ( d ee p _ s ou t h
_ NN _1 , u r ban _ c e n t e r _ NN _ ) ( r .
_ e .
_ b y r d_
NN _1 , m ilit a r y _ad v i s o r _ NN _1 ) ( m a l c o l m _x_
NN _1 , e m a n c i pa ti o n i s t
_ NN _ 1 ) ( ce n t r a l _a m
e
r
i c a _
NN _1 , c .
_a m .
_ na ti o n_
NN _ 1 ) ( nup t s e _ NN _1 , u r ban _ c e n t
e r _ NN _ ) ( r .
_ e .
_ b y r d_
NN _1 , a i d e - d e - c a m p _ NN _ ) ( no r t h_ p l a tt e _ r
i ve r _
NN _1 , u r ban _ ce n t e r _ NN _ ) ( m a l c o l m _ x _
NN _1 , e n v i r on m e n t a li s t _ NN _ 1 ) ( ti c i no _ NN _1 , u r ban _ c e n t e r _ NN _ ) ( t a m pa_ba y _
NN _1 , u r ban_
ce n t e r _
NN _1 ) ( o s l o_
NN _1 , u r ba n _ ce n t e r _
NN _1 ) ( t h e _ n a z a r e n e _ NN _ 1 , c h r
i s ti a n _ J J_ 1 ) ( a e g e an_ s e a _ NN _1 , a e g e an _
i s l and _ NN _ ) ( ti d e w a t e r _ r
e
g
i on_
NN _1 , s ou t h_
NN _1 ) ( z a i r e _ r i ve r _
NN _1 , u r ban _ ce n t e r _ N N _ ) ( t h o m a s _aqu i na s _
NN _ 1 , c hu r c h_ f a t
h e r _
NN _1 ) ( c o w p e n s _
NN _1 , w a r _o f _a m .
_ i nd .
_ NN _1 ) ( r .
_ e .
_ b y r d_
NN _1 , e x ec u ti ve _o f ﬁ ce r _
NN _1 ) ( t r an s y l v an i an_a l p s _ NN _1 , u r ban _ ce n t e r _ NN _ ) ( t h e _ na z a r e n e _ NN _1 , e l _ n i n o_
NN _2 ) s yn s e t _ do m a i n_ t op i
c _o f ( C ) ( li m it a ti on_
NN _4 , t r a mm e l _
VB _2 ) ( r o ll - o n _ r o llo ff _
NN _1 , t r a n s po r t _ NN _1 ) ( e t y m o l og i
z i n g_
NN _1 , e xp l a n a ti o n _
NN _1 ) ( ﬂ a t _
JJ _ 5 , m a tt e _
NN _2 ) ( li gh t _ c o l on e l _ NN _1 , c
o l on e l _
NN _1 ) ( h
i
z
b_ u t - t
ah r i r _
NN _1 , a s i a_
NN _1 ) (f e rr y _ VB _3 , t r a ve l _ VB _ ) ( e t y m o l o g
i z
i ng _ NN _1 , d e r i ve _
VB _ 3 ) ( nu rs e _ VB _1 , nu rs i n g_
NN _1 ) ( s l u gg e r _ NN _1 , s o f t ba ll _ga m e _
NN _1 ) ( pub li c _p r o s ec u t
o r _ NN _1 , p r o s ec
u t e _
VB _2 ) ( h o l e _ o u t _
VB _ 1 , h o l e _
NN _ 3 ) ( s e r m on_ NN _1 , p r oph
e s y _
VB _2 ) ( s e r m on_ NN _1 , s e r m on i z
e _
VB _1 ) ( a l c h
e
m i z e _
VB _1 , m od if y _
VB _3 ) ( r e l a ti o n s h i p _ NN _ 4 , r e l a ti o n _
NN _ 1 ) ( li b e l _
NN _1 , p r a c ti ce _
o f _
l a w _ NN _1 ) ( d r a w _ VB _6 , d r a w e r _ NN _3 ) ( li b e l _
NN _1 , li b e l _ VB _1 ) ( d r i ve _ NN _1 2 , ba
d m i n t o n _
NN _ 1 ) ( s l u gg e r _ NN _1 , ba s e ba ll _p l a ye r _ NN _1 ) ( t u r n _
VB _12 , p l o w _
NN _1 ) ( t u r n_
VB _12 , till _ VB _1 ) ( e t y m o l o g i z i n g_ NN _1 , e t y m o l o g i z
e _ VB _2 ) ( r n a_
NN _1 , c
h e m i s t
r
y
NN _1 ) ( a ss i s t _
NN _2 , s o f t ba ll _ga m e _
NN _1 ) ( h it _
NN _1 , h it _ VB _1 ) ( m a t
r
i x _ a l g e b r a_ NN _1 , d i a g on a li z a ti on _
NN _ 1 ) ( m e t r if y _ VB _1 , v e rs if y _
VB _1 ) ( c o un c il _
NN _2 , a ss e m b l y _ NN _4 ) ( ﬁ r e _
VB _3 , ﬂ a m i n g _ NN _1 ) ( c ab i n e t w o r k _ NN _ 2 , w o od w o r k i n g_
NN _ 1 ) ( t r i a l _ i m p r e ss i on_
NN _1 , pub li s h_
VB _1 ) ( t
h r o u g h p u t _
NN _1 , t u r n o u t _
NN _4 ) ( r i ng _ NN _4 , c
h e m i c a l _
c ha i n_ NN _1 ) ( c a b i n e t w o r k _ NN _2 , b o tt o m _ VB _ 1 ) ( t u r n _
VB _12 , p l o w m a n _
NN _1 ) ( c r e a m _ VB _1 , c
r e a m _
NN _2 ) ( li b i d i n a l _ e n e r g y _
NN _1 , c h a r g e _
NN _ 9 ) ( c ab i n e t w o r k _ NN _ 2 , up ho l s t e r _ VB _1 )
T a b l e 12 : “
O t h e r ” f ac t s a s p
r
e
d i c t e d by T u
c k E
R .
R e l a ti on ( T yp e ) σ ( φ ( e s , r , e o ) )
≈ 0 .σ
( φ ( e s , r , e o ) )
≈ 0 .σ
( φ ( e s , r , e o ) )
≈ 0 .σ
[Table]
( e s , r , e o ) )
[Table]
d e r i v a ti on a ll y_ r e l a t e d _ f o r m ( R ) ( t y m pan
i s t _
NN _1 , gong_
NN _2 ) ( m a s h _ NN _ 2 , m ill _ VB _ ) ( t a ke _ c
han
ce
s _ VB _1 ,
ve n t u r e _
NN _1 ) (
[Table]
t
[Table]
e
[Table]
_ NN _
[Table] [Table]
NN _
[Table]
) ( i n d i c a ti o n _
NN _1 , s i g n a li z e _ VB _ ) ( w a l k _
VB _9 , z
i mm e r _
f
r a m e _
NN _1 ) ( s hu tt e r _ NN _1 ,
[Table]
_up_
VB _3 ) (
[Table]
na m
[Table]
t _
NN _1 ,
[Table]
y
[Table]
NN _1 ) ( t u r n _ o ve r _
VB _3 , r o t a ti o n _ NN _3 ) ( u s e _
VB _5 , u tilit y _
NN _2 ) ( e x it _
NN _3 , l e a ve _
VB _1 ) ( l o ve _
VB _3 , l o ve r _
NN _2 ) ( da t e _
VB _5 , g e
o l og i c a l _da ti n g_
NN _1 ) ( m u s i c a l _
i n s t r u m e n t _ NN _1 , w r it e _ VB _6 ) ( t r e m b l e r _
NN _1 , v i b r a t e _
VB _1 ) ( s nu gg l e _
NN _1 , s q u ee z e _ VB _ ) ( s e t _
VB _23 , e m b l e m _ NN _ ) ( li n i n g _ NN _ 3 , w r ap_ u p_
VB _1 ) ( m o ti v a t o r _
NN _1 , t r i p_ VB _4 ) ( d e pa r t e d_
NN _1 , d i e _ VB _2 ) ( t y r o_
NN _1 , s t a r t _
VB _5 ) ( s c r
ap
_ VB _2 , s t r u gg l e _ NN _2 ) ( s u ppo r t _ VB _6 , i n do rs e m e n t _
NN _1 ) ( po s iti o n _ VB _1 , p l a ce m e n t _
NN _1 ) ( i d e n ti ﬁ c a ti o n _ NN _1 , n a m e _ VB _5 ) ( t ap e _
VB _3 , t ap e _ r ec o r d e r _
NN _1 ) (f e d e r a t e _
VB _2 , c o n f e d e r a ti o n _ NN _ 1 ) ( r e p e n t an t _ JJ _1 , r e p e n t an t _ JJ _1 ) ( s t abb
e
r _ NN _1 , t h r u s t _
VB _5 ) ( v i n d i c a t e _
VB _2 , j u s ti ﬁ c
a
ti o n _
NN _2 ) ( t ak e _ o ve r _
VB _6 , r e t u r n _
NN _ 7 ) ( t r e ad _ NN _ 1 , s t e p _ VB _ ) ( j u s ti ﬁ c
a
ti on _ NN _1 , apo l og y _ NN _ ) ( l e a c h i n g_
NN _1 , p e r c o l a t e _ VB _3 ) ( w a it _ o n _ VB _1 , s u ppo r t e r _
NN _ 3 ) ( s t
o ck
i s t _ NN _1 , s t
o ck
i s t _
NN _1 ) ( m a nu f a c t
u
r
e
_ VB _1 , p r ev a r
i c a ti o n _
NN _1 ) ( s y n c h r o n i
z
e _ VB _2 , s y n c
h r o s c op e
_ NN _1 ) ( d e n o t e _ VB _3 , p r o m u l ga ti o n _
NN _1 ) ( ph il an t h r op i s t _ NN _1 , ph il an t h r op i s t _
NN _1 ) i n s t a n ce _ hyp e
r ny m ( S ) ( d ee p _
s ou t h
_ NN _1 , s ou t h_
NN _1 ) ( t ho m a s _aqu i na s _
NN _1 , b i s hop _ NN _ ) ( c o w p e n s _
NN _1 , s i e g e _
NN _1 ) ( r . _ e .
_ b y r d_
NN _1 , s i e g e _
NN _1 ) ( s t .
_ pau l _
NN _2 , o r gan i s t _
NN _1 ) ( i r i s h
c
ap it
a l _ NN _1 , u r ba n _ ce n t e r _
NN _1 ) ( m cc o r
m
i ck _ NN _1 , a r m s _ m anu f a c t u r e r _
NN _1 ) ( s h a w _ NN _ 3 , w o m e n ’ s _ r
i g
h ti s t _ NN _1 ) ( h e l s i n k i _ NN _1 , u r ba n _ ce n t e r _
NN _1 ) ( t h o m a s _ t h e _apo s tl e _ NN _1 , apo s tl e _
NN _2 ) ( t ho m a s _ t h e _apo s tl e _ NN _1 , ev a ng e li s t _ NN _ ) ( a e g e an_ s e a_
NN _1 , a e g e an _
i s l and _ NN _ ) ( m a l c o l m _x_
NN _1 , e m a n c i pa ti o n i s t
_ NN _ 1 ) ( s t .
_ pau l _
NN _2 , apo s tl e _
NN _3 ) ( m cc o r
m
i
c k_
NN _1 , t ec hn
o l
og i s t _
NN _1 ) ( t ho m a s _aqu i na s _
NN _1 , c hu r c h_ f a t
h e r _
NN _1 ) ( t ho m a s _ t h e _apo s tl e _ NN _1 , c hu r c h_
f a t
h e r _ N
N _1 ) ( m cc
o r m i ck _ NN _1 , pa i n t e r _
NN _1 ) ( s t .
_g r e go r y _ i _ NN _ 1 , c hu r c h _
f a t h e r _
NN _1 ) ( s t .
_g r e go r y _
o f _ n .
NN _1 , s e r m o n
i
z
e r _
NN _1 ) ( t ho m a s _ t h e _apo s tl e _ NN _1 , t r og l od y t e _ NN _ ) ( r ob e r t _o w e n _ NN _1 , m o v
i e _ m a ke r _
NN _1 ) ( m cc o r
m
i ck _ NN _1 , e l ec t r i c a l _ e ng i n ee r _
NN _1 ) ( t h e r e s a_ NN _1 , m on k _
NN _1 ) ( m i ss i ss i pp
i
_ r i ve r _
NN _1 , a m e r i c an_ s t a t e _ NN _ ) ( s t .
_ pau l _
NN _2 , ph il o s oph e r _
NN _1 ) ( i bn - r
o s hd_
NN _1 , p e dagogu e _ NN _1 ) s yn s e t _ do m a i n_ t op i
c _o f ( C ) ( r o ll - o n _ r
o llo ff _
NN _1 , m o t
o r c a r _ NN _1 ) ( d r i ve _ NN _ 12 , bad m i n t o n _ NN _ 1 ) ( li b e l _
NN _1 , l e g i s l a t u r e _ NN _1 ) ( r o ll - o n _ r
o llo ff _
NN _1 , pa ss e n g e r _ ve h
i c l e _
N N _1 )
[Table]
NN
[Table]
_ NN _1 ) ( li g h ti n g_ NN _4 , s e t _ o n _ ﬁ r e _ VB _1 ) ( s nu f ﬂ e _ VB _3 , w h i n e _
NN _1 ) ( m ag n it u d e _
NN _1 , t a ll _JJ_ 1 ) ( v i o l a t e _ VB _2 , v i
o
l a ti o n _
NN _3 ) ( s y m pha l angu s _
NN _1 , on e - ha lf _ NN _1 ) ( n a s a li z a ti o n _ NN _1 , s o un d _o u t _
VB _1 ) ( t e s ti f y _ VB _2 , i n f o r m a ti o n _
NN _1 ) ( s u ff o c a t e _
VB _1 , s t r a n g l e r _ t r ee _
NN _1 ) ( j u s t _
JJ_3 , v a li d it y _
NN _1 ) ( t r e ad _ NN _1 , s t e p_
VB _7 ) ( c o nn ec t _ VB _6 , c o n ve r g i n g_ NN _1 ) ( nu m b e r _ VB _3 , po i n t _
NN _12 ) ( r e p r o ve _ VB _1 , t a l k
i ng_ t o_
NN _1 ) ( ye a r n _ VB _1 , p i n i n g _ NN _ ) ( c o nn ec t _ VB _6 , c o nn e x i o n _ NN _ ) ( d eve l op_
VB _10 , o r ga n i c _p r
o ce ss _
NN _1 ) ( s u s t a i n _ VB _5 , b e a m _ NN _2 ) ( un r e li ab l e n e ss _ NN _1 , a r b it r a r y _
JJ_1 )
( op e r a t e _ VB _4 , p s y op
_ NN _1 ) ( p li c a ti o n _ NN _1 , t w i s t _ VB _4 ) ( s p r i n g_ NN _6 , hu
r d l e _
VB _1 ) ( t r a ve s t y _
NN _2 , t r a ve s t y _
NN _2 ) ( m a r k e t _ VB _1 , t r ad e _
NN _4 ) ( s p lit _ u p_ VB _3 , s e pa r a ti o n _ NN _5 ) ( s pa r k _ NN _1 , s c i n till a t e _
VB _1 ) ( s pa r k _ NN _1 , s pa r k l e _ VB _1 ) ( op e r a t e _ VB _4 , m i ss i o n _
NN _2 ) ( p li c a ti o n _ NN _1 , w r i n k l e _
VB _2 ) ( u tilit y _ NN _2 , f un c ti o n a l _JJ_1 ) ( s t
o ck
i s t _ NN _1 , s t
o ck
i s t _
NN _1 ) i n s t a n ce _ hyp e
r ny m ( S ) ( m a l c o l m _ x _
NN _1 , h i p s t e r _
NN _1 ) ( ce n t r a l _a m e r
i c a_ NN _1 , ce n t r a l _a m e r
i c a _ NN _1 ) ( t h e r e s a_ NN _1 , m on k _ NN _1 ) ( t h e _ na z a r e n e _ NN _1 , j uda i s m _
NN _2 ) ( s t .
_g r e go r y _ i _ NN _1 , c hu r c h _
f a t
h e r _ NN _1 ) ( n i ppon_
NN _1 , e u r op e an_na ti on_
NN _ 1 ) ( o l d_ li n e _ s t a t e _ NN _1 , r i ve r _ NN _1 ) ( s ou t h _ k
o r e a _ NN _1 , a f r i c an _na ti on_ NN _1 ) ( g r e a t _ l a ke s _
NN _1 , r i ve r _ NN _1 ) ( r .
_ e .
_ b y r d_
NN _1 , c
o mm i ss i on e d_ o f
ﬁ
ce
r
_ NN _1 ) ( m a l c o l m _ x _
NN _1 , pa ss i ve _ r e
s i s t e r _ NN _1 ) ( r .
_ e .
_ b y r d_
NN _1 , n o n c
o mm i ss i o n e d _
o f ﬁ ce r _
NN _1 ) ( s ou t h _ k
o r e a _ NN _1 , p e n i n s u l a_ NN _1 ) ( m a l c o l m _ x _
NN _1 , b i r t h - c on t r o l _ r e f o r m e r _ NN _1 ) ( w o r l d _ w a r _ i _ NN _1 , p it c h e d _ba ttl
e
_ NN _ ) ( s t .
_g r e go r y _
o f _n .
NN _1 , v i c a r _o f _
c h r
i s t _
NN _ 1 ) ( l o s _a n g e l e s _
NN _1 , po r t _
NN _1 ) ( i r i s h
c
ap it a l _
NN _1 , u r ba n _ ce n t e r _
NN _1 ) ( n i ppon_
NN _1 , it a li an_ r e g i on _ NN _1 ) ( g r e a t _ l a ke s _
NN _1 , c anad i an _p r o v i n ce _ NN _1 ) ( v o l c ano_
i s l and s _
NN _1 , u r ban_ ce n t e r _
NN _1 ) ( r ob e r t _o w e n _ NN _1 , t yc oon _ NN _1 ) ( t r an s y l v an i an_a l p s _ NN _1 , u r ban _ ce
n t e r _ NN _1 ) ( n i ppon_
NN _1 , a m e r i c an _ s t a t e _
NN _ 1 ) ( m anda l a y _
NN _1 , na ti ona l _ c
ap it a l _
NN _1 ) ( g e tt y s b u r g_
NN _2 , s i e g e _
NN _1 ) ( h e l s i n k i _ NN _1 , u r ba n _ ce n t e r _ NN _ 1 ) ( nan_ li ng_ NN _1 , u r ban_ ce n t e r _ NN _1 ) ( w i s c
o n s i n _
NN _2 , g e og r ap h
i c a l _ r e
g
i o n _
NN _1 ) ( c ap it a l _o f _ ga m b i a_
NN _1 , u r ba n _ ce n t e r _
NN _1 ) s yn s e t _ do m a i n_ t op i
c _o f ( C ) ( li b e l _
NN _1 , c
r i m i n a l _ l a w _ NN _ ) ( w r it e _
VB _8 , t r a n s c r
i
p ti o n _
NN _5 ) ( a ss i s t _
NN _2 , a m .
_ f oo t ba ll _ ga m e _
NN _1 ) ( b r y nh il d_ NN _1 , m y t
h o l og y _ NN _2 ) ( t e m p l e _ NN _4 , m u s li m i s m _ NN _2 ) ( d r i ve _ NN _ 12 , c
o u r t _ga m e _ NN _1 ) ( s l u gg e r _ NN _1 , s po r t _ NN _1 ) ( a ss i s t _
NN _2 , h
o c k ey _ NN _ ) ( s e ll _
VB _1 , o ff e n s e _
NN _3 ) ( s e ll _
VB _1 , l a w _ NN _1 ) ( r e l a ti o n s h i p _ NN _4 , b i o l og y _ NN _1 ) ( s l u gg e r _ NN _1 , s o f t ba ll _ga m e _ NN _ 1 ) ( s e m iti c
d e it y _
NN _1 , m y t
h o l og y _ NN _1 ) ( apo s tl e _ NN _3 , w e s t e r n _ c hu
r c h _ NN _ ) ( d r i ve _ NN _ 12 , bad m i n t o n _ NN _ ) ( nu c l e a r _d e t e rr e n ce _ NN _1 , l a w _
NN _1 ) ( a ss i s t _
NN _2 , s po r t _
NN _1 ) ( r ece p ti o n _
NN _5 , ba s e ba ll _ga m e _ NN _1 ) ( t r o t _
VB _2 , e q u e s t
r
i a n _ s po r t _ NN _1 ) ( p h o t o
s y n t
h e s i s _
NN _1 , c
h e m i s t
r
y
NN _1 ) ( r n a_
NN _1 , c
h e m i s t
r
y
NN _1 ) ( i s o l d e _ NN _1 , pa r ab
l e _ NN _ ) ( a ss i s t _
NN _2 , s o cce r _
NN _1 ) ( a ss i s t _
NN _2 , c o u r t _ ga m e _
NN _1 ) ( a ss i s t _
NN _2 , f oo t ba ll _ga m e _ NN _ )