Modelling Monotonic and Non-Monotonic Attribute Dependencies with Embeddings: A Theoretical Analysis
Abstract
During the last decade, entity embeddings have become ubiquitous in Artiﬁcial Intelligence.
Such embeddings essentially serve as compact but semantically meaningful representations of the entities of interest.
In most approaches, vectors are used for representing the entities themselves, as well as for representing their associated attributes.
An important advantage of using attribute embeddings is that (some of the) semantic dependencies between the attributes can thus be captured.
However, little is known about what kinds of semantic dependencies can be modelled in this way.
The aim of this paper is to shed light on this question, focusing on settings where the embedding of an entity is obtained by pooling the embeddings of its known attributes.
Our particular focus is on studying the theoretical limitations of diﬀerent embedding strategies, rather than their ability to eﬀectively learn attribute dependencies in practice.
We ﬁrst show a number of negative results, revealing that some of the most popular embedding models are not able to capture even basic Horn rules.
However, we also ﬁnd that some embedding strategies are capable, in principle, of modelling both monotonic and non-monotonic attribute dependencies.
1. Introduction
Vector space embeddings are currently the dominant representation framework in Natural
Language Processing, Computer Vision and Machine Learning.
Essentially, these embeddings represent each entity of interest as a dense vector in some ﬁxed-dimensional space.
In addition, the attributes that are used to describe these entities are typically also encoded as vectors.
For example, in the case of word embeddings, the entities correspond to words and the attributes correspond to the contexts in which these words occur, e.g. in the form of co-occurring words [Mikolov et al., 2013, Pennington et al., 2014], syntactic dependencies [Levy and Goldberg, 2014, Vashishth et al., 2019] or even full sentences [Devlin et al., 2019].
In inductive knowledge graph embedding, the entities from our framework correspond to the previously unseen entities for which we want to learn a representation, with the attributes representing links to known entities [Hamaguchi et al., 2017].
In the case of embeddingbased topic models, the entities correspond to documents and the attributes correspond to the associated topics and words [Das et al., 2015, Li et al., 2016,
He et al., 2017, Xun et al., 2017, Dieng et al., 2020].
In zero-shot learning, the entities of interest are the category prototypes and the attributes correspond to semantic attributes of the categories [Lampert et al., 2013] or associated natural language terms [Frome et al., 2013].
An important advantage of attribute embeddings is that they can implicitly capture some of the dependencies that hold between the attributes.
For instance, the use of embeddings for topic modelling stems from the desire to capture topic correlations [He et al., 2017, Xun et al., 2017].
However, it is currently unclear what kinds of dependencies can be captured in this way.
Before we can address this question, we ﬁrst need to clarify what it means that an embedding captures some dependency.
To this end, consider an embedding model in which σ ( a · e ) represents the probability that entity e has attribute a , where σ is the sigmoid function and a and e are the embeddings of a and e .
Among many others, the popular skip-gram word embedding model is of this kind [Mikolov et al., 2013].
Now consider the attributes parent , female and mother , and the associated dependency that mother ≡ parent ∧ female .
What condition would need to be satisﬁed for the embedding to capture this dependency?
One possibility is to require that σ ( mother · e ) = σ ( parent · e ) · σ ( female · e ) for each vector e in the embedding space R n .
However, this condition can clearly not be satisﬁed; e.g. for e =
we obtain the condition 0 .
5 = 0 .
Similarly, it is easy to see that the condition σ ( mother · e ) =
min( σ ( parent · e ) , σ ( female · e )) only admits trivial solutions.
Viewed from this angle, it is clear that popular embedding models are not able to capture even basic logical dependencies.
For this reason, several alternative models have been proposed, in which attributes are modelled as linear subspaces [Garg et al., 2019], axis-aligned cones [Vendrov et al., 2016, ¨ Oz¸cep et al., 2020], hyperboxes [Vilnis et al., 2018] or polytopes [Guti´errez-Basulto and Schockaert, 2018], among others.
In this paper, we follow a diﬀerent direction, analysing whether embedding models can capture logical dependencies in a less demanding sense.
In particular, we consider the common setting where the embedding of an entity e has to be learned from its known attributes, and the aim of the resulting embedding is to infer what other attributes e is likely to satisfy.
We then say that the rule a 1 ∧ ... ∧ a n → b is captured if an entity which is known to have the attributes a 1 , ..., a n would be represented by a vector from which the attribute b would be predicted. Note that diﬀerent variants of this setting can be considered, which depend on (i) how exactly entity vectors are constructed and (ii) how the resulting vectors are used for predicting the attributes of the entity. The aim of this paper is to analyse, for diﬀerent variants, whether it is always possible to ﬁnd an embedding which captures the rule a 1 ∧ ... ∧ a n → b iﬀ that rule is entailed by some propositional knowledge base K . Note that in practice we typically do not have access to such a knowledge base K .
However, the question of whether arbitrary propositional knowledge bases can, in principle, be modelled by a given embedding strategy is important, because if this is not the case, then it also means that some (combinations of) dependencies cannot be learned.
In addition to standard propositional entailment, we also look at non-monotonic consequence relations, which is important because attribute dependencies are often defeasible or probabilistic in nature.
For instance, in a topic modelling context, we may assume that a document containing the word safari is related to the topic nature , and a document containing the word apple is related to the topic food .
However, documents containing both of these words are more likely to be related to technology instead, given that Safari is the name of Apple’s internet browser. We may thus want that the rule safari → topic:nature is captured, while the rule safari ∧ apple → topic:nature is not. 2. Problem Setting We assume that each entity is associated with some [-embedding function Emb : 
-]attributes from a given set A .
In the context of word embeddings, for instance, the set A could be the set of all contexts. Note that we do not consider knowledge graphs, where entities are described in terms of their relationship to other entities, rather than attributes. However, many inductive knowledge graph embedding models can still be cast as special cases of our framework, with the attributes then representing links to entities with known (or pre-computed) embeddings. Embedding and Labelling Functions
If an entity e is associated with the attributes a 1 , ..., a n , we assume that its embedding is given by e = Emb ( a 1 , ..., a n ) for some embedding function Emb : 2 A → R m . The embedding functions that we consider in this paper will rely on pooling attribute vectors. In particular, we consider embedding functions of the following form: Emb ( a 1 , ..., a n ) =
φ ( a 1 , ..., a n ) for some pooling function φ , where a ∈ R m represents the embedding of attribute a ∈ A . For instance, we may have Emb ( a 1 , ..., a n ) = n ( a 1 + ... + a n ).
Let us furthermore assume that we have a function Lab : R m → 2 A that predicts attributes of entities based on their embeddings.
Similar as for Emb , we will assume that the labelling function Lab relies on a scoring function that compares the embedding of e with an embedding of the considered attribute.
In particular, we consider labelling functions of the following form: Lab ( e ) = { b ∈
A | ψ ( e , ˜ b ) ≥ λ b } .
The embedding ˜ b of the attribute b may in general be diﬀerent from the attribute embedding b that is used for Emb , similar to how word embedding models learn two types of embeddings for each word.
The scalar λ b represents a threshold, which we allow to be attribute-dependent for generality.
For instance, we could have Lab ( e ) = { b ∈
A | σ ( e · b ) ≥ 0 . 5 } = { b ∈ A | e · b ≥ 0 } .
In the following, we will refer to ( φ, ψ ) as an embedding strategy and to ( Emb , Lab ) as an embedding .
Note that the embedding ( Emb , Lab ) is determined by the embedding strategy ( φ, ψ ), together with the embeddings a
and ˜a of the attributes in A .
The function Lab ◦ Emb can be viewed as a logical consequence relation.
In particular, we say that an embedding ( Emb , Lab ) captures the rule a 1 ∧ ... ∧ a n → b iﬀ b ∈ Lab ( Emb ( a 1 , ..., a n )), i.e. if an entity that is initially associated with the attributes a 1 , ..., a n would be predicted to have attribute b based on its embedding.
Note that under this view, the dependency mother ≡ parent ∧ female can be satisﬁed for the aforementioned choices of Emb and Lab , for instance by choosing parent =
( − 1 , 1), female = (1 , 1) and mother = (0 , 1).
However, now the issue is that a number of unwanted dependencies are also satisﬁed, such as female → mother and parent → mother .
Therefore, rather than treating rules in isolation, the question we are interested in is the following: given a propositional knowledge base K and a given embedding strategy ( φ, ψ ), does there exist a corresponding embedding ( Emb , Lab ) (or, equivalently, do there exist attribute embeddings) such that for all b ∈ A and { a 1 , ..., a n } ⊆
A we have that b ∈ Lab ( Emb ( a 1 , ..., a n )) iﬀ K | = a 1 ∧ ... ∧ a n → b holds, where | = is the standard entailment relation from propositional logic.
If this is the case, we say that the embedding strategy ( φ, ψ ) can simulate K .
In Section 4, we will similarly look at whether non-monotonic consequence relations can be simulated with embeddings.
Throughout this paper, we will assume that the number of dimensions m can be chosen arbitrarily large, as our focus is on identifying limitations that exist regardless of dimensionality.
An overview of our results is shown in Table 1.
3. Monotonic Reasoning
In Sections 3.1–3.3, we ﬁrst discuss embedding strategies which are not capable of capturing certain kinds of propositional knowledge bases.
Crucially, these strategies cover many of
[Table]
the most popular embedding models.
Section 3.4 then discusses embedding strategies which are capable of modelling arbitrary propositional knowledge bases. 3.1 Averaging Based Embeddings
One of the most natural choices for the embedding function Emb consists in averaging the attribute embeddings, i.e.:
[Equation]
This choice corresponds, among many others, to the common strategy of learning sentence or document vectors by averaging word vectors. It is also closely related to the CBOW model from Mikolov et al.
As the labelling function, we ﬁrst consider the common choice to model the probability that entity e has attribute a as σ ( e · a ).
However, since each condition of the form σ ( e · a ) ≥ δ , with 0 < δ < 1, is equivalent to the condition e · a ≥ σ − 1 ( δ ), we use a simple dot product in the formulation:
[Equation]
where λ b ∈ R is a threshold that may in general be attribute-speciﬁc.
We will use φ avg and ψ dot to denote the pooling functions associated with Emb avg and Lab dot . For the ease of presentation, throughout the paper, we will similarly use subscripts to link embedding and labelling functions to their corresponding pooling functions, rather than each time introducing these notations explicitly. Note that the embedding strategy ( φ avg , ψ dot ) also covers models where a linear transformation is applied to the average of the attribute embeddings, as is the case for the `a la carte method from Khodak et al.
As the following counterexample shows, not all propositional knowledge bases can be simulated with embeddings of the form ( Emb avg , Lab dot ).
Counterexample 1.
Let K = { a ∧ b → x, c ∧ d → x } .
We show that K cannot be simulated by ( φ avg , ψ dot ) . Indeed, if a suitable embedding existed, among others the following inequalities would have to be satisﬁed:
[Equation]
This is not possible, since the ﬁrst two inequalities imply ( a + b + c + d ) · ˜x ≥ 4 λ whereas the last two imply ( a + b + c + d ) ·
˜x < 4 λ .
Another natural choice for the labelling function is to rely on Euclidean distance:
[Equation]
where θ b ≥ 0.
It is easy to verify that the knowledge base from Counterexample 1 can be simulated by choosing a = ( − 1 , 0), b
= (1 , 0), c
= (0 , − 1), d = (0 , 1) and ˜x = (0 , 0).
However, as the following counterexample shows, not all knowledge bases can be simulated using Emb avg and Lab dist either.
Counterexample 2.
Let K = { a ∧ b → x, c ∧ d → x, a ∧ c → y, b ∧ d → y } .
We show that K cannot be simulated by ( φ avg , ψ dist ) . Indeed, suppose that a suitable embedding ( Emb avg , Lab dist ) existed.
From the rules with x in the head, it follows that d 2 (cid:0) a + b 2 , ˜x (cid:1) + d 2 (cid:0) c + d 2 , ˜x (cid:1) <
d 2 (cid:0) a + c 2 , ˜x (cid:1) + d 2 (cid:0) b + d 2 , ˜x (cid:1) .
This is equivalent with:
[Equation]
[Equation]
(cid:107) c (cid:107) 2 + (cid:107) d {+(cid:107) 2 + 8 (cid:107) x (cid:107) 
+}
{++ 2( a · b + c · d ) − 4( a + b + c + d ) · ˜x < (cid:107) a (cid:107) 2 + (cid:107) c (cid:107) 
 + (cid:107) b (cid:107) 
 + (cid:107) d (cid:107) 2 + 8 (cid:107) x (cid:107) 2 + 2( a · c + b · d ) − 4( a + b + c + d ) ·+}
˜x which simpliﬁes to a · b + c · d < a · c + b · d .
In the same way, using the rules with y in the head, we ﬁnd a · b + c · d > a · c + b · d , which is a contradiction.
To illustrate the relevance of these results, let the attributes a 1 , ..., a n correspond to the words that are observed in a document d , and suppose that other attributes, which are not observed, correspond to document categories.
We may want the embedding model to capture rules such as tennis ∧ player ∧ won → cat:sports , meaning that if the words tennis , player and won appear in a document, then it should belong to the category sports .
Our results show that, in general, such dependencies cannot be captured when Emb avg is used in combination with Lab dot or Lab dist .
For instance, this means that there are theoretical limitations to the kinds of categories that may be predicted from document embeddings when using the `a la carte method from [Khodak et al., 2018] together with a linear classiﬁer.
3.2 Embeddings as Normalised Averages
Let us now consider the following embedding function, which represents entities using normalised vectors:
[Equation]
provided (cid:107) a 1 + ... + a n (cid:107) > 0, and Emb norm ( a 1 , ..., a n ) = 0 otherwise.
Entity vectors can then be understood as maximum likelihood estimates, if we view attributes as von MisesFisher distributions, which is a common choice when modelling text [Banerjee et al., 2005,
Batmanghelich et al., 2016, Meng et al., 2019].
In particular, if p ( e | a ) is a von Mises-Fisher distribution with mean a (cid:107) a (cid:107) and concentration parameter (cid:107) a (cid:107) , for each a ∈ A , then we have: Emb norm ( a 1 , ..., a n ) =
[Equation]
The question of whether propositional knowledge bases can be simulated with Emb norm is thus relevant for understanding the limitations of von Mises-Fisher based topic models and document representations [Meng et al., 2019, Batmanghelich et al., 2016].
The following counterexample shows that not all knowledge bases can be simulated with Emb norm .
While the use of normalised averages is intuitively similar to the averages from Section 3.1, the counterexample in this case is more involved.
Counterexample 3.
Let K = { a ∧ b → x, c ∧ d → x, a ∧ c → y, b ∧ d → y, a ∧ d → y, b ∧ c → y } .
We show that K cannot be simulated by ( φ norm , ψ dot ) . Indeed, suppose that a suitable embedding ( Emb norm , Lab dot ) existed.
Then we must have λ x > 0 and λ y > 0 , which follows immediately from the fact that Emb norm ( ∅ ) = {+
+}
while K (cid:54)| = (cid:62) → x and K (cid:54)| = (cid:62) → y .
For the ease of presentation, let us introduce the following abbreviations: y ab = a + b (cid:107) a + b (cid:107) and similar for y ac , y ad , y bc , y bd , y cd .
Consider the hyperplane H deﬁned by H = { e | e · ( λ x ˜y − λ y ˜x )
= 0 } .
If e belongs to the positive half-space H + = { e | e · ( λ x ˜y − λ y ˜x ) ≥ 0 } , we have:
[Equation]
This implies that either e · ˜x < λ x or e · ˜y ≥ λ y .
We thus ﬁnd in particular that y ab and y cd do not belong to this positive half-space H + .
In the same way, we ﬁnd that y ac , y bd , y ad and y bc do not belong to the negative half-space H − =
{ e | e · ( λ x ˜y − λ y ˜x ) ≤ 0 } .
Note that when e 1 , e 2 ∈ H
+ , we also have e 1 + e 2 ∈ H + and e 1 + e 2 (cid:107) e 1 + e 2 (cid:107) ∈ H + , and similar for H − .
Since y ab , y cd ∈ H − , at least one of a , b must thus belong to H − and at least one of c , d must belong to H − .
Assume for instance that a ∈ H − and c ∈ H − ; the other cases follow by symmetry.
From a ∈ H − and c ∈ H − , we ﬁnd that y ac ∈ H − , which is a contradiction.
While we used Lab dot in the above counterexample, the result also holds for Lab dist .
Indeed, since (cid:107) e (cid:107) = 1, we have d ( e , ˜b ) ≤ θ b iﬀ d 2 ( e , ˜b )
≤ θ 2 b iﬀ e · ˜b ≥ 12 (1 + (cid:107) ˜b (cid:107) 2 − θ 2 b ).
We thus have that K can be simulated by ( φ norm , ψ dot ) iﬀ it can be simulated by ( φ norm , ψ dist ).
3.3 Sigmoid Based Embeddings
We now turn to the common choice of modelling attribute probabilities using the sigmoid function, i.e. let us assume that σ ( e · a ) represents the probability that entity e has attribute a .
A natural choice for inferring the embedding of e is then to maximise the likelihood of the observed attributes a 1 , ..., a n : Emb
[Equation]
where κ > 0 is a constant.
This embedding strategy closely corresponds to the skip-gram model [Mikolov et al., 2013], with two diﬀerences.
First, the standard skip-gram model does not include the regularisation term κ (cid:107) e (cid:107) 2 .
Here, we need to add this term, which amounts to imposing a Gaussian prior, to ensure that Emb sig is well-deﬁned 1 .
Second, the skip-gram model also includes negative samples.
However, since 1 − σ ( e · a ) = σ ( − e · a ), negative samples can be considered as a special case where some of the attributes a i capture the fact that another attribute b i is not present, by constraining the embedding such that a i = − b i .
The limitations of the embedding function Emb sig thus still apply to settings where negative samples are used.
The following counterexample shows that arbitrary propositional knowledge bases cannot be modelled with Emb sig and Lab dot .
Counterexample 4.
The main idea is to follow the same strategy as in Counterexample 3, which is possible thanks to the fact that Emb sig ( a, b ) is a conical combination of Emb sig ( a ) and Emb sig ( b ) , provided cos( a , b ) > − 1 .
Some care is needed to ensure that the latter condition is satisﬁed for various pairs of attributes.
Let K = { a ∧ b → x, c ∧ d → x, a ∧ c → y, b ∧ d → y, a ∧ d → y, b ∧ c → y, a → z ab , b → z ab , a → z ac , c
→ z ac , a → z ad , d → z ad , b → z bc , c → z bc , b → z bd , d → z bd , c → z cd , d → z cd } .
We show that K cannot be simulated by ( φ sig , ψ dot ) .
Suppose that a suitable embedding ( Emb sig , Lab dot ) existed.
Note that y a = α a a and y b = α b b for some α a , α b > 0 .
Since we have λ z ab > 0 (which follows in the same way as λ x > 0 ), it must be the case that cos( y a , ˜z ab ) > 0 and cos( y b , ˜z ab ) > 0 , which means cos( a , ˜z ab ) >
0 and cos( b , ˜z ab ) > 0 , which implies cos( a , b ) > − 1 .
This, in turn, implies that y ab is a conical combination of y a and y b .
We similarly have that y ac , y ad , y bc , y bd and y cd are conical combinations of the corresponding attribute vectors.
Consider again the hyperplane H deﬁned by H = { e | e · ( λ x ˜y − λ y ˜x )
= 0 } from Counterexample 3, and the positive and negative half-spaces H + and H − .
When e 1 , e 2 ∈ H + , we also have µ 1 e 1 + µ 2 e 2 ∈ H + for µ 1 , µ 2 ≥ 0 , i.e. if e 1 and e 2 are in H
+
then the same is true for any conical combination of e 1 and e 2 , and similar for H − .
We thus obtain a contradiction in the same way as in Counterexample 3.
In the appendix, we provide a similar counterexample for the strategy ( φ sig , ψ dist ). Note that Counterexample 4 only relies on the fact that Emb sig ( a 1 , ..., a n ) is a conical combination of a 1 , ..., a n . The same counterexample can thus be used for other embeddings strategies that rely on a weighted average of attribute vectors with non-negative weights.
3.4 Modelling Monotonic Dependencies with Embeddings
Thus far, we have found that standard embedding strategies are not capable of simulating even basic sets of Horn rules.
It turns out, however, that this limitation can be solved by adding a non-linearity to the labelling function:
[Equation]
where the ReLU function is applied component-wise.
Note that in the deﬁnition of Lab relu we ﬁxed the threshold at {+
+} and we ﬁxed a =
˜a for all a ∈ A , to highlight the fact that this restricted deﬁnition is already suﬃcient for modelling propositional dependencies.
Proposition 1.
For any propositional knowledge base K over A , there exist embeddings of the attributes in A such that b ∈ Lab relu ( Emb avg ( a 1 , ..., a n )) iﬀ K | = a 1 ∧ ... ∧ a n → b .
Proof.
Let mod ( K ) = { ω 1 , ..., ω l } be the set of models of K .
We deﬁne the embedding a = ( x a 1 , ..., x al +1 ) of the attribute a as follows:
[Equation]
where δ is a constant which is chosen such that δ > 2 |A| .
Let e
=
( y 1 , ..., y l +1 ) =
Emb avg ( a 1 , ..., a n ).
Let n i be the number of atoms from { a 1 , ..., a n } that are satisﬁed in ω i .
Then we have y l +
= 1 and for i ≤ l we have
[Equation]
In particular, we have y i
= 1 if ω i |
=
{ a 1 , ..., a n } and y i < 0 otherwise (since δ > |A| ≥ n i ).
We thus have:
[Equation]
if ω i | = { a 1 , ..., a n } otherwise
For b ∈ A , we ﬁnd
ReLU
[Equation]
In particular, we have ReLU ( e ) · b ≥ 0 iﬀ |{ ω | ω |
=
K ∪ { a 1 , ..., a n , ¬ b }| = 0, which is equivalent to K | = a 1 ∧ ... ∧ a n → b .
Note that the proof can be straightforwardly adapted to other types of non-linearities.
For instance, with sigmoid instead of ReLU, by choosing δ suﬃciently large, we can ensure that σ ( y i ) is arbitrarily close to 0 when ω
i (cid:54)| = { a 1 , ..., a n } , and rely on the same argument.
Next we consider the following embedding function:
[Equation]
where we write (cid:12) for the Hadamard product (i.e. the component-wise product of vectors).
In the appendix, we show the following result, using a construction that is very similar to the one from the proof of Proposition 1.
Proposition 2.
For any propositional knowledge base K over A , there exist embeddings of the attributes in A such that b ∈ Lab dot ( Emb had ( a 1 , ..., a n )) iﬀ K [Equation] =
[Equation] [Equation] ∧ a n → b . Note that in contrast to the setting from Proposition 1, here we allow a (cid:54) =
˜a .
It is easy to see that this additional freedom is necessary, since otherwise any embedding modelling a rule of the form a → b would also model the reversed rule b → a .
Finally, we consider the embedding strategy that is used in the order embeddings from Vendrov et al.
[Equation]
where we write max for the component-wise maximum of the vectors and (cid:22) is the product order, i.e. ( x 1 , ..., x n ) (cid:22) ( y 1 , ..., y n )
iﬀ x i ≤ y i for all i ∈ { 1 , ..., n } .
This embedding model was proposed to improve how hierarchical relations can be encoded.
However, as the next result shows, it also allows us to simulate other kinds of propositional dependencies.
Proposition 3.
For any propositional knowledge base K over A , there exist embeddings of the attributes in A such that b ∈ Lab ord ( Emb ord ( a 1 , ..., a n )) iﬀ K | = a 1 ∧ ... ∧ a n → b .
Proof.
Let mod ( K ) = { ω 1 , ..., ω l } be the set of models of K .
If K is inconsistent, we can simply choose a = 0 for every a ∈ A , with 0 a vector of zeroes of an arbitrary dimension.
Now suppose | mod ( K ) | > 0.
We deﬁne the embedding a = ( x a 1 , ..., x al ) of the attribute a as follows:
[Equation]
Let e
=
( y 1 , ..., y l )
=
Emb ord ( a 1 , ..., a n ).
Then we have y i
=
if ω i |
=
{ a 1 , ..., a n } and y i
=otherwise.
We thus have b (cid:22) e iﬀ ∀ i. ( x bi = 1) ⇒ ( y i = 1) iﬀ ∀ i. ( ω i (cid:54)| = b ) ⇒ ( ω (cid:54)| = { a 1 , ..., a n } ) iﬀ K | = a 1 ∧ ...
∧ a n → b .
Order embeddings essentially represent each attribute as an axis-aligned cone (where the vector components are viewed as lower bounds).
Other region based embedding models can be used to model monotonic dependencies in a similar way, including e.g. box embeddings [Vilnis et al., 2018] and hyperbolic entailment cones [Ganea et al., 2018].
4. Non-Monotonic Reasoning
We now consider a standard ranking-based semantics of default rules [Lehmann and Magidor, 1992].
Let Θ be a stratiﬁed knowledge base, i.e. a ranked list of formulas ( α 1 , ..., α k ).
Then we say that Θ | = α (cid:66) β iﬀ there is some i ∈ { 0 , ..., k } such that α 1 ∧ ...
∧ α i ∧ α | = β and α 1 ∧ ...
∧ α i ∧ α (cid:54)| = ¬ β .
The formula α (cid:66) β intuitively means that “if α holds then typically also β holds”.
This semantics of default rules can equivalently be characterised in terms of the maximum a posteriori (MAP) consequences of a probabilistic model, i.e. α (cid:66) β is inferred iﬀ β is true in the most probable models of α [Kuzelka et al., 2016].
Example 1.
Consider the following stratiﬁed knowledge base: Θ = ( ¬ cat:technology ∨ ¬ cat:food , apple ∧ safari → cat:technology , apple → cat:food )
It can be veriﬁed that Θ | = apple (cid:66) cat:food and Θ | = apple ∧ safari (cid:66) cat:technology, while Θ (cid:54)| = apple ∧ safari (cid:66) cat:food.
We now analyse whether an embedding can be found such that b ∈ Lab ( Emb ( a 1 , ..., a n )) iﬀ Θ | = a 1 ∧ ... ∧ a n (cid:66) b .
Clearly, embedding strategies which cannot be used to simulate monotonic reasoning cannot be used to simulate this form of non-monotonic reasoning either.
This is because we can choose Θ 1 = ( α 1 ), where α 1 is the conjunction of all formulas in a propositional knowledge base K .
However, we ﬁnd that the strategy ( φ avg , ψ relu ) can be used to model non-monotonic attribute dependencies.
Proposition 4.
For any stratiﬁed knowledge base Θ over A , there exist embeddings of the attributes in A such that b ∈ Lab relu ( Emb avg ( a 1 , ..., a n )) iﬀ Θ | = a 1 ∧ ... ∧ a n (cid:66) b .
Proof.
Let Θ =
( α 1 , ..., α m ) and let ω 1 , ..., ω l be an enumeration of all interpretations over
A .
To deﬁne the embeddings a = ( x a 1 , ..., x al ), we use the mapping µ deﬁned by µ ( ω )
= max { i | ω | = α 1 ∧ ... ∧ α i } , where we assume µ ( ω ) = 0 if ω (cid:54)| =
α 1 .
We deﬁne:
[Equation]
otherwise where δ is chosen such that δ > 2 |A| .
Let e
= ( y 1 , ..., y l ) =
Emb avg ( a 1 , ..., a n ).
Similar as in the proof of Proposition 1 we then
ﬁnd y i < 0 iﬀ ω i (cid:54)| = { a 1 , ..., a n } , i.e.: [Equation] [Equation] n
[Equation]
otherwise For b ∈ A , we ﬁnd that ReLU ( Emb avg ( a 1 , ..., a n )) · b is given by (cid:88)
[Equation]
b }
[Equation]
(cid:88)
[Equation]
We thus have ReLU ( Emb avg ( a 1 , ..., a n )) · b ≥ 0 iﬀ (cid:88) ω i | = { a 1 ,...,a n ,b } δ 4 µ ( ω i ) ≥ (cid:88) ω i | = { a
[Equation]
...,a n , ¬ b } δ (1+4 µ ( ω i )) Let m + = max { µ ( ω i ) | ω i | = { a 1 , ..., a n , b }} and m − = max { µ ( ω i ) | ω i | = { a 1 , ..., a b , ¬ b }} , where we deﬁne m − = − 1 if { a 1 , ..., a n , ¬ b } is inconsistent (i.e. if b = ¬ a i for some i ).
Then we have that Θ | = a 1 ∧ ... ∧ a n (cid:66) b is equivalent to m + > m − .
If m + > m − , we ﬁnd (cid:88) [Equation] | [Equation]
(cid:88)
[Equation]
Conversely, if m + ≤ m −
we ﬁnd (cid:88)
[Equation]
where the last step relies on the fact that m + ≤ m − implies m − ≥
and thus there must be some ω
i such that ω i | = { a 1 , ..., a n , b } .
We thus have that Θ | = a 1 ∧ ... ∧ a n (cid:66) b is equivalent to m + > m − , which is equivalent to (9) and ReLU ( Emb avg ( a 1 , ..., a n )) · b ≥ 0.
In the appendix we show the following result, using a similar construction.
Proposition 5.
For any stratiﬁed knowledge base Θ over A , there exist embeddings of the attributes in A such that b ∈ Lab dot ( Emb had ( a 1 , ..., a n )) iﬀ Θ | = a 1 ∧ ... ∧ a n (cid:66) b .
Finally, we show that the combination Emb ord and Lab ord cannot be used to model nonmonotonic dependencies.
A similar limitation arises for all approaches which learn embeddings by taking the intersection of region-based attribute representations.
Counterexample 5.
Let Θ =
( a ∧ b → ⊥ , a → x ) .
To simulate Θ with an embedding of the form ( Emb ord , Lab ord ) , we need x (cid:22) a and x (cid:54)(cid:22) max( a , b ) .
However, this is impossible since a (cid:22) max( a , b ) .
5. Concluding Remarks
It remains poorly understood how we can design embedding models to encourage diﬀerent kinds of dependencies to be captured (with the problem of embedding hierarchies being a notable exception [Vendrov et al., 2016, Nickel and Kiela, 2017, Ganea et al., 2018]).
The analysis presented in this paper provides a step towards such an understanding.
The ability of the ReLU-based labelling function to model monotonic and non-monotonic attribute dependencies seems of particular interest, given how close it stays to standard embedding models.
While we have focused on propositional dependencies, our results also have implications for knowledge graph embedding.
For instance, bilinear models can be viewed as instances of the sigmoid based embedding strategy, where attributes represent links to other entities.
The attribute vectors then depend on the embeddings of other entities, which are iteratively updated.
This makes it possible to capture certain types of relational dependencies.
Developing a better understanding of which types of relational dependencies can be modelled in this way is an important avenue for future work.
However, the results presented in this paper already show that such models are not able to capture arbitrary dependencies.
Appendix A. Counterexample for Emb sig with Lab dist
Let K be deﬁned as in Counterexample 4, and let us again use the abbreviations of the form y ab and y a .
We show that there must always exist a hyperplane H that separates y ab and y cd , on the one hand, from y ac , y ad , y bc and y bd , on the other hand.
The fact that K cannot be modelled using Emb sig ( a ) and Lab dist then follows in the same way as in
Counterexample 4.
Let us write S x for the hypersphere around ˜x of radius θ x , i e. S x = { e | d ( e , ˜x ) ≤ θ x } , and let S y similarly be the hypersphere around ˜y of radius θ y .
If d (˜ x, ˜ u ) ≥ θ x + θ y , then we can simply deﬁne H as any hyperplane that separates S x and S y .
If d (˜ x, ˜ u ) <
θ x + θ y , then we choose H as the unique hyperplane that contains the intersection of the boundaries of S x and S y , noting that this boundary cannot be empty, since y ab ,
y cd ∈ S x \ S y and y ac , y ad , y bc , y
bd ∈ S
y \ S x , meaning that we cannot have S x ⊆ S y or S y ⊆ S x .
Moreover, since y ab , y cd ∈ S x \ S y and y ac , y ad , y bc , y
bd ∈ S
y \ S x , in both cases we clearly have that
H separates y ab , y cd from y ac , y ad , y bc , y bd .
Appendix B. Proof of Proposition
Let mod ( K ) = { ω 1 , ..., ω l } be the set of models of K .
We deﬁne the embeddings a
=
( x a 1 , ..., x al +1 )
and ˜a =
(˜ x a 1 , ..., ˜ x al +1 ) of the atom a as follows: x ai =  
 if i ≤ l and ω i | = a if i ≤ l and ω i (cid:54)| = a if i =
l +˜ x ai = 
  if i ≤ l and ω i | = a − δ if i ≤ l and ω i (cid:54)| = a if i = l + where δ is a constant satisfying δ > 2 |A| .
Let us write Emb had ( a 1 , ..., a n ) =
y =
( y 1 , ..., y l +1 ).
Then we clearly have: y i =
(cid:40)if ω i | = { a 1 , ..., a n } otherwise
For b ∈ A , we ﬁnd
[Equation]
Note that we have y · ˜b ≥ 0 iﬀ |{ ω | ω | =
K ∪{ a 1 , ..., a n , ¬ b }| = 0, since we assumed δ > 2 |A| .
We have |{ ω | ω | =
K ∪ { a 1 , ..., a n , ¬ b }| = 0 iﬀ K ∪ { a 1 , ..., a n } | =
b iﬀ K | = a 1 ∧ ... ∧ a n → b .
In particular, we have:
[Equation]
Emb had
[Equation]
a
[Equation]
a n )
[Equation] ≥ [Equation] Appendix C. Proof of Proposition
Let Ω =
( α 1 , .., α m ) and let ω 1 , ..., ω l be an enumeration of all interpretations over A .
We deﬁne the embedding a = ( x a 1 , ..., x al ) as follows:
[Equation]
otherwise
To deﬁne the embedding ˜a =
(˜ x a 1 , ..., ˜ x al ), we use the mapping µ deﬁned by µ ( ω )
= max { i | ω | = α 1 ∧ ... ∧ α i } , where we assume µ ( ω ) = 0 if ω (cid:54)| = α 1 :
[Equation]
+2 µ ( ω i ))
otherwise where δ is a constant which is chosen such that δ > 2 |A| .
Let us write Emb had ( a 1 , ..., a n ) =
y =
( y 1 , ..., y l ).
Note that we have:
[Equation] if ω i | = { a 1 , ..., a n } otherwise
For b ∈ A , we ﬁnd y ·
[Equation]
= (cid:88) ω i | =
[Equation]
[Equation]
a n ,b }
[Equation]
(cid:88) ω
[Equation]
| [Equation] [Equation] We thus have y · ˜b ≥ 0, i.e. b ∈ Lab dot ( y ), iﬀ (cid:88) ω i | =
[Equation]
a n
[Equation]
b } δ
[Equation] [Equation] ≥ (cid:88) ω i | = [Equation]
a
[Equation]
b }
[Equation] [Equation] [Equation] Let m + = max { µ ( ω i ) | ω i | = { a 1 , ..., a n , b }} and m − = max { µ ( ω i ) | ω i | = { a 1 , ..., a n , ¬ b }} , where we deﬁne m − = − 1 if { a 1 , ..., a n , ¬ b } is inconsistent (i.e. if b = ¬ a i for some i ).
Then we have Θ | = a 1 ∧ ... ∧ a n (cid:66) b iﬀ m + > m − .
If m + > m − , we ﬁnd (cid:88)
[Equation]
Now, conversely, suppose m + ≤ m − .
Then we have
[Equation]
where the last step relies on the fact that m + ≤ m − implies m − ≥ 0, and hence { a 1 , ..., a n , ¬ b } must be consistent.
We thus have that m + > m − is equivalent to (cid:80) ω
i | = { a 1 ,...,a n ,b } δ 2 µ ( ω i ) ≥
[Equation]
..,a n ¬ b } δ (1+2 µ ( ω i )) , which is equivalent to b ∈ Lab dot ( Emb had ( a 1 , ..., a n )).