E QUIVARIANT E NTITY -R ELATIONSHIP N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
The relational model is a ubiquitous representation of big-data, in part due to its extensive use in databases.
However, recent progress in deep learning with relational data has been focused on (knowledge) graphs.
In this paper we propose Equivariant Entity-Relationship Networks, a general class of parameter-sharing neural networks derived from the entity-relationship model.
We prove that our proposed feed-forward layer is the most expressive linear layer under the given equivariance constraints, and subsumes recently introduced equivariant models for sets, exchangeable tensors, and graphs.
The proposed feed-forward layer has linear complexity in the data and can be used for both inductive and transductive reasoning about relational databases, including database embedding, and the prediction of missing records.
This provides a principled theoretical foundation for the application of deep learning to one of the most abundant forms of data.
I NTRODUCTION
In the relational model of data, we have a set of entities , and one or more instances of each entity.
These instances interact with each other through a set of ﬁxed relations between entities.
A set of attributes may be associated with each type of entity and relation.
This simple idea is widely used to represent data, often in the form of a relational database, across a variety of domains, from shopping records, social networking data, and health records, to heterogeneous data from astronomical surveys.
Learning and inference on relational data has been the topic of research in machine learning over the past decades.
The relational model is closely related to ﬁrst order and predicate logic, where the existence of a relation between instances becomes a truth statement about a world.
The same formalism is used in AI through a probabilistic approach to logic, where the ﬁeld of statistical relational learning has fused the relational model with the framework of probabilistic graphical models .
Examples of such models include plate models, probabilistic relational models, Markov logic networks, and relational dependency networks (Getoor & Taskar, 2007).
A closely related area that has enjoyed accelerated growth in recent years is relational and geometric deep learning , where the term “relational” is used to denote the inductive bias introduced by a graph structure.
Although relational and graph-based terms are often used interchangeably in the machine learning community, they could refer to different data structures: graph-based models such as graph databases (Robinson et al., 2013) and knowledge graphs simply represent data as an attributed (hyper-)graph, while the relational model, common in databases, uses the entity-relation (ER) diagram (Chen, 1976) to constrain the relations of each instance (corresponding to a node), based on its entity-type; see Fig. 1(a).
Inspired by the success of equivariant deep learning, we use invariance and equivariance to encode the structure of relational data.
This type of inductive bias informs a model’s behaviour under various transformations.
Equivariant models have been successfully used for deep learning on data with a variety of structures, from translation equivariant images (LeCun et al., 1998), to geometric settings (Cohen et al., 2018a; 2019), to discrete objects such as sets (Zaheer et al., 2017), and graphs (Kondor et al., 2018b).
By adopting this perspective, we present a maximally expressive feedforward layer that achieves equivariance w.r.t.
exchangeabilities in relational data as it is expressed by the entity-relationship model.
Our feedforward layer generalizes recently proposed layers for sets (Zaheer et al., 2017), exchangeable tensors (Hartford et al., 2018), and graphs (Maron et al., 2018).
Here, the parameter-tying in the weight matrix W of the EERL (Fig. 2) guarantees that any permutation of elements of vec( (cid:88) ) corresponding to a shufﬂing of entities in this tensor representation (and only these permutations), results in the same permutation of the output of the layer.
R EPRESENTATIONS OF R ELATIONAL D ATA
We represent the relational model by a set of entities (cid:68) = { 1 , ..., D } and a set of relations R ⊆ [-
-]2 (cid:68) , indicating how the entities interact with one another.
For each entity d ∈ (cid:68) we have a set { 1 , .., N d } of instances of that entity.
For each relation (cid:82) ∈ R , where (cid:82) =
{ d 1 , . . .
, d | (cid:82) | } is a set of entities, we observe data in the form of a set of tuples (cid:88) (cid:82) = {(cid:104) n d 1 , . .
. , n d | (cid:82) | , x (cid:105)
| n d i ∈ [ N d i ] , x ∈ R K } . That is, for each element of (cid:88) (cid:82) we associate the feature vector x with the relationship between the instances indexed by n d i for each entity d i ∈ (cid:82) . See Fig.
1 for a detailed, concrete example.
Note that the singleton relation (cid:82) = { d i } can be used to incorporate individual entity attributes (such as professors’ evaluations in Fig. 1(a)).
In the most general case, we allow for both R , and any (cid:82) ∈ R to be multisets ( i.e. , to contain duplicate entries).
R is a multiset if we have multiple relations between the same set of entities.
For example, we may have a SUPERVISES relation between STUDENTS and PROFESSORS , in addition to the
WRITES REFERENCE relation.
A particular relation (cid:82) is a multiset if it contains multiple copies of the same entity.
Such relations are ubiquitous in the real world, describing for example, the connection graph of a social network, the sale/purchase relationships between a group of companies, or, in our running example, the COURSE - COURSE relation capturing prerequisite information.
For our derivations we make the simplifying assumption that each attribute x ∈ R is a scalar.
Extension to x ∈ R K using multiple channels is trivial and discussed in Appendix F.
Another common feature of relational data, the one-to-many relation, is addressed in Appendix E.
2. T UPLES , T ABLES AND T ENSORS
In relational databases the set of tuples (cid:88) (cid:82) is often represented using a table, with one row for each tuple; see Fig. 1(b).
An equivalent representation for (cid:88) (cid:82) is using a “sparse” | (cid:82) | -dimensional tensor
X (cid:82) ∈ R N d 1 × ...
× N d | (cid:82) | , where each dimension of this tensor corresponds to an entity d ∈ (cid:82) , and the the length of that dimension is the number of instances N d .
In other words
[Equation]
We work with this tensor representation of relational data.
We use (cid:88) =
{ X (cid:82) | (cid:82) ∈ R } to denote the set of all sparse tensors that deﬁne the relational data(base); see Fig. 1(c).
For the following discussions around exchangeability and equivariance, we assume that for all (cid:82) , X (cid:82) are fully observed, dense tensors.
Subsequently, we will discard this assumption and attempt to make predictions for (any subset of) the missing records.
Note that relations (cid:82) can be multisets.
For simplicity, we handle this in the main text by considering equal elements as distinct through indexing ( e.g. , d ( i ) =
d ( j ) ), while leaving a formal deﬁnition of multisets for the supplementary material; see Appendix B.
E XCHANGEABILITIES OF E NTITY -R ELATIONSHIP D ATA
Recall that in the representation (cid:88) , each entity d ∈ { 1 , ..., D } has a set of instances indexed by n d ∈ { 1 , .., N d } .
The ordering of { 1 , ..., N d } is arbitrary, and we can shufﬂe these instances, affecting only the representation, and not the “content” of the relational data.
However, in order to maintain consistency across data tables, we also have to shufﬂe all the tensors X (cid:82) , where d ∈ (cid:82) , using the same permutation applied to the tensor dimension corresponding to d .
At a high level, this simple indifference to shufﬂing deﬁnes the exchangeabilities of relational data.
A mathematical group formalizes this idea.
A mathematical group is a set equipped with a binary operation between its members, such that the set and the operation satisfy closure, associativity, invertability and existence of a unique identity element.
(cid:83) M refers to the symmetric group , the group of all permutations of M objects.
A natural representation for a member of this group (cid:103) M ∈ (cid:83) M , is a permutation matrix G ∈ { 0 , 1 } M × M .
Here, the binary group operation is the same as the product of permutation matrices.
In this notation, (cid:83) N d is the group of all permutations of instances 1 , . . .
N d of entity d ∈ { 1 , .., D } .
To consider permutations to multiple dimensions of a data tensor we can use the direct product of groups.
Given two groups (cid:71) and (cid:72) , the direct product (cid:71) × (cid:72) is deﬁned by
[Equation] and ( (cid:103) , (cid:104) [Equation]
( (cid:103) (cid:48) [Equation] (cid:104) (cid:48) [Equation]
[Equation] (cid:104) [Equation] [Equation]
[Equation] That is, the underlying set is the Cartesian product of the underlying sets of (cid:71) and (cid:72) , and the group operation is the component-wise operation.
Observe that we can associate the group (cid:83) N 1 × . . .
× (cid:83) N D with the D entities in a relational model, where each entity d has N d instances.
Intuitively, applying permutations from this group to the corresponding relational data should not affect the underlying contents, while applying permutations from outside this group should.
To see this, consider the tensor representation of Fig.
1(c): permuting students, courses or professors shufﬂes rows or columns of (cid:88) , but preserves its underlying content.
However, arbitrary shufﬂing of the elements of this tensor could alter its content.
Our goal is to deﬁne a feed-forward layer that is “aware” of this structure.
For this, we ﬁrst need to formalize the action of (cid:83) N 1 × . . .
× (cid:83) N D on the vectorized form of (cid:88) .
Vectorization.
For each tensor X (cid:82) ∈ (cid:88) , N (cid:82) = (cid:81) d ∈ (cid:82) N d refers to the total number of elements of tensor X (cid:82) (note that for now we are assuming that the tensors are dense).
We will refer to
N = (cid:80) (cid:82) ∈ R N (cid:82) as the number of elements of (cid:88) .
Then vec( X (cid:82) )
∈ R N (cid:82) refers to the vectorization of X (cid:82) , obtained by successively stacking its elements along its dimensions, where the order of dimensions is given by d ∈ (cid:82) .
We use vec − 1 ( · ) to refer to the inverse operation of vec( · ) , so that vec − 1 (vec( X ))
=
X .
With a slight abuse of notation, we use vec( (cid:88) )
∈
R N to refer to [vec( X (cid:82) 1 ); . .
; vec( X (cid:82) | R | )] , the vectorized form of the entire relational data.
The weight matrix W that we deﬁne later creates a feed-forward layer σ ( W vec( (cid:88) )) applied to this vector.
Group Action.
The action of (cid:103) ∈ (cid:83) N 1 × . . .
× (cid:83) N D on vec( (cid:88) )
∈ R N , permutes the elements of vec( (cid:88) ) .
Our objective is to deﬁne this group action by mapping (cid:83) N 1 × . . .
× (cid:83) N D to a group of permutations of all N = (cid:80) (cid:82) ∈ R (cid:81) d ∈ (cid:82) N d entries of the database – i.e. , a homoomorphism into (cid:83) N .
To this end we need to use two types of matrix product.
Let G ∈ R N 1 × N 2 and H ∈ R N 3 × N 4 be two matrices.
The direct sum G ⊕ H is an ( N 1 + N 3 ) × ( N 2 + N 4 ) block-diagonal matrix, and the
Kronecker product G ⊗ H is an ( N 1 N 3 ) ×
( N 2 N 4 ) matrix:
[Equation]
Note that in the special case that both G and H are permutation matrices, G ⊕ H and G ⊗ H will also be permutation matrices.
Both of these matrix operations can represent the direct product of permutation groups.
That is, given two permutation matrices G 1 ∈ (cid:83) N 1 , and G 2 ∈ (cid:83) N 2 , we can use both G 1 ⊗ G 2 and G 1 ⊕ G 2 to represent members of (cid:83) N 1 × (cid:83) N 2 .
However, the resulting permutation matrices, can be interpreted as different actions : while the ( N 1 + N 2 ) × ( N 1 + N 2 ) direct sum matrix G 1 ⊕ G 2 is a permutation of N 1 + N 2 objects, the ( N 1 N 2 ) × ( N 1 N 2 )
Kronecker product matrix G 1 ⊗ G 2 is a permutation of N 1 N 2 objects.
Claim 1. Consider the vectorized relational data vec( (cid:88) ) of length N .
The action of (cid:83) N 1 × . . .
× (cid:83) N D on vec( (cid:88) ) is given by the following permutation group
[Equation]
where the order of relations in (cid:76) (cid:82) ∈ R is consistent with the ordering used for vectorization of (cid:88) .
Proof.
The Kronecker product (cid:78) d ∈ (cid:82) G d when applied to vec( X (cid:82) ) , permutes the underlying tensor
X (cid:82) along the axes d ∈ (cid:82) .
Using direct sum, these permutations are applied to each tensor vec( (cid:88) (cid:82) ) in vec( (cid:88) )
= [vec( X (cid:82) 1 ); . .
, vec( X (cid:82) D )] .
The only constraint, enforced by (2) is to use the same permutation matrix G d for all (cid:82) when d ∈ (cid:82) .
Therefore any matrix-vector product G N vec( (cid:88) ) is a “legal” permutation of vec( (cid:88) ) , since it only shufﬂes the instances of each entity.
E QUIVARIANT E NTITY -R ELATIONSHIP L AYER
Our objective is to constrain the standard feed-forward layer f : R K × N → R K (cid:48)
× N —where
K, K (cid:48) are the number of input and output channels, and N = | vec( (cid:88) ) | — such that any “legal” transformation of the input, as deﬁned in (2), should result in the same transformation of the output.
Doing so amounts to augmenting our input data by applying all such transformations and using an unconstrained feed-forward layer.
This layer would be quadratic in the size of the database, and the model would require exponentially more data to train.
In the case of even moderately sized databases this approach is simply infeasible, further motivation the our EERL (Deﬁnition 1).
For clarity, we limit the following deﬁnition to the case where K = K (cid:48)
= 1 ; see Appendix F for the case of multiple channels.
Deﬁnition 1 (Equivariant Entity-Relationship Layer; EERL) .
Let G (cid:88) be any N × N permutation of vec( (cid:88) ) .
A fully connected layer σ ( W vec( (cid:88) )) with W ∈ R N × N is called an Equivariant
Entity-Relationship Layer
if
[Equation]
That is, an EERL is a layer that commutes with the permutation G (cid:88) if and only if G (cid:88) is a legal permutation, as deﬁned by (2).
An EERL 2 captures the notion that the order of the entities in the relational data (cid:88) is not important, and the layer should not produce different outputs when given different orderings.
We now propose a procedure to tie the entries of W so as to guarantee the conditions of Deﬁnition 1.
Moreover, we show that the resulting linear map W : R N × N is the most expressive linear map under the equivariance constraints.
We build up the matrix W ∈ R N × N block-wise, with blocks W i,j ∈ R N (cid:82) i × N (cid:82) j corresponding to each pair of relations (cid:82) i , (cid:82) j : W =   W
[Equation]
W
[Equation]
W 1 , | R | ...
W | R
[Equation]
W
[Equation]
W
[Equation]  [Equation] We tie parameters within each block, and not across blocks.
To concisely express the somewhat complex tying scheme, we use the following indexing notation.
Indexing Notation.
The parameter block W i,j is an N (cid:82) i × N (cid:82) j matrix, where N (cid:82) i =
(cid:81) d ∈ (cid:82) i N d .
We want to index rows and columns of W i,j .
Given the relation (cid:82) i = { d 1 , . . .
, d | (cid:82) i | } , we use the tuple n i .
= (cid:104) n id 1 , . . .
, n id | (cid:82) i | (cid:105) ∈
[ N d 1 ] × . . .
×
[ N d | (cid:82) i | ] to index an element in the set [ N (cid:82) i ] .
Since each element of n i indexes instances of a particular entity, n i can be used as an index both for data block vec( X (cid:82) i ) and for the rows of parameter block W i,j .
In particular, to denote an entry of W i,j , we use W i,j n i , n j .
Moreover, we use n id to denote the element of n i corresponding to entity d ∈ (cid:82) i .
Note that this is not necessarily the d th element of the tuple n i .
For example, if (cid:82) i = { 1 , 4 , 5 } and n i = (cid:104) 400 , 12 , 3 (cid:105) , then n i 4 = 12 and n i 5 = 3 .
When (cid:82) i is a multiset, we can use n id ( k ) to refer the to the element of n i corresponding to the k -th occurrence of entity d (where the order corresponds to the ordering of elements in n i ).
Table 1 in the Appendix summarizes our notation.
4. P ARAMETER T YING
Let W i,j n i , n j and W i,j m i , m j denote two arbitrary elements of the parameter matrix W i,j .
Our objective is to decide whether or not they should be tied together to ensure the resulting layer is an EERL (Deﬁnition 1).
To this end we deﬁne an equivalence relation between the indices: (cid:104) n i , n j (cid:105) ∼ (cid:104) m i , m j (cid:105) , and tie together all entries of W that are equivalent under this relation.
Index (cid:104) n i , n j (cid:105) is equivalent to index (cid:104) m i , m j (cid:105) if they have the same equality patterns over their indices for each unique entity d .
We consider n i,j , the concatenation of n i with n j , and examine the sub-tuples n i,jd , where n i,j is restricted to only those indices that index entity d .
We do this so that we can ask if these indices refer to the same instance ( e.g. , the same student), and we can only meaningfully compare indices of the same entity ( e.g. , two entries in the STUDENT table may refer to the same student, but an entry in the
STUDENT table cannot refer to the same thing as an entry in the COURSE table.
Also, recall that we allow (cid:82) i and (cid:82) j to be multisets ( e.g. , the COURSE - COURSE relation (cid:82) 4 in Fig. 1).
We say two tuples n i,jd = (cid:104) n d (1) , . . .
n d ( L ) (cid:105) and m i,jd = (cid:104) m
d (1) , . . .
m d ( L ) (cid:105) are equivalent iff the have the same equality pattern n d ( l ) =
n d ( l (cid:48) )
⇔ m d ( l ) = m d ( l (cid:48) )
∀ l ∈ [ L ] .
Accordingly, two index tuples n i,j and m i,j are equivalent iff they are equivalent for all d ∈ (cid:82) i ∪ (cid:82) j .
Because of this tying scheme, the total number of free parameters in W i,j is the product of the number of possible different partitionings of n i,jd for each unique entity d ∈ (cid:82) i ∪ (cid:82) j , and so is a product of
Bell numbers 3 , which count the number of partitions of a set given size; see Appendix C for details.
This relation to Bell numbers was previously shown for equivariant graph networks (Maron et al., 2018) which, as we see in Section 6, are closely related to, and indeed a special case of, our model.
This parameter-sharing scheme admits a simple recursive form, if the database has no self-relations ( i.e. , the (cid:82) i are not multisets); see Appendix E.
Example 1.
[Fig. 2(a)]
To get an intuition for this tying scheme, consider a simpliﬁed version of the relational structure of Fig.
1, restricted to the three relations (cid:82)
= { 1 , 2 } , self-relation (cid:82) 2 = { 2 , 2 } , and R 3 = { 1 , 3 } with N 1 = 5 STUDENTS , N 2 = 4 COURSES , and N 3 = 3 PROFESSORS .
Then
N = 5 × 4+4 × 4+5 × 3 = 51 , so W ∈ R 51 × 51 and will have nine blocks: W 1 , 1 ∈ R 20 × 20 , W 1 , 2 ∈ R 20 ×. W 2 , 2 ∈ R 16 × 16 and so on.
We use tuple n 1 = (cid:104) n 11 , n 12 (cid:105) ∈
[ N 1 ] × [ N 2 ] =
[5] × [4] to index the rows and columns of W 1 , 1 .
We also use n 1 to index the rows of W 1 , 2 , and use n 2 = (cid:104) n 22 (1) , n 22 (2)
(cid:105) ∈
[ N 2 ] × [ N 2 ] =
[4] × [4] to index its columns.
Other blocks are indexed similarly.
Suppose n 1 = (cid:104) 1 , 4 (cid:105) , n 2 = (cid:104) 4 , 5 (cid:105) , m 1 = (cid:104) 2 , 3 (cid:105) and m 2 = (cid:104) 3 , 2 (cid:105) and we are trying to determine whether W 1 , 2 n 1 , n 2 and W 1 , 2 m 1 , m 2 should be tied.
Then n 1 , 2 = (cid:104) 1 , 4 , 4 , 5 (cid:105) and m 1 , 2 = (cid:104) 2 , 3 , 3 , 2 (cid:105) .
When we compare the sub-tuples restricted to unique entities, we see that the equality pattern of n 1 , 2= (cid:104) 1 (cid:105) matches that of m 1 , 2= (cid:104) 2 (cid:105) (since it is only a singleton, it matches trivially), and the equality pattern of n 1 , 2 2 = (cid:104) 4 , 4 , 5 (cid:105) matches that of m 1 ,
=
(cid:104) 3 , 3 , 2 (cid:105) .
So these weights should be tied.
We now establish the optimality of our constrained linear layer for relational data.
Theorem 4.1.
Let (cid:88) =
{ X (cid:82) | (cid:82) ∈ R } be the tensor representation of some relational data and vec( (cid:88) )
its vectorized form.
If we deﬁne W block-wise according to (4), with blocks given by the tying scheme above, then the feed-forward layer σ ( W vec( (cid:88) )) is an Equivariant EntityRelationship Layer (Deﬁnition 1).
This theorem assures us that a layer constructed with our parameter-sharing scheme achieves equivariace w.r.t.
the exchangeabilities of the relational data.
However, one may wonder whether an alternative, more expressive parameter-sharing scheme (with the same form of output) may be possible.
The following theorem asserts that this is not the case, and that our model is the most expressive parameter-tying scheme possible for an EERL.
Theorem 4.2.
Let (cid:88) =
{ X (cid:82) | (cid:82) ∈ R } be the tensor representation of some relational data.
If
W ∈ R N × N is any parameter matrix other than the one deﬁned above, then the feed-forward layer σ ( W vec( (cid:88) )) is NOT an Equivariant Entity-Relationship Layer (Deﬁnition 1).
In practice, we approach this constrained layer in a much more efﬁcient way: the operations of the layer can be performed efﬁciently using pooling and broadcasting over tensors; see Appendix E.
E XPERIMENTS 5. S YNTHETIC D ATA
To continue with our running example we synthesize a toy dataset, restricted to (cid:82) 1 = { 1 , 2 } ( STUDENTCOURSE ), (cid:82) 2 = { 1 , 3 } ( STUDENT - PROFESSOR ), and (cid:82) 3 = { 2 , 3 } ( PROFESSOR - COURSE ).
4 Each matrix
[Figure] [Figure] X { d 1 ,d 2 } ∈ R N d 1 × N d 2 in the relational database, (cid:88) =
(cid:104) X { 1 , 2 } , X { 1 , 3 } , X { 2 , 3 } (cid:105) , is produced by ﬁrst uniformly sampling an h-dimensional embedding for each entity instance Z d ∈ R N d × h , followed by matrix product X { d 1 ,d 2 } :
=
Z d 1 Z d 2 T .
A sparse subset of these matrices are observed by our model in the following experiments.
Note that while we use a simple matrix product to generate the content of tables from latent factors, the model is oblivious to this generative process.
We use a factorized auto-encoding architecture consisting of a stack of EERN followed by pooling that produces code matrices Z d ∈ R N d × h (cid:48) ∀ d ∈ { 1 , 2 , 3 } for each entity, STUDENT , COURSE and PROFESSOR .
The code is then fed to a decoding EERN to reconstruct the sparse vec( (cid:88) ) .
For all experiments, the encoder consists of 7 EERLs, each with 64 hidden units, each using batch normalization (Ioffe & Szegedy, 2015) and channel-wise dropout.
We then apply mean pooling to produce encodings.
We found that batch normalization dramatically sped up the training procedure.
See Appendix H for more details.
Once we ﬁnish training the model on a relational dataset, we can apply it to another instantiation — that is a dataset with completely different STUDENTS , COURSES and PROFESSORS .
Embedding and Missing Record Prediction.
We use a small embedding dimension h = h (cid:48)
=for visualization of COURSE embeddings.
Fig. 3(a) and (b) suggest that the learned embeddings agree with the ground truth.
The structure within the data points has been preserved: in (a), points with a small x -coordinate in the ground-truth embeddings have a small x -coordinate in the learned embeddings (indicated by the darker color), and similarly for the y -coordinate and for (b).
Note that in the best case, the two embeddings can agree up to a diffeomorphism.
Next, we set out to predict missing records in the STUDENT - COURSE table using observed data from the whole database.
For this, the factorized auto-encoding architecture is trained to only minimize the reconstruction error for “observed” entries in the STUDENT - COURSE tensor.
We use N 1 = N 2 = N 3 = 200 , with an encoding size of 10.
To study the model’s behaviour, we repeat the experiment while changing two quantities: 1) the percentage of observed entries ( i.e. , sparsity of all database tensors) at training time; and 2) the sparsity of test-time observations, after the model is trained.
Fig. 3(c) visualizes the average prediction error, conﬁrming our expectation that the amount of data seen during both training and testing can increase the prediction accuracy.
More importantly, once the model is trained at a particular sparsity level, it shows robustness to changes in the sparsity level at test time.
Inductive Setting.
Once we ﬁnish training the model on a relational dataset, we can apply it to another instantiation — that is a dataset with completely different STUDENTS , COURSES and PROFESSORS .
Note that this is possible because the unique values in W do not grow with the number of instances of each entity in our database – i.e. , we can resize W by repeating its pattern to ﬁt the dimensionality of our data.
In practice, since we use pooling-broadcasting operations, this resizing is implicit.
Fig. 3(b) reproduces the embedding experiment, and Fig. 3(d) shows the missing record prediction results for the inductive setting.
In both cases the model performs well when applied to a new database.
This setting can have interesting real-world applications, as it enables transfer learning across databases and allows for predictive analysis without training for new entities in a database as they become available.
The Value of Side Information.
Do we gain anything by using the “entire” database for predicting missing entries of a particular table, compared to simply using the target tensor X { 1 , 2 } ( STUDENT - COURSE table) for both training and testing?
To answer this question, we ﬁx the sparsity level of the STUDENTCOURSE table at 0.1%, and train models with increasing levels of sparsity for other tensors
X { 1 , 3 } and X { 2 , 3 } in the range [0 . 025 , 0 . 7] .
This ﬁgure shows that the side information in the form of STUDENT - PROFESSOR and COURSE - PROFESSOR tables can signiﬁcantly improve the prediction of missing records in the STUDENT - COURSE table.
5. P REDICTING ON S OCCER D ATABASE
We use the European Soccer database to build a simple relational model with three entities: PLAYER , TEAM and MATCH .
The database contains information for about 11,000 PLAYERS , 300 TEAMS and 25,000 MATCHES in European soccer leagues.
We extract a
COMPETES - IN relation between TEAMS and MATCHES , as well as a PLAYS - IN relation between PLAYERS and MATCHES that identiﬁes which players played in each match.
Our objective is to predict whether the outcome of a match was Home Win , Away Win , or Draw .
A simple baseline is to always predict Home Win , which obtains 46% accuracy.
By engineering features from temporal statistics (such as the result of recent games for a team relative to a particular target match, recent games two teams played against each other, as well as recent goal statistics) the best model reported on Kaggle achieve 56% accuracy.
Without using any temporal data, by simply taking the average for any such time series, our model achieves 53% accuracy.
This also matches the accuracy of professional bookies.
See Appendix H for details.
R ELATED L ITERATURE
For a more detailed literature review see Appendix A.
Our work is related to a large body of work in statistical relational learning (Getoor & Taskar, 2007; Raedt et al., 2016) and knowledge-graph (KG) embedding (Nickel et al., 2016), as well as relational ( e.g. , Battaglia et al., 2018; Hamilton et al., 2017b), and equivariant ( e.g. , Cohen & Welling, 2016a; Ravanbakhsh et al., 2017b; Kondor & Trivedi, 2018;
Cohen et al., 2019) deep learning.
When dealing with a single relation, such as
STUDENT - COURSE or USER - ITEM - TAG one may use tensor factorization methods for both embedding and tensor completion.
However, with more than one relation, one needs to jointly factorize a set of tensors X (cid:82) ∈ (cid:88) , so that the factors in common between any two tensors are in agreement – e.g. , in
Fig.
1, X { 1 , 2 } and X { 1 , 3 } have a common factor, corresponding to the STUDENT entity.
Most relevant to our work in the statistical relational learning community is the relational neural network of Kazemi & Poole (2017).
However, they focus on a single relation and the proposed model is more directly comparable to Hartford et al.
See Appendix A.1.1 for discussion.
EERL generalizes several equivariant layers proposed for structured domains: Our model reduces to (equivariant model in)
Deep-Sets (Zaheer et al., 2017) when we have a single relation (cid:82) = { 1 } with a single entity – i.e. , D = 1 ; i.e. , a set of instances.
Hartford et al.
(2018) consider a more general setting of interaction across different sets, such as user-tag-movie relations.
EERLs reduce to their parameter-sharing when we have a single relation R = { (cid:82) } with multiple entities D ≥ 1 , where all entities appear only once.
Maron et al.
(2018) further relax the assumption above, and allow for multiset relations.
Intuitively, this form of relational data can model the interactions within and between sets; for example interaction within nodes of a graph is captured by an adjacency matrix, corresponding to D = 1 and (cid:82)
= { 1 , 1 } .
Our model reduces to this scheme when restricted to a single relation.
For detailed discussion on these relations see Appendix A.
C ONCLUSION AND F UTURE W ORK
We have outlined a novel and principled approach to deep learning with relational data(bases).
In particular, we introduced a simple constraint in the form of tied parameters for the standard feedforward layer and proved that any other tying scheme either ignores the exchangeabilities of relational data or can be obtained by further constraining our model.
The proposed layer can be applied in inductive settings, where the relational databases used during training and test have no overlap.
While our model enjoys a linear computational complexity in the size of the database, we have to overcome one more hurdle before applying this model to large-scale real-world databases: relational databases often hold large amount of data, and in order for our model to be applied in these settings we need to perform mini-batch sampling.
However, any such sampling has the effect of sparsifying the observed relations.
A careful sampling procedure is required that minimizes this sparsiﬁcation for a particular subset of entities or relations.
While several recent works propose solutions to similar problems on graphs and tensors ( e g. , Hamilton et al., 2017a; Hartford et al., 2018;
Ying et al., 2018;
Eksombatchai et al., 2017; Chen et al., 2018; Huang et al., 2018) we leave this important direction for relational databases to future work
A A M ORE D ETAILED R EVIEW OF R ELATED L ITERATURE
To our knowledge there are no similar frameworks for direct application of deep models to relational databases, and current practice is to automate feature-engineering for speciﬁc prediction tasks (Lam et al., 2018).
A. S TATISTICAL R ELATIONAL L EARNING AND K NOWLEDGE -G RAPH E MBEDDING
Statistical relational learning extends the reach of probabilistic inference to the relational model (Raedt et al., 2016).
In particular, a variety of work in lifted inference procedures extends inference methods in graphical models to the relational setting, where in some cases the symmetry group of the model is used to speed up inference (Kersting, 2012).
Most relevant to our work from this community is the
Relational Neural Network model of Kazemi & Poole (2017); see Appendix A.1.1.
An alternative to inference with symbolic representations of relational data is to use embeddings.
In particular, Tensor factorization methods that offer tractable inference in latent variable graphical models (Anandkumar et al., 2014), are extensively used for knowledge-graph embedding (Nickel et al., 2016).
A knowledge-graph can be expressed as an ER diagram with a single relation (cid:82) =
{ 1 (1) , 2 , 1 (2) } , where 1 (1) , 1 (2) representing HEAD and TAIL entities and 2 is an entity representing the RELATION .
Alternatively, one could think of knowledge-graph as a graph representation for an instantiated ER diagram (as opposed to a set of of tables or tensors).
However, in knowledge-graphs, an entity-type is a second class citizen, as it is either another attribute, or it is expressed through relations to special objects representing different “types”.
Therefore, compared to ER diagrams, knowledge-graphs are less structured, and more suitable for representing a variety of relations between different objects, where the distinction between entity types is not key.
A.1. R ELATION TO R EL NN OF K AZEMI & P OOLE (2017)
An alternative approach explored in the statistical relational learning community includes extensions of logistic regression to relational data (Kazemi et al., 2014), and further extensions to multiple layers (Kazemi & Poole, 2017).
The focus of these works is primarily on predicting properties of the various entity instances (the example they use is predicting a user’s gender based on the ratings given to movies).
Their model works by essentially counting the number of instances satisfy a given properties, but is easiest understood by interpreting it as a series of convolution operations using row- and column-wise ﬁlters that capture these properties.
Consider Example 3 from (Kazemi & Poole, 2017) (also depicted in their Figure 4).
We have a set of USERS and a set of MOVIES , and there is a matrix L ∈ R N × M , denoting which movies where liked by which users.
As ﬁlters, they use binary vectors a ∈ R M , and o ∈ R N , representing which movies are ACTION and which users are OLD , respectively.
The task they pose is to predict the gender of a user 5 , given this information.
To do so, they include a third ﬁlter, φ ∈ R M , of learnable, “numeric latent properties”.
Each layer of their model then convolves each of these ﬁlters with the LIKES matrix, then applies a simple linear scale and shift and a sigmoid activation.
The result is three new ﬁlter vectors that can be used to make predictions or as ﬁlters in the next layer.
For one layer, the outputs are then
[Equation]
where each w is a scalar.
Observe that, for example, the n th element of La counts the number of action movies liked by user n .
Observe also that v a , v φ ∈ R N , while v o ∈ R M .
So v a and v φ can be used to make predictions about individual users.
Note that the number of parameters in their model grows both with the number of movies and with the number of layers in the network.
Application of EERL to this example, reduces the 4 parameter model of (Hartford et al., 2018).
Indeed, most discussions and “all” experiments in Kazemi & Poole (2017) assume a single relation.
For completeness, we explain EERL in this setting.
Consider the LIKES matrix, and the ACTION and
OLD ﬁlters as tables.
We predict the gender of the n th user, as
[Equation]
a
[Equation]
(cid:19)
[Equation]
The main difference between their model and ours is that they require per-item parameters (e.g., one parameter per movie), while, as can be seen from (5), the number of parameters in our model is independent of the number of instances and so does not grow with the number of users or movies (note that we have the option of adding such features to our model by having unique one-hot features for each USER and MOVIE .)
As a result, our model can be applied in inductive settings as well.
One may also draw a parallel between row and column convolution in Kazemi & Poole (2017) with two out of four pooling operations when we have single relation between two entities.
However these operations become insufﬁcient when moving to models of self-relation ( e.g. , 15 parameters for a single self-relation) and does cannot adequately capture the interaction between multiple relations as discussed in our provably optimal linear layer.
A. R ELATIONAL , G EOMETRIC AND E QUIVARIANT D EEP L EARNING
See Hamilton et al.
(2017b)
; Battaglia et al.
(2018) for a detailed review.
Scarselli et al.
(2009) introduced a generic framework that iteratively updates node embeddings using neural networks.
Li et al.
(2015) integrated this iterative process in a recurrent architecture.
Gilmer et al.
(2017) proposed a similar iterative procedure that updates node embeddings and messages between the neighbouring nodes, and show that it subsumes several other deep models for attributed graphs (Duvenaud et al., 2015;
Schütt et al., 2017; Li et al., 2015; Battaglia et al., 2016; Kearnes et al., 2016), including spectral methods.
Their method is further generalized in (Kondor et al., 2018b) as well as (Maron et al., 2018), which is in turn subsumed in our framework.
Spectral methods extend convolution to graphs (and manifolds) using eigenvectors of the Laplacian as the generalization of the Fourier basis (Bronstein et al., 2017; Bruna et al., 2014).
Simpliﬁed variations of this approach leads to an intuitive yet non-maximal parameter-sharing scheme that is widely used in practice (Kipf & Welling, 2016).
This type of simpliﬁed graph convolution has also been used for relational reasoning with knowledge-graphs (Schlichtkrull et al., 2018).
An alternative generalization of convolution is deﬁned for functions over groups (Cohen & Welling, 2016a) or more generally homogeneous spaces (Cohen et al., 2018b).
Moreover, convolution can be performed in the Fourier domain in this setting, where irreducible representations of a group become the Fourier bases (Kondor & Trivedi, 2018).
Particularly relevant to our work are the models of (Kondor et al., 2018b) and (Maron et al., 2018) that operate on graph data using an equivariant design.
Equivariant deep models for a variety of structured domains is explored in several other recent works.
( e.g. , Worrall et al., 2017; Cohen et al., 2018a;
Kondor et al., 2018a; Sabour et al., 2017; Weiler et al., 2018); see also (Cohen & Welling, 2016b; Weiler et al., 2017;
Kondor et al., 2018b; Anselmi et al., 2019).
A. P ARAMETER -S HARING , E XCHANGEABILITY AND E QUIVARIANCE
The notion of invariance is also studied under the term exchangeability in statistics (Orbanz & Roy, 2015); see also (Bloem-Reddy & Whye Teh, 2019) for a probabilistic approach to equivariance.
In graphical models exchangeability is often encoded through plate notation, where parameter-sharing happens implicitly.
In the AI community, this relationship between the parameter sharing and “invariance” properties of the network was noticed in the early days of the Perceptron (Minsky &
Papert, 2017; Shawe-Taylor, 1989; 1993).
This was rediscovered in (Ravanbakhsh et al., 2017b), where this relation was leveraged for equivariant model design.
Ravanbakhsh et al.
(2017b) show that when the group action is discrete “equivariance” to any group action can be obtained by parametersharing.
In particular, one of the procedures discussed there ties the elements of W based on the orbits of the “joint” action of the group on the rows and columns of W .
The parameter-tying scheme in the following can be obtained in this way:
A.3. R ELATION TO D EEP -S ETS OF Z AHEER ET AL . (2017)
propose an equivariant model for set data.
Our model reduces to their parameter-tying when we have a single relation (cid:82)
= { 1 } with a single entity – i.e. , D = 1 ; i.e. , a set of instances; see also Examplein Appendix E.
Since we have a single relation, W matrix has a single block W = W 1 , 1 , indexed by n 1 .
The elements of n 1 , 1 = n 1 , 1 1 = (cid:104) n 1 (1) , n 1 (2) (cid:105) index the elements of this matrix, for entity 1 (the only entity).
There are two types of equality patterns n 1 (1) = n 1 (2) , and n 1 (1) (cid:54) = n 1 (2) , giving rise to the permutation equivariant layer introduced in (Ravanbakhsh et al., 2017a; Zaheer et al., 2017).
A.3. R ELATIONP TO E XCHANGEABLE T ENSOR M ODELS OF H ARTFORD ET AL . (2018)
Hartford et al.
(2018) consider a more general setting of interaction across different sets, such as user-tag-movie relations.
Our model produces their parameter-sharing when we have a single relation
R = { (cid:82) } with multiple entities D ≥ 1 , where all entities appear only once – i.e. , κ ( d ) = 1 ∀ d ∈ (cid:82) .
Here, again W = W 1 , 1 , and n 1 , 1 , the concatenation of row-index n 1 and column index n 1 , identiﬁes an element of this matrix.
Since each d ∈ (cid:82) has multiplicity 1 , n 1 , 1 d = (cid:104) n 1 , 1 d (1) , n 1 , 1 d (2) (cid:105) ∀ d ∈ [ D ] , and therefore n 1 , 1 d can have two class of equality patterns.
This gives 2 D equivalence classes for n 1 , 1 , and therefore 2 D unique parameters for a rank D exchangeable tensor.
A.3. R ELATIONSHIP TO E QUIVARIANT G RAPH N ETWORKS OF M ARON ET AL . (2018)
Maron et al.
(2018) further relax the assumption of Hartford et al.
(2018), and allow for κ ( d ) ≥ 1 .
Intuitively, this form of relational data can model the interactions within and between sets; for example interaction within nodes of a graph is captured by an adjacency matrix, corresponding to
D = 1 and (cid:82) = { 1 , 1 } .
This type of parameter-tying is maximal for graphs, and subsumes the parameter-tying approaches derived by simpliﬁcation of Laplacian-based methods.
When restricted to a single relation, our model reduces to the model of (Maron et al., 2018); however, when we have multiple relations, W i,j for j (cid:54)
=
i , our model captures the interaction between different relations / tensors.
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation] [Figure] [Equation] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table] B M ULTISET R ELATIONS
Because we allow a relation (cid:82) to contain the same entities multiple times, we formally deﬁne a multiset as a tuple (cid:101) (cid:82) =
(cid:104) (cid:82) , κ (cid:105) , where (cid:82) is a set, and κ : (cid:82) → N maps elements of (cid:82) to their multiset counts.
We will call (cid:82) the elements of the multiset (cid:101) (cid:82) =
(cid:104) (cid:82) , κ (cid:105) , and κ ( d ) the count of element d .
We deﬁne the union and intersection of two multisets (cid:82) 1 and (cid:82) 2 as (cid:101) (cid:82) 1 ∪ (cid:101) (cid:82) 2 = (cid:104) (cid:82) 1 ∪ (cid:82) 2 , κ 1 + κ 2 (cid:105) and (cid:101) (cid:82) 1 ∩ (cid:101) (cid:82) 2 = (cid:104) (cid:82) 1 ∩ (cid:82) 2 , min { κ 1 , κ 2 }(cid:105) .
In general, we may also refer to a multiset using typical set notation ( e.g. , (cid:82) = { d 1 , d 1 , d 2 } ).
We will use bracketed superscripts to distinguish distinct but equal members of any multiset ( e.g. , (cid:82) =
{ d 1 , d 1 , d 2 } = { d (1)1 , d (2)1 , d (1)2 } ).
The ordering of equal members is speciﬁed by context or arbitrarily.
The size of a multiset (cid:101) (cid:82) accounts for multiplicities: | (cid:101) (cid:82) | = (cid:80)
d ∈ (cid:82) κ ( d ) .
C N UMBER OF FREE PARAMETERS
For the multiset relations (cid:101) (cid:82) (cid:105) = (cid:104) (cid:82) (cid:105) , κ (cid:105) and (cid:102) (cid:82) (cid:106) =
(cid:104) (cid:82) (cid:106) , κ (cid:105) , recall that two parameters W i,j n i , n j and
W i,j m i , m j are tied together if n i,j , the concatenation of n i with n j , is in the same equivalence class as m i,j .
We partition each n i,jd into sub-partitions (cid:80) ( n i,jd ) .
= { (cid:80) 1 , . .
, (cid:80) L } of indices whose values are equal, and consider n i,j and m i,j to be equivalent if their partitions are the same for all d :
[Equation]
i
[Equation]
See 4.1 for details.
This means that the total number of free parameters in W i,j is the product of the number of possible different partitionings for each unique entity d ∈ (cid:82) i ∪ (cid:82) j :
[Equation]
(7) where w i,j is the free parameter vector associated with W i,j , and b ( κ ) is the κ th Bell number, which counts the possible partitionings of a set of size κ .
Example 2.
[Fig. 2(a)] Consider again the simpliﬁed version of the relational structure of Fig.
1, restricted to the three relations (cid:82)
= { 1 , 2 } , self-relation (cid:82) 2 = { 2 , 2 } , and R 3 = { 1 , 3 } with N 1 =
STUDENTS , N 2 = 4 COURSES , and N 3 = 3 PROFESSORS .
We use tuple n 1 = (cid:104) n 11 , n 12 (cid:105) ∈
[ N 1 ] × [ N 2 ] to index the rows and columns of W 1 , 1 .
We also use n 1 to index the rows of W 1 , 2 , and use n 2 = (cid:104) n 22 (1) , n 22 (2)
(cid:105) ∈
[ N 2 ] × [ N 2 ] to index its columns.
Other blocks are indexed similarly.
The elements of W 1 , 1 take b (2) b (2) =
4 different values, depending on whether or not n 11 = n 1 (cid:48) 1 and n 12 = n 1 (cid:48) 2 , for row index n 1 and column index n 1 (cid:48) (where b ( k ) is the k-th Bell number).
The elements of W 1 , 2 take b (1) b (3) =
5 different values: the index n 11 can only be partitioned in a single way ( b (1) = 1 ).
However index n 12 and indices n 22 (1) and n 22 (2) all index into the COURSES table, and so can each potentially refer to the same course.
We thus have a unique parameter for each possible combination of equalities between these three items, giving us a factor of b (3)
= 5 different parameter values; see Fig. 2(a), W 1 , 1 is the upper left block, and W 1 , 2 is the block to its right.
The center block of Fig. 2(a), W 2 , 2 produces the effect of (cid:82) 2 = { 2 , 2 } on itself.
Here, all four index values could refer to the same course, and so there are b (4) = 15 different parameters.
D B IAS P ARAMETERS
For full generality, our deﬁnition of EERL could also include bias terms without affecting its exchangeability properties.
We exclude these in the statements of our main theorems for the sake of simplicity, but discuss their inclusion here for completeness.
For each relation (cid:82) i = { d 1 , ..., d | (cid:82) i | } , we deﬁne a bias tensor B i ∈
R N d 1 × ...
× N d | (cid:82) i | .
The elements of B i are tied together in a manner similar to the tying of elements in each W i,j : Two elements B i n i and B
i m i are tied together iff n i ≡ n j , using the deﬁnition of equivalence from Section 4.1.
Thus, we have a vector of additional free parameters b i for each relation (cid:82) i , where
[Equation]
Consistent with our previous notation, we deﬁne (cid:66) =
{ B i | (cid:82) i ∈ R } , and vec( (cid:66) )
= (cid:104) vec( B 1 ) , ..., vec( B | R | )
[Equation]
Then an EERL with bias terms is given by
[Equation]
The following Claim asserts that we can add this bias term without affecting the desired properties of the EERL.
Claim 2.
If σ (cid:0) W vec( (cid:88) ) (cid:1) is an EERL, then σ (cid:0) W vec( (cid:88) )
+ vec( (cid:66) ) (cid:1) is an EERL.
The proof (found in Appendix G.1) argues that, since σ (cid:0) W vec( (cid:88) ) (cid:1) is an EERL, we just need to show that G (cid:88) vec( (cid:66) )
=
vec( (cid:66) )
iff G (cid:88) ∈ (cid:71) (cid:88) , which holds due to the tying of patterns in each B i .
E S IMPLIFICATIONS FOR M ODELS WITHOUT S ELF -R ELATIONS
In the special case that the multi relations (cid:101) (cid:82) i and (cid:101) (cid:82) j are sets — i.e. , have no self-relations— then the parameter tying scheme of Section 4.1 can be simpliﬁed considerably.
In this section we address some nice properties of this special setting.
E. E FFICIENT I MPLEMENTATION U SING S UBSET -P OOLING
Due to the particular structure of W when all relations contain only unique entities, the operation
W vec( (cid:88) ) in the EERL can be implemented using (sum/mean) pooling operations over the tensors
X (cid:82) for (cid:82) ∈ R , without any need for vectorization, or for storing W directly.
For X i ∈ R N d 1 × ...
× N d | (cid:82) | and (cid:83) ⊆
(cid:82) i , let pool( X i , (cid:83) ) be the summation of the tensor X i over the dimensions speciﬁed by (cid:83) .
That is, pool( X i , (cid:83) )
∈ R N d (cid:48) 1 × ...
× N d (cid:48)| (cid:82) i |−| (cid:83) | where (cid:82) i \ (cid:83) =
{ d (cid:48) 1 , . . .
, d (cid:48)| (cid:82) |−| (cid:83) | } .
Then we can write element n i in the i -th block of W vec( (cid:88) ) as
[Equation]
where n i (cid:82) i \ (cid:83) is the restriction of n i to only elements indexing entities in (cid:82) i \ (cid:83) .
This formulation lends itself to a practical, efﬁcient implementation where we simply compute each pool( X i , (cid:83) ) term and broadcast-add them back into a tensor of appropriate dimensions.
E. O NE - TO -O NE AND O NE - TO -M ANY R ELATIONS
In the special case of a one-to-one or one-to-many relations ( e.g. , in Fig.
1, one PROFESSOR may teach many COURSES , but each COURSE has only one PROFESSOR ), we may further reduce the number of parameters due to redundancies.
Suppose (cid:82) i ∈
R is some relation, and entity d ∈ (cid:82) i is in a one-to- relation with the remaining entities of (cid:82) i .
Consider the 1D sub-array of (cid:88) (cid:82) i obtained by varying the value of n id while holding the remaining values n i (cid:82) i \{ d } ﬁxed.
This sub-array contains just a single non-zero entry.
According to the tying scheme described in Section 4.1, the parameter block W i,i will contain unique parameter values w (cid:82) i and w (cid:82) i \{ d } .
Intuitively however, these two parameters capture exactly the same information, since the sub-array obtained by ﬁxing the values of (cid:82) i \{ d } contains exactly the same data as the sub-array obtained by ﬁxing the values of (cid:82) i ( i.e. , the same single value).
More concretely, to use the notation of Appendix E.1, we have pool( X i , (cid:82) i \{ d } ) n i { d } = pool( X i , (cid:82) i ) in (10), and so we may tie w i,i (cid:82) i and w i,i (cid:82) i \{ d } .
In fact, we can reduce the number of free parameters in the case of self-relations ( i.e. , relations with non-unique entities) as well in a similar manner.
E. R ECURSIVE D EFINITION OF THE W EIGHT M ATRIX
We are able to describe the form of the parameter matrix W i,j concisely in a recursive fashion, using
Kronecker products: For any N ∈ N , let 1 N ∈ R N × N be the N × N matrix of all ones, and I N the N × N identity matrix.
Given any set of (unique) entities (cid:83) =
{ d 1 , ..., d | (cid:83) | } ⊆ { 1 , .., D } , for k = 1 , ..., | (cid:83) | , recursively deﬁne the sets
[Equation]
with the base case of (cid:87) (cid:83)
= R .
Then for each block W i,j of (4) we simply have W i,j ∈ (cid:87) (cid:82) i ∩ (cid:82) j | (cid:82) i ∩ (cid:82) j | .
Writing the blocks of the matrix (4) in this way makes it clear why block W i,j contains 2 | (cid:82) i ∩ (cid:82) j | unique parameter values in the case of distinct entities: at each level of the recursive deﬁnition we are doubling the total number of parameters by including terms from two elements from the level below.
It also makes it clear that the parameter matrix for a rank- k tensor is built from copies of parameter matrices for rank- ( k − 1) tensors.
Example 3.
In the simple case where we have just one relation and one entity, (cid:82) = { d } and the parameter matrix is an element of (cid:87) (cid:82) 1 = { w ⊗ 1 N d + v ⊗
I N d | w, v ∈ R } , which matches the parameter tying scheme of Zaheer et al.
If instead we have a single relation with two distinct entities (cid:82) = { d 1 , d 2 } , then the parameter matrix is an element of (cid:87) (cid:82)
= { W ⊗ 1 N d 2 + V ⊗ I N d 2 | W , V ∈ (cid:87) R 1 } , which matches the tying scheme of Hartford et al.
F U SING M ULTIPLE C HANNELS
Equivariance is maintained by composition of equivariant functions.
This allows us to stack EERLs to build “deep” models that operate on relational databases.
Using multiple input ( K ) and output ( K (cid:48) ) channels is also possible by replacing the parameter matrix W ∈ R N × N , with the parameter tensor
W ∈ R K × N × N × K (cid:48) ; while K × K (cid:48) copies have the same parameter-tying pattern — i.e. , there is no parameter-sharing “across” channels.
The single-channel matrix-vector product in σ ( W vec( (cid:88) ))
where ( W vec( (cid:88) ))
n = (cid:80) n (cid:48)
W n , n (cid:48) vec( (cid:88) ) n (cid:48) is now replaced with contraction of two tensors ( W vec( (cid:88) )) n ,k (cid:48)
= (cid:80) n (cid:48) ,k ∈
[ K ] W k, n , n (cid:48) ,k (cid:48) vec( (cid:88) ) n (cid:48) ,k , for k (cid:48) ∈
[ K (cid:48) ] .
G P ROOFS
Observe that for any index tuple n , we can express (cid:80) ( n d ) (Section 4.1) as
[Equation]
We will make use of this formulation in the proofs below.
G. P ROOF OF C LAIM
Proof.
We want to show that
[Equation]
iff G (cid:88) ∈ (cid:71) (cid:88) .
Since σ (cid:0) W vec( (cid:88) ) (cid:1) is an EERL, this is equivalent to showing
[Equation]
[Equation]
( ⇐ = ) Suppose G (cid:88) ∈ (cid:71) (cid:88) , with (cid:71) (cid:88) deﬁned as in Claim 1.
Fix some relation (cid:101) (cid:82) (cid:105) and consider the i -th block of G (cid:88) vec( (cid:66) ) :
[Equation]
[Equation] [Equation]
Where G (cid:88) =
diag( K 1 , ..., K | R | ) , and n i ∗ is the unique index such that K i n i , n i ∗ = 1 .
As above, let n id be the restriction of n i to elements indexing entity d .
Then we want to show that (cid:80) ( n id ) =
(cid:80) ( n i ∗ d ) for all d ∈ (cid:82) i .
Now, since G (cid:88) ∈ (cid:71) (cid:88) and K i n i , n i ∗ =
1 we have
G dn i
d
[Equation]
,n i
[Equation]
(18) for all d ∈ (cid:82) i and all k .
That is g d (cid:0) n i ∗ d ( k ) (cid:1) =
n i d ( k ) .
Consider (cid:83) ∈ (cid:80) ( n i d ) .
we have
[Equation] [Equation]
[Equation] [Equation]
So (cid:80) ( n id ) ⊆
(cid:80) ( n i ∗ d ) .
A similar argument has (cid:80) ( n id ) ⊇ (cid:80) ( n i ∗ d ) .
Thus, we have vec( B i ) n i ∗ = vec( B i ) n i , which completes the ﬁrst direction.
(= ⇒ )
Let G (cid:88) vec( (cid:66) )
=
vec( (cid:66) ) .
First, suppose for the sake of contradiction that G (cid:88) (cid:54)∈ (cid:76) (cid:82) ∈ R (cid:83) N (cid:82) and consider dividing the rows and columns of G (cid:88) into blocks that correspond to each relation (cid:82) i .
Then since G (cid:88) (cid:54)∈ (cid:76) (cid:82) ∈ R (cid:83) N (cid:82) , there exist (cid:82) i , (cid:82) j ∈ R , with i (cid:54) =
j and n i ∈
[ N d i 1 ] × ... ×
[ N d i | (cid:82) i | ] and n j ∈ [ N d j 1 ] × ... ×
[ N d j | (cid:82) j | ] such that G (cid:88) maps ( i, n i ) to ( j, n j ) .
That is g (cid:88) (cid:0) ( i, n i ) (cid:1) =
( j, n j ) and thus G (cid:88) ( j, n j ) , ( i, n i ) = 1 .
So
[Equation] [Equation]
[Equation] [Equation]
25) by the deﬁnition of (cid:66) .
And so G (cid:88) vec( (cid:66) )
(cid:54) =
vec( (cid:66) ) .
Next, suppose G (cid:88) ∈ (cid:76) (cid:82) ∈ R (cid:83) N (cid:82) .
Then for all (cid:82) i , there exist K i ∈ (cid:83) N i such that
G (cid:88) =
diag( K 1 , ..., K | (cid:82) i | ) .
For any n i we have
[Equation] [Equation]
Where n i ∗ is the unique index such that K i n i , n i ∗ = 1 .
That is k
i maps n i ∗ to n i .
Then by the deﬁnition of B i we have
[Equation]
29) for all d ∈ (cid:82) i .
(29) says that for each d the action of K i on elements of n i ∗ is determined only by the values of those elements, not by the values of elements indexing other entities, and so
K i ∈ (cid:78) d ∈ (cid:82) i (cid:83) N d .
But (29) also says that for all k, l
[Equation]
which says that the action of K i is the same across any duplications of d ( i.e. , d ( k ) and d ( l ) ), and so
K i = (cid:78) d ∈ (cid:82) i G d , for some ﬁxed G d , and therefore G (cid:88) ∈ (cid:71) (cid:88) .
G. L EMMA 1 AND P ROOF
To prove our main results about the optimality of EERLs we require the following Lemma.
Lemma 1.
For any permutation matrices K i ∈ (cid:83) N i and K j ∈ (cid:83) N j we have
[Equation]
for constrained W i,j as deﬁne above.
That is K i and K j should separately permute the instances of each entity in the multisets (cid:101) (cid:82) i and (cid:101) (cid:82) j , applying the same permutation to any duplicated entities, as well as to any entities common to both (cid:101) (cid:82) i and (cid:101) (cid:82) j .
To get an intuition for this lemma, consider the special case of i = j .
In this case, the claim is that
W i,i commutes with any permutation matrix that is of the form K i = (cid:78)
d ∈ (cid:82) i G d .
This gives us the kind of commutativity we desire for an EERL, at least for the diagonal blocks of W .
Equivalently, commuting with K i means that applying permutation
K i to the rows of W i,i has the same effect as applying K i to the columns of W i,i .
In the case of i (cid:54) =
j , ensuring that K i and K j are deﬁned over the same underlying set of permutations, { G d ∈ (cid:83) N d | d ∈ (cid:82)
i ∪ (cid:82) j } , ensures that permuting the rows of W i,j with K
i has the same effect as permuting the columns of W i,j with K j .
It is this property that will allow us to show that a network layer deﬁned using such a parameter tying scheme is an EERL.
See Fig.
2 for a minimal example, demonstrating this lemma.
We require the following technical Lemma for the proof of Lemma 1.
Lemma 2.
Let (cid:82) i , (cid:82) j ∈ R , and for each d ∈ [ D ] let G d ∈ (cid:83) N d .
If G dn id ( k ) ,n i (cid:48) d ( k ) =
1 for all d ( k ) ∈ (cid:82) i with d ( k ) =
d , and G dn j (cid:48)
d ( k ) ,n jd ( k ) = 1 for all d ( k ) ∈ (cid:82) j with d ( k ) =
d , then for all d ( k ) ∈ (cid:82) i and d ( l ) ∈ (cid:82) j with d ( k ) =
d ( l ) =
d , we have n i (cid:48)
d ( k ) =
n jd ( l ) ⇐⇒ n id ( k ) =
n j (cid:48)
d ( l ) .
Proof.
Suppose G d n id ( k ) ,n i (cid:48)
d ( k ) =
1 for all d ( k ) ∈ (cid:82) i , and G dn j (cid:48)
d ( k ) ,n j d ( k ) = 1 for all d ( k ) ∈ (cid:82) j .
We prove the forward direction (= ⇒ ) .
The backward direction follows from an identical argument.
Fix some d ( k ) ∈ (cid:82) i and d ( l ) ∈ (cid:82) j and suppose n i (cid:48)
d ( k ) =
n jd ( l ) .
By assumption we have G dn id ( k ) ,n i (cid:48)
d ( k ) =
and so
[Equation]
Similarly, we have G dn j (cid:48)
d ( l ) ,n jd ( l ) =
and so
[Equation]
But n i (cid:48)
d ( k ) =
n jd ( l ) and substituting into (31) we have
[Equation]
And combining (32) and (33) gives n id ( k ) =
n j (cid:48)
d ( l ) , as desired.
We are now equipped to prove our main claims, starting with Lemma 1:
Proof.
( ⇐ = ) Let (cid:101) (cid:82) (cid:105) =
{ d i 1 , ..., d i | (cid:102) (cid:82) (cid:105) | } and (cid:102) (cid:82) (cid:106) =
{ d j 1 , ..., d j | (cid:102) (cid:82) (cid:106) | } and ﬁx some { G d ∈ (cid:83) N d | d ∈ (cid:82) i ∪ (cid:82) j } .
We index the rows of W i,j , and the rows and columns of K i , with tuples n i , n i (cid:48) ∈
[ N d i 1 ] × ... ×
[ N d i | (cid:82) i | ] .
Similarly, the columns of W i,j , and rows and columns of K j , are indexed with tuples n j , n j (cid:48) ∈
[ N d j 1 ] × ...
×
[ N d j | (cid:82) j | ] .
Since K i = (cid:78) d ∈ (cid:82) i G d
we have
[Equation]
And thus,
[Equation]
The same is true for (cid:82) j .
That is
[Equation]
Now, ﬁx some n i and n j .
Since K i is a permutation matrix, and so has only one 1 per row, we have
[Equation]
36) where n i ∗ is the (unique) element of [ N d i 1 ] × ... ×
[ N d i | Ri | ] which satisﬁes G dn id ( k ) ,n i ∗ d ( k )
=
for all d ( k ) ∈ (cid:101) (cid:82) (cid:105) with d ( k )
=
d .
Similarly,
[Equation]
37) where n j ∗ is the (unique) element of [ N d j 1 ] × ... ×
[ N d j | (cid:82) j | ] which satisﬁes G dn j ∗ d ( k ) ,n jd ( k ) = 1 for all d ( k ) ∈ (cid:102) (cid:82) (cid:106) with d ( k )
=
d .
We want to show that (cid:80) ( n i ∗ ,j d )
=
(cid:80) ( n i,j ∗ d ) for all d ∈ (cid:82) i ∪ (cid:82) j .
Fix d ∈ (cid:82) i ∪ (cid:82) j and let (cid:101) (cid:83) ∈
(cid:80) ( n i ∗ ,j d ) .
Then (cid:101) (cid:83) =
{ d (1) , ..., d ( K ) } , where d ( k )
=
d for all d ( k ) ∈ (cid:101) (cid:83) , and for all d ( k ) , d ( l ) ∈ (cid:101) (cid:83) , n i ∗ ,j d ( k ) =
n i ∗ ,j d ( l ) .
Then by Lemma 2 we have n i,j ∗ d ( k ) =
n i,j ∗ d ( l ) , and so (cid:101) (cid:83) ∈
(cid:80) ( n i,j ∗ d ) .
So we have (cid:80) ( n i ∗ ,j d ) ⊆
(cid:80) ( n i,j ∗ d ) , and the other containment follows identically by symmetry.
So n i ∗ ,j ≡ n i,j ∗ by our deﬁnition in Section 4.1, and so W i,j n i ∗ , n j = W i,j n i , n j ∗ and by (36) and (37) above,
K
i W i,j = W
i,j K j .
( = ⇒ )
Suppose K
i W i,j = W
i,j K j .
Fix some n i , n j .
Let n i ∗ be the unique index such that
K i n i , n i ∗ = 1 , and n j the unique index such that K j n j ∗ , n j = 1 .
Then
[Equation] [Equation]
and so (cid:80) ( n i ∗ ,j d ) =
(cid:80) ( n i,j ∗ d ) for all d ∈ (cid:82) i ∪ (cid:82) j .
But this implies that (cid:80)
[Equation]
(39) says that for each d the action of K i on elements of n i ∗ is determined only by the values of those elements, not by the values of elements indexing other entities, and so K i ∈ (cid:78) d ∈ (cid:82) i (cid:83) N d .
But (39) also means that for all k, l
[Equation]
which says that the action of K i is the same across any duplications of d ( i.e. , d ( k ) and d ( l ) ), and so
K i = (cid:78) d ∈ (cid:82) i G d , for some ﬁxed G d .
Similarly,
[Equation]
which shows that K j = (cid:78) d ∈ (cid:82) j G N d (cid:48) .
Finally, since (cid:80) ( n i ∗ ,j d ) =
(cid:80) ( n i,j ∗ d ) , we also have
[Equation] [Equation]
for all k, l , which means
[Equation] [Equation]
(44) says that K i and K j apply the same permutations to all duplications of any entities they have in common, and so G d = G d (cid:48) , which completes the proof.
G. P ROOF OF T HEOREM 4.
Proof.
Let G (cid:88) ∈ (cid:83) N and (cid:71) (cid:88) be deﬁned as in Claim 1.
We need to show that
G (cid:88) σ ( W vec( (cid:88) ))
=
σ ( WG (cid:88) vec( (cid:88) ))
iff G (cid:88) ∈ (cid:71) (cid:88) for any assignment of values to the tables (cid:88) .
We prove each direction in turn.
(= ⇒ )
We prove the contrapositive.
Suppose G (cid:88) (cid:54)∈ (cid:71) (cid:88) .
We ﬁrst show that G (cid:88) W (cid:54) =
WG (cid:88) and then that G (cid:88) σ ( W vec( (cid:88) )) (cid:54) =
σ ( WG (cid:88) vec( (cid:88) )) for an appropriate choice of (cid:88) .
There are three cases.
First, suppose G (cid:88) (cid:54)∈ (cid:76) (cid:82) ∈ R (cid:83) N (cid:82) and consider dividing the rows and columns of G (cid:88) into blocks that correspond to the blocks of W .
Then since G (cid:88) (cid:54)∈ (cid:76) (cid:82) ∈ R (cid:83) N (cid:82) , there exist (cid:82) i , (cid:82) j ∈ R , with i (cid:54) =
j and n i ∈
[ N d i 1 ] × ... ×
[ N d i | (cid:82) i | ] and n j ∈ [ N d j 1 ] × ... ×
[ N d j | (cid:82) j | ] such that G (cid:88) maps ( i, n i ) to ( j, n j ) .
That is g (cid:88) (cid:0) ( i, n i ) (cid:1) =
( j, n j ) and thus, G (cid:88) ( j, n j ) , ( i, n i ) = 1 .
And so
[Equation]
by the deﬁnition of W .
Similarly,
[Equation]
But W i,i n i , n i (cid:54) =
W j,j n j , n j since i (cid:54) =
j .
And so G (cid:88) W (cid:54) =
WG (cid:88) .
Next, suppose G (cid:88) ∈ (cid:76) (cid:82) ∈ R (cid:83) N (cid:82) , but G (cid:88) (cid:54)∈ (cid:76) (cid:82) ∈ R (cid:78) d ∈ (cid:82) (cid:83) N d and consider the diagonal blocks of G (cid:88) WG (cid:88) T that correspond to those of W .
If G (cid:88) ∈ (cid:76) (cid:82) ∈ R (cid:83) N R then it is block diagonal with blocks corresponding to each (cid:82) ∈ R .
But since G (cid:88) (cid:54)∈ (cid:76) (cid:82) ∈ R (cid:78) d ∈ (cid:82) (cid:83) N d , there exists some (cid:82) i ∈
R such that the i th diagonal block of G (cid:88) is not of the form (cid:78) d ∈ (cid:82) i G d for any G d .
Then by
Lemma 1 we will have inequality between G (cid:88) WG (cid:88) T and W in the i th diagonal block.
Finally, suppose G (cid:88) ∈ (cid:76) (cid:82) ∈ R (cid:78) d ∈ (cid:82) (cid:83) N d .
Then G (cid:88) =
K 1 ⊕ ...
⊕ K | R | , where K i ∈ (cid:78) d ∈ (cid:82) (cid:83) N d for all i .
Since G (cid:88) (cid:54)∈ (cid:71) (cid:88) , there exist (cid:101) (cid:82) (cid:105) , (cid:102) (cid:82) (cid:106) ∈ R , possibly with i = j , and a d ∗ ∈ (cid:82)
i ∩ (cid:82) j such that
[Equation]
, and
[Equation]
but G d ∗ ( k ) (cid:54) =
G d ∗ ( l ) .
Since G d ∗ ( k ) (cid:54) =
G d ∗ ( l ) there exists n ∈ [ N d ∗ ] with g d ∗ ( l ) ( n ) (cid:54) =
g d ∗ ( l ) ( n ) .
Pick some n i and n j with n id ∗ = n jd ∗ = n .
Let n i ∗ be the result of applying K i to n i and n j ∗ the result of applying K j to n j .
Then we have
[Equation] [Equation]
and
[Equation]
Now, by construction we have n id ∗ = n jd ∗ , but n i ∗ d ∗ (cid:54) =
n j ∗ d ∗ .
So (cid:80) ( n i,jd ∗ ) (cid:54) =
(cid:80) ( n i ∗ ,j ∗ d ∗ ) and therefore
W i,j n i , n j (cid:54) =
W i,j n i ∗ , n j ∗ .
And so by (45) and (46) we have G (cid:88) W (cid:54) =
WG (cid:88) .
And so in all three cases G (cid:88) W (cid:54) =
WG (cid:88) .
Thus, there exists some (cid:88) , for which we have
G (cid:88) W vec( (cid:88) )
(cid:54) =
WG (cid:88) vec( (cid:88) ) .
Since σ is strictly monotonic, we have σ ( G (cid:88) W vec( (cid:88) ))
(cid:54) =
σ ( WG (cid:88) vec( (cid:88) )) .
And since σ is element wise we have G (cid:88) σ ( W vec( (cid:88) ))
(cid:54) =
σ ( WG (cid:88) vec( (cid:88) )) , which proves the ﬁrst direction.
( ⇐ =) Suppose G (cid:88) ∈ (cid:71) (cid:88) .
That is, for all d ∈ [ D ] , let G d ∈ (cid:83)
N
d be some ﬁxed permutation of N d objects and let G (cid:88) =
(cid:76) (cid:82) ∈ R (cid:78) d ∈ (cid:82) G d .
Observe that G (cid:88) is block-diagonal.
Each block on the diagonal corresponds to an (cid:82) ∈ R and is a Kronecker product over the matrices G d for each d ∈ (cid:82) .
Let K i
= (cid:78) d ∈ (cid:82) i G d for each i ∈
[ | R | ] .
That is,
[Equation]
And so the i, j -th block of G (cid:88) WG (
cid:88) T is given by:
[Equation]
The equality at (47) follows from Lemma 1.
Thus, we have G (cid:88) W = WG (cid:88) , and so for all (cid:88) ,
G
[Equation]
W
[Equation]
) =
WG
[Equation]
vec( (cid:88) ) .
Finally, since σ is applied element-wise, we have
[Equation]
Which proves the second direction.
And so σ ( W vec( (cid:88) )) is an exchangeable relation layer, completing the proof.
G. P ROOF OF T HEOREM 4.
The idea is that if W is not of the form (4) then it has some block W i,j containing two elements whose indices have the same equality pattern, but whose values are different.
Based on these indices, we can explicitly construct a permutation which swaps the corresponding elements of these indices.
This permutation is in (cid:71) (cid:88) but it does not commute with W .
Now we present a detailed proof.
Proof.
Let G (cid:88) ∈ (cid:83) N .
For any relation (cid:82) = { d 1 , ..., d | (cid:82) | } ∈ R , let (cid:78) (cid:82) =
[ N d 1 ] × ...
×
[ N d | (cid:82) | ] be the set of indices into (cid:88) (cid:82) .
If W is not of the form described in Section 4 then there exist i, j ∈ [ | R | ] , with n i , n i (cid:48) ∈ (cid:78) (cid:82)
i and n j , n j (cid:48) ∈ (cid:78) (cid:82) j such that
[Equation]
but
[Equation]
That is, the pairs n i , n j and n i (cid:48) , n j (cid:48) have the same equality pattern over their elements, but the entries of W i,j which correspond to these pairs have differing values, and thus violate the deﬁnition of W in
Section 4.
To show that the layer σ ( W vec( (cid:88) )) is not an EERL, we will demonstrate a permutation
G (cid:88) ∈ (cid:71) (cid:88) for which G (cid:88) W (cid:54) =
WG (cid:88) , and thus G (cid:88) σ ( W vec( (cid:88) ))
(cid:54) =
σ ( WG (cid:88) vec( (cid:88) )) for some (cid:88) .
Let G (cid:88) =
(cid:76) (cid:82) ∈ R (cid:78) d ∈ (cid:82) G d , with the G d deﬁned as follows.
For d ∈ (cid:82) i ∩ (cid:82) j and all k :
G d ( n ) =
  n i (cid:48)
d ( k ) n = n id ( k ) n id ( k ) n = n i (cid:48)
d ( k ) n j (cid:48)
d ( k ) n = n jd ( k ) n jd ( k ) n = n j (cid:48)
d ( k ) n otherwise .
For d ∈ (cid:82) i \ (cid:82) j and all k :
G d ( n ) = 
 n i (cid:48)
d ( k ) n = n id ( k ) n id ( k ) n = n i (cid:48)
d ( k ) n otherwise .
For d ∈ (cid:82) j \ (cid:82) i and all k :
G d ( n ) = 
 n j (cid:48)
d ( k ) n = n jd ( k ) n jd ( k ) n = n j (cid:48)
d ( k ) n otherwise .
And for d (cid:54)∈ (cid:82) i ∪ (cid:82) j :
G d ( n ) =
n.
That is, each G d swaps the elements of n i with the corresponding elements of n i (cid:48) , and the elements of n j with those of n j (cid:48) , so long as the relevant indices are present.
For the case where d ∈ (cid:82) i ∩ (cid:82) j , we need to make sure that this is a valid permutation.
Speciﬁcally, we need to make sure that it is injective (it is clearly surjective from [ N d ] to [ N d ] ).
But it is indeed injective, since we have n id ( k )
=
n jd ( k ) iff n i (cid:48)
d ( k ) =
n j (cid:48)
d ( k ) for all d ∈ (cid:82) i ∩ (cid:82) j and all k , since (cid:80) ( n i,jd ) =
(cid:80) ( n i (cid:48) ,j (cid:48)
d ) .
Now, for all i , let K i
=
(cid:78) d ∈ (cid:82) i G d be the i th diagonal block of G (cid:88) .
By deﬁnition of the G d , for all d ∈ (cid:82) i we have G dn id ,n i (cid:48)
d = 1 , and thus by the observation at (34) we have K i n i , n i (cid:48)
= 1 .
And so
[Equation]
Similarly, K j n j , n j (cid:48) = 1 , so
[Equation]
Finally, by (48), G (cid:88) W (cid:54) =
WG (cid:88) , completing the proof.
H D ETAILS OF E XPERIMENTS
The synthetic data we constructed used 200 instances for each of the three entities.
Unless otherwise noted, we considered a 0 .
1 fraction as ‘observed’.
We ensure that each row and column has at least 5 observations.
For training, we pass our observed data through the network, producing encodings for each entity.
We attempt to reconstruct the original input from these encodings.
At test time, we use the training data to produce encodings as before, but now use these encodings to attempt to reconstruct test observations as well, reporting loss only on these test observations.
For all experiments, we used the following architecture:
The encoder consists of 7 EERLs, each with 64 hidden units, each followed by a batch normalization (Ioffe & Szegedy, 2015) layer and a channel-wise dropout layer.
We then apply a mean pooling layer to produce encodings.
We found that batch normalization dramatically sped up the training procedure.
The number of units in the pooling layer, and hence the size of the encodings, varied by experiment (see below for details).
The decoder architecture is identical to that of the encoder, except that there is no pooling layer before the output.
We used leaky ReLU activations (with parameter 0.1) in all layers except the ﬁnal ones in both the encoder and decoder, where no activation is applied.
In all experiments we used the Adam (Kingma & Ba, 2014) optimizer with initial learning rate 0.0001 to optimize a loss between the observed input values and our predicted output values in the relevant table(s).
FOr the synthetic experiments the loss was RMSE, while for the soccer experiment we used cross-entropy.
We did not perform any tuning of hyperparameters.
H. E MBEDDING
We ran this experiment for 5000 epochs, and used an encoding size of 2 to facilitate plotting.
We trained the model on 90% of the data, using 80% for training and 10% for validation.
The remaining 10% was used to evaluate the quality of the model learned.
H. M ISSING R ECORD P REDICTION
We held out a special 10% portion of the data for testing, then trained models using various amounts of the remaining data.
For a 90%, 80%, ..., 10% fraction of the remaining data, we trained a model using this fraction of the data as observed (using a 90%/10% training/validation split).
We then used each of these trained models to predict the held-out test set.
When predicting the test set, we again varied the fraction of observed data we gave to the model to make its predictions, from 90% to 10%.
For the inductive setting, the trained models were given data from the same distribution as they were trained on, while in the inductive setting the models were given data generated from a new set of entity embeddings.
For each sparsity level, we trained the model for 4000 epochs, using an encoding size of 10.
We repeated this experiment 15 times and averaged the results.
H. P REDICTIVE V ALUE OF S IDE I NFORMATION
We ﬁx the sparsity level of the STUDENT - COURSE table at 10% and calculate loss only on this table.
We vary the sparsity level of the other tables over the range [ . 025 , .
We use an 80%/10%/10% training/validation/test split and trained for 8000 epochs, producing an embedding of size 10.
We repeat this experiment 17 times and average the results.
H. P REDICTIONS ON THE S OCCER D ATABASE
The dataset is available at https://www.kaggle.com/hugomathien/soccer .
We constructed two tables for this experiment.
The ﬁrst table represents the relation between TEAMS and
MATCHES .
For input features we used all the player position features, as well as the match’s country, season, stage and date.
We did not include any of the bookie predictions or the number of goals, as this would be cheating.
For each team, we include all the team-related features found in the
Team_Attributes table.
When a team appears multiple times in the Team_Attributes table we aggregate the entries by averaging numerical features, and taking the most common value for categorical ones.
We also include an additional binary feature indicating which team is home and which is away.
The second table captures the relation between PLAYERS and MATCHES .
For each player participating in a match, we include all the features in the Player_Attributes table, aggregating entries for each player as described above.
Again, we also include a binary feature indicating whether a player played for the home or away team.
We use a model built from ﬁve EERLs.
All
but the last one have 52 units (this is based on the limitation of our GPU memory), use ReLU activations, and are followed by a batch normalization layer (Ioffe & Szegedy, 2015) and channel-dropout.
Chennel-dropout zeros whole channels at random for all entries of all tables, at a rate of 50%.
The ﬁnal EERL uses no activation and 8 output units.
This is followed by a layer which sum-pools both tables into a single feature vector for each match.
Finally, a linear layer produces a length-3 vector of predictions for each match.
We train by minimizing cross-entropy loss between this output and the true match outcomes for 4000 steps.
We report test results on a 90% / 10% training/test split.