R ELATIONAL L EARNING WITH V ARIATIONAL B AYES
Anonymous authors
Paper under double-blind review
A BSTRACT
In psychology, relational learning refers to the ability to recognize and respond to relationship among objects irrespective of the nature of those objects.
Relational learning has long been recognized as a hallmark of human cognition and a key question in artiﬁcial intelligence research.
In this work, we propose an unsupervised learning method for addressing the relational learning problem where we learn the underlying relationship between a pair of data irrespective of the nature of those data.
The central idea of the proposed method is to encapsulate the relational learning problem with a probabilistic graphical model in which we perform inference to learn about data relationship and other relational processing tasks.
I NTRODUCTION
American Psychological Association deﬁnes relational learning as (VandenBos & APA, 2007):
Deﬁnition 1.1 ( Relational learning ) .
Learning to differentiate among stimuli on the basis of relational properties rather than absolute properties.
In other words, relational learning refers to the ability to recognize and respond to relationship (called relational property ) among objects irrespective of the nature of those objects (called absolute property ).
For example (attributed to Doumas & Hummel (2013)), how do we come to understand that two circles are the same-shape in the same way that two squares are?
(in this example, “same-shape” is the relational property and object shape is the absolute property).
Relational learning has long been recognized as a hallmark of human cognition with strong implications for both human-like learning capabilities and generalization capacity (Biederman, 1987;
Medin et al., 1993; Gentner, 2003;
Penn et al., 2008; Holyoak, 2012; Gentner & Smith, 2012).
We refer the interested readers to the provided references for a comprehensive discusson on this subject.
Contemporaneously, the research on learning data relationships—also commonly called “relational learning”—has ﬂourished in the machine learning community where the overarching goal is learning in a context where there may be relationships between learning examples, or where these examples may have a complex internal structure (i.e., consist of multiple components and there may be relationships between these components) (Kersting & Raedt, 2000; Džeroski & Lavra ˇ c, 2001; Heckerman et al., 2001; Jensen &
Neville, 2002; Richardson & Domingos, 2006; Getoor & Taskar, 2007; Nickel et al., 2016).
We argue that the key differnce between the two “relational learning” deﬁnitions and their learning objectives is that Deﬁnition 1.1 takes the relationship learning problem one step further by requiring the data relationships be learned only on the basis of relational properties rather than absolute properties.
To the best of our knowledge, this important distinction—learning relationships irrespective of the absolute properties—has not been rigorously studied in the unsupervised learning community.
More speciﬁcally, most existing unsupervised methods either encourage or do not constrain the relationships learning through absolute properties and, as we’ll show later in this paper, these methods cannot effectively solve relational learning tasks as deﬁned by Deﬁnition 1.1.
In this work, we propose an unsupervised learning method—variational relational learning (VRL)— for addressing the relational learning problem as deﬁned by Deﬁnition 1.1.
At its core, VRL encapsulates the relational learning problem with a probabilistic graphical model (PGM) in which we perform inference to learn about relational property and other relational processing tasks.
Our contribution in this paper is threefold: First, we propose a probabilistic formulation for the relational learning problem deﬁned by Deﬁnition 1.1.
Second, we encapsulate the relational learning problem with a PGM in which we perform inference to learn about relational property and other relational processing tasks.
Third, we propose an efﬁcient and effective learning algorithm that can be trained end-to-end and completely unsupervised, i.e., the learning does not require a labeled training dataset nor training examples that have the same (or different) relational property.
P ROBLEM DEFINITION
We focus on a canonical form of the relational learning problem where we observed a paired dataset
X = { ( a ( i ) , b ( i ) )
| i ∈
[1 ..N ] } consisting of N i.i.d.
samples generated from a joint distribution p ( a ∈ A , b ∈ B ) .
We dissect the information in X into absolute property and relational property where absolute property represents speciﬁc features that describe individual a and b , and relational property represents the relationship between a and b irrespective of their absolute property.
In this work, we interpret the absolute property of a and b as any information that characterizes (even if only partially) the marginal distribution p ( a ) and p ( b ) .
We propose to represent the relational property as a latent random variable (r.v.)
z that satisﬁes the following constraints:
[Equation]
where in Eq. 1(i) and 1(ii)
we interpret the speciﬁcation of relational property in Deﬁnition 1.1— learning relationships irrespective of the absolute properties—as meaning statistical independence , while in Eq.
1(iii) and 1(iv)
we ensure r.v.
z contains relevant (relationship) information that further informs a and b about one another, i.e., H ( a | b , z ) <
H ( a | b ) , H ( b | a , z ) <
H ( b | a ) where H ( · | · ) is the conditional entropy.
It is easy to see that the following conditions are necessary for r.v.
z to exist: (1) H ( b | a ) > 0 and H ( a | b ) > 0 , i.e., a and b cannot be fully determined by each other; (2) r.v. a , b , z are not mutually independent, i.e., p ( a , b , z ) (cid:54) =
p ( a ) p ( b ) p ( z ) (note, however, that they may be pairwise independent).
Our goal for relational learning is to learn about relational property z that satisﬁes Eq.
1 in a completely unsupervised fashion.
In addition, we are interested in two related relational processing tasks: relational discrimination and relational mapping deﬁned as (VandenBos & APA, 2007):
Deﬁnition 2.1 ( Relational discrimination in condition) .
A discrimination based on the relationship between or among stimuli rather than on absolute features of the stimuli.
Deﬁnition 2.2 ( Relational mapping ) .
The ability to apply what one knows about one set of elements to a different set of elements.
Relational discrimination allows us to differentiate ( a ( i ) , b ( i ) ) from ( a ( j ) , b ( j ) ) based on their relational properties.
And relational mapping allows us to apply the relational property of ( a ( i ) , b ( i ) ) to a different set of data, for example, deduce that b ( j ) is related to a ( j ) in the same way that b ( i ) is related to a ( i ) .
M ETHOD
Learning and inferencing relational property z that satisﬁes all four constraints in Eq.
1 is a challenging problem due to the hard independence constraints in Eq. 1(i) and 1(ii).
To overcome this challenge, we ﬁrst introduce VRL as a tractable learning method that satisﬁes 3 (out of 4) constraints in Eq.
Eq. 1(i), 1(iii), 1(iv).
We then discuss VRL’s unique optimization challenges, which are partly attributable to the absence of independence requirement in Eq. 1(ii).
3. V ARIATIONAL RELATIONAL LEARNING
The proposed VRL method consists of two parts: ﬁrst, we encapsulate the relational learning problem with a PGM, called VRL-PGM; we then formulate various relational processing tasks as performing inference and learning in VRL-PGM.
The VRL-PGM model, shown in Fig.
1, samples data a , z , and b from parametric families of distributions— p θ ( a ) , p θ ( z ) , p θ ( b | a , z ) —that are differentiable almost everywhere with respect to (w.r.t.)
a , z , and θ .
In practice, we observe only a set of independent realizations { ( a ( i ) , b ( i ) )
| i ∈
[1 ..N ] } while the true parameter θ ∗ and the corresponding latent variables z ( i ) are unobserved.
A well-known property of the PGM shown in Fig.
1 is that r.v. a and z are independent with no variables observed, but not conditionally independent when b
[Figure]
is observed, i.e., p θ ( a , z ) =
p θ ( a ) p θ ( z ) , p θ ( a , z | b ) (cid:54) =
p θ ( a | b ) p θ ( z | b ) (Bishop, 2006).
Consequently, VRL-PGM can be viewed as a parametric relational learning model that satisﬁes 3 (out of 4) constraints in Eq.
1—Eq. 1(i), 1(iii), 1(iv)
(note that Eq. 1(iv) is trivially satisﬁed in VRL-PGM).
Further discussions on the connection between VRL-PGM and the relational learning problem is provided in Appendix B.1.
Having established VRL-PGM, our primary learning objective is to approximate the unknown true likelihood function p θ ( b | a , z ) and posterior p θ ( z | a , b ) .
Learning p θ ( z | a , b ) provides us a way to infer ( a ( i ) , b ( i ) ) ’s
relational property z ( i ) ; moreover, it serves as a basis for performing relational discrimination where we compare relational properties between different pairs of data.
Learning p θ ( b | a , z ) allows us to perform relational mapping where we use the relational property of ( a ( i ) , b ( i ) ) to map a ( j ) to b ( j ) , i.e., b ( j ) ∼ p θ ( b | a ( j ) ,
z ( i ) ) where z ( i ) ∼ p θ ( z | a ( i ) , b ( i ) ) .
We estimate the parameter for p θ ( b | a , z ) by following the maximum-likelihood (ML) principle, and approximate the true posterior p θ ( z | a , b ) with variational Bayesian approach.
More speciﬁcally, we use a variational distribution q φ ( z | a , b ) , parameterized by φ , to approximate the unknown (and often intractable) true posterior.
Both θ and φ are learned through maximizing the following variational lower bound (derivation is provided in Appendix C):
[Equation]
Recall that learning z independent of a is central to our relational learning goal.
While this independence assumption is built into VRL-PGM, the learning objective L ( θ, φ ; a ( i ) , b ( i ) ) (abbreviated as L ( i ) ) does not explicitly force z to be independent of a nor penalize learning a dependent z .
In practice, there may be numerous reasons that could break this independence assumption, e.g., insufﬁcient training data, failure to reach the global optimum, non-identiﬁability of the model, etc., and it may be desirable to explicitly enforce independence between z and a .
One way to achieve this is to introduce a non-positive function that measures the dependency between a and z with maximum attained when they are independent.
For example, we can append the negative mutual information between z and a , − I ( z ; a ) = − D KL ( p θ ( z , a ) (cid:107) p θ ( z ) p θ ( a )) , to L ( i ) :
[Equation]
Since I ( z ; a ) ≥
and I ( z ; a ) =
if and only if z and a are independent, the addition of − I ( z ; a ) to L ( i ) not only maintain the validity of the lower bound, but also retain its quality ( z and a are independent in VRL-PGM).
3. O PTIMIZATION CHALLENGES
A limitation of the proposed VRL method is its inability to enforce the independence constraint between z and b in Eq.
1(ii).
This may lead to a worst-case scenario of learning a degenerated posterior q φ ( z | a , b ) =
q φ ( z | b ) where relational property z only depends on the absolute property of b .
Another limitation of VRL is exposed when the assumption H ( b | a ) > 0 is violated and b can be completely determined by a .
In this case, a degenerated likelihood p θ ( b | a , z ) =
p θ ( b | a ) may be ﬁtted and relational property z plays no role in the learning process.
To further explain these limitations and potential mitigation strategies for avoiding the worst-case learning scenario, we ﬁrst dissect VRL’s gradient updating process into the following steps (using a single data point ( a ( i ) , b ( i ) ) as an example): (1) sample z ( i ) ∼ q φ k
( z | a ( i ) , b ( i ) ) by using the current parameter φ k ; (2) evaluate
L ( i ) and calculate gradients g = ∇ θ k ,φ k L ( i ) by using φ k , θ k ; (3) use gradients g to update φ k , θ k and get new parameters φ k +1 , θ k +1 .
This process is depicted with an information ﬂow diagram shown in Fig.
2a where ideally we would like every path to contribute to the evaluation of all the terms in its reachable nodes in order to obtain meaningful gradients for updating its associated parameters.
However, there are two situations where this is not the case: The ﬁrst situation, called informationp θ ( b ( i ) | a ( i ) , z ( i ) ) , p θ ( z ( i ) )
z ( i ) ∼ q φ ( z | a ( i ) , b ( i ) )
b ( i ) φ a ( i ) φ θ a ( i ) θ (a) p θ ( b ( i ) | a ( i ) , z ( i ) ) , p θ ( z ( i ) )
z ( i ) ∼ q φ ( z | a (cid:48) ( i ) , b (cid:48) ( i ) )
b (cid:48)
( i ) φ a (cid:48)
( i ) φ θ a ( i ) θ (b) process improved shortcut , occurs when the learning of p θ ( b ( i ) | a ( i ) , z ( i ) ) rely entirely on the dashed path in Fig.
2a; more speciﬁcally, the dashed (shortcut) path directly propagates b ( i ) through z ( i ) to p θ ( b ( i ) | a ( i ) , z ( i ) ) and, as a result, the relational property z ( i ) may only learn to encode the absolute property of b ( i ) , i.e., q φ ( z | a ( i ) , b ( i ) ) =
q φ ( z | b ( i ) ) .
The second situation, called deterministic-mapping , occurs when b ( i ) can be fully characterized by a ( i ) ; in this case, the learning of p θ ( b ( i ) | a ( i ) , z ( i ) ) may only rely on the dash-dotted path in Fig.
2a, i.e., p θ ( b ( i ) | a ( i ) , z ( i ) )
=
p θ ( b ( i ) | a ( i ) ) .
While both situations can be viewed as overﬁtting problem, deterministic-mapping is mainly caused by the data itself and is beyond our control.
On the other hand, information-shortcut is caused by short-cutting the gradient update process, which we may overcome with additional regularization techniques.
Here we propose two approaches for mitigating the information-shortcut problem by disrupting the ﬂow of information passing through the shortcut path.
In the ﬁrst approach, we restrict the ﬂow of information by constraining the expressiveness of the latent variable z , e.g., by adopting a discrete categorical r.v.
(assuming we know a priori the underlying relational property is discrete).
In the second approach, we propose a novel data augmentation strategy—relation-preserving data augmentation (RPDA)—that aims to eliminate the shortcut path entirely.
First, we deﬁne a set of relation preserving functions
D = { d ( a , b ; r ) | r ∈ R
(some index set) , d : A×B→A×B } that preserve data relationship in the following sense: p
[Equation]
Assuming we have access to
D , the proposed RPDA strategy then seek to optimize a modiﬁed lower bound L ( i ) RPDA :
[Equation]
where ( a (cid:48) ( i ) , b (cid:48)
[Equation]
= [Equation] b
[Equation]
i
[Equation]
r
[Equation]
i
[Equation]
r ( i
[Equation]
R )
[Equation]
Note that due to the relation preserving property of D , we have q φ ( z ( i ) | a (cid:48)
( i ) , b (cid:48)
( i ) )
=
q φ ( z ( i ) | a ( i ) , b ( i ) ) , and thus L ( i ) RPDA is equivalent to L ( i ) in Eq.
When we optimize with L ( i ) RPDA , the gradient update process can be redrawn in Fig.
2b, where now the learning of p θ ( b ( i ) | a ( i ) , z ( i ) ) can no longer rely solely on the shortcut path to propagate b (cid:48)
( i ) since it differs from b ( i ) by a non-deterministic factor r ( i ) .
In practice, it may seem unrealistic to assume that we can construct a set of RPDA functions D without extensive knowledge of the underlying relational property.
However, we can treat data augmentation as a form of regularization and construct a D that reﬂects our prior knowledge and belief of the underlying system (Ronneberger et al., 2015; Perez & Wang, 2017).
For example, if we want the learning to be rotation invariance—a common theme in computer vision applications—we can construct a D that consists of image rotation augmentations, e.g., d ( a , b ; r ) =
(rot( a , r ) , rot( b , r )) where rot( x , r ) rotates the image x by r ∈ R = [0 , 360) degrees (note that both a and b are rotated by the same amount).
The full training procedure for VRL with
RPDA is summarized in Appendix A. Additional remarks on the practical applicability of RPDA is provided in Appendix B.2.
R ELATED WORK
Machine learning approaches for relational processing have gained increasing interest and attention in the literature.
Existing work on relational learning are mostly based on supervised or semi-supervised approaches (Chen et al., 2019; Lu et al., 2012; Puebla et al., 2019).
Other recent work focus on high-level cognitive tasks, such as visual Q&A and state prediction for complex-physics systems, and derive their relational processing capabilities from learning with clever designed neural networks (Hill et al., 2019; Santoro et al., 2017; Raposo et al., 2017; Battaglia et al., 2018; 2016; Wu et al., 2015;
Reed et al., 2015; van Steenkiste et al., 2018;
Chang et al., 2016; Fragkiadaki et al., 2015; Kipf et al., 2018).
Our work differ from these methods in two ways: (1) we focus on learning an independent relational property in a completely unsupervised fashion; (2) we derive our relational learning capability from a PGM formulation, which gives us the ﬂexibility to use any compatible inference method or function approximation.
Many existing unsupervised learning methods can also be applied to our problem setting (Kingma &
Welling, 2014; Goodfellow et al., 2014; Mikolov et al., 2013a;b; Song et al., 2007;
Dilokthanakul et al., 2017; Makhzani et al., 2016); however, most of these methods learn a single representation with superimposing information about the relational and absolute property.
The difﬁculty of decoupling the relational property from the learned representation constitute a major obstacle to relational learning.
A closely related method is proposed by Guu et al.
(Guu et al., 2018), but their learned latent “edit vector” (loosely related to our relational property) may be coupled with absolute property.
Other related work include methods on learning a disentangled representations with applications in style-transfer, image-to-image translation, domain adaptation, etc.
(Huang et al., 2018;
Chen et al., 2016;
Tenenbaum & Freeman, 1997; Higgins et al., 2017; Bousmalis et al., 2016; Mathieu et al., 2016; Tulyakov et al., 2017; Denton & Birodkar, 2017; Villegas et al., 2017; Donahue et al., 2017;
Shen et al., 2017).
Most of these methods strive to learn a disentangled representations of content and style (or pose for video sequence data) where content is generically deﬁned as the underling spatial structure, and style as the rendering of the structure.
In comparison, our work can be viewed as learning a disentangled representations of relational and absolute property .
We argue that stylecontent separation is fundamentally different from relational-absolute separation; more speciﬁcally, we consider both style and content information as absolute property (both describe features of an individual data), while relational property provides additional information about data relationships.
E XPERIMENT
In this section, we present experimental results from applying VRL to a variety of relational learning tasks.
In our implementation, we parameterize p θ ( b | a ( i ) , z ( i ) ) and q φ ( z | a (cid:48) ( i ) , b (cid:48)
( i ) ) with fullyconnected networks (MLPs) f pθ ( a ( i ) ,
z ( i ) ) and f qφ ( a (cid:48) ( i ) , b (cid:48)
( i ) ) , respectively.
For binary valued data, we let p θ ( b | a ( i ) , z ( i ) ) be a multivariate Bernoulli distribution whose probability parameters are computed from f p θ ( a ( i ) ,
z ( i ) ) .
For real-valued data, we let p θ ( b | a ( i ) , z ( i ) ) be a multivariate Gaussian distribution with a ﬁxed diagonal covariance and the mean is computed from f pθ ( a ( i ) , z ( i ) ) .
We experimented with both continuous and discrete r.v. z .
For continuous z , we let the prior p θ ( z ) be a bivariate normal distribution z ∼ N ( 0 , I ) and q φ ( z | a (cid:48) ( i ) , b (cid:48)
( i ) ) be a bivariate Gaussian distribution with diagonal covariance whose mean and covariance are computed from f qφ ( a (cid:48)
( i ) , b (cid:48)
( i ) ) .
For discrete z , we adopted two categorical r.v., z
= [ z 1 , z 2 ] , each having a uniform prior over ﬁve categories and let q φ ( z | a (cid:48)
( i ) , b (cid:48)
( i ) ) represents two categorical r.v. reparameterized with Gumbel-Softmax distributions whose class probabilities are computed from f qφ ( a (cid:48)
( i ) , b (cid:48)
( i ) )
(Jang et al., 2017; Maddison et al., 2017).
For RPDA, we used random image rotation as data augmentation functions.
Parameters θ and φ were jointly trained to maximize (cid:101) L ( i ) RPDA in Eq.
(4) using Adam optimizer (Kingma & Ba, 2014).
Full implementation details are provided in Appendix D.1 and source code for reproducible results is available at: [Public link provided after review].
5. R ELATIONAL LEARNING WITH DECOUPLED RELATIONSHIPS
We ﬁrst present two relational learning tasks designed with the following datasets:
MNIST dataset consists of 28 × 28 grayscale images of standard handwritten digits with labels; the dataset is composed of 60,000 training samples and 10,000 testing samples (LeCun & Cortes, 2010).
Omniglot
dataset contains 1623 different handwritten characters ( 105 × 105 grayscale images) from 50 different alphabets; the dataset is split into a background
training) set of 30 alphabets and an evaluation ( testing set
of
20 alphabets
(Lake et al., 2015).
A paired MNIST (and Omniglot) dataset X M (and X O )
= { ( a ( i ) , b ( i ) ) | i ∈
[1 ..N ] } is constructed where each a ( i ) is a resized 32 × 32 and randomly rotated MNIST (and Omniglot) binary image and b ( i ) is another random counter-clockwise rotation of a ( i ) by 0 ◦ , 72 ◦ , 144 ◦ , 216 ◦ , or 288 ◦ .
There are ﬁve uniquely deﬁned relative rotational relationships between ( a ( i ) , b ( i ) ) in X M (and
X O ); furthermore, the relative relationships are decoupled from their absolute properties, i.e., the relationship cannot be inferred from a ( i ) or b ( i ) alone.
The goal of relational learning is to discover the underlying relative rotational relationship.
We ﬁrst demonstrate VRL’s relational discrimination and relational mapping capabilities on the Omniglot relational learning task:
Relational discrimination.
We used the trained posterior q φ ( z | a , b ) to infer the relational property of a hold-out dataset constructed from the evaluation alphabets.
Figure 3a shows a scatter plot of the relational property inferred by VRL where we can see that the approximated posterior accurately cluster (discriminate) data with the same (different) relative rotational relationship together (apart).
Relational mapping.
We used the trained likelihood p θ ( b | a , z ) to generate images from a given a and z .
We chose a from a hold-out dataset and z from: (1) direct sampling in the latent space; (2) relational property inferred from a source data point ( a s , b s ) .
Figure 3b shows predicted images with z sampled from the latent space shown in Fig. 3a.
Figure 3c shows examples of relational mappings from a ( c ) to b ( r,c ) by applying the relational property inferred from a source data point ( a s , b ( r ) s ) .
Next, we compare VRL with other unsupervised learning methods: Variational autoencoder (VAE) (Kingma & Welling, 2014); VAE with Gaussian mixture prior (GMVAE)
(Dilokthanakul et al., 2017); Adversarial autoencoder (AAE) (Makhzani et al., 2016); Spatial transformer networks (STN) (Jaderberg et al., 2015); Learning independent causal mechanisms (LICM)
(Parascandolo et al., 2018); Neural relational inference (NRI) with graph neural networks (Kipf et al., 2018); Learning linear structure in representation space through vector arithmetic (Mikolov et al., 2013a;
Radford et al., 2016).
In addition, we also compare VRL with self-supervised representation learning method
BYOL (Grill et al., 2020).
Each method is assessed on both MNIST and Omniglot relational learning tasks through unsupervised clustering.
VAE, GMVAE, and AAE all trained directly using ( a ( i ) , b ( i ) ) and we perform clustering on their learned latent space.
For vector-arithmetic, we ﬁrst train an autoencoder on individual a ( i ) , b ( i ) and then used the trained encoder (Enc) to compute a latent vector representation of ( a ( i ) , b ( i ) ) through Enc ( b ( i ) )
− Enc ( a ( i ) ) , on which the clustering is applied during evaluation.
We select dim( z )=10 for VAE, GMVAE, AAE, and vector-arithmetic via cross-validation.
For LICM, we trained a committee of 5 experts to learn a set of independent mechanisms that generates b ( i ) from the canonical a ( i ) , and used the trained discriminator during evaluation to select the winning expert (as a categorical variable).
We applied NRI to infer the interaction between a ( i ) and b ( i ) and used the NRI-Encoder to predict the relationship type (as a categorical variable).
We trained
STN, denoted as STN-Afﬁne, to minimize the difference between b ( i ) and an afﬁne transformation of a ( i ) by using a spatial transformer with a localization network that takes both a ( i ) , b ( i ) as input; during evaluation, we perform clustering on the output of the trained localization network.
We further experimented with a variation of STN-Afﬁne that only allows rotation transformation, denoted as [Figure]
where z ( r ) ∼ q φ ( z | a s , b ( r ) s )
STN-Rotate, in which case the localization network only need to learn the rotation angle.
We applied BYOL to learn a joint image represenation for ( a ( i ) , b ( i ) ) , on which the clustering is applied during evaluation.
Here, BYOL used random image rotation augmentations to generate “positive paris” for training. For fair comparison, we adopted image rotation augmentation during the training of all the methods.
In our comparison, STN-Afﬁne and STN-Rotate are considered as the benchmark methods as they are architecturally designed to learn about the spatial transformation between a ( i ) and
b ( i ) .
In contrast, VRL does not make any assumption about the underlying relationship.
To quantitatively evaluate the clustering performance, we calculate the classiﬁcation error rate of a hold-out dataset based on a simple classiﬁer that was trained on 5% of the data with label.
For continuous latent space, we trained a standard multi-class support vector machine classiﬁer with radial basis function kernel (Manning et al., 2008).
For discrete (categorical) latent space, we follow the evaluation protocol on clustering assignment described in Makhzani et al.
Results are summarized in Table 1 where each entry reports the average and standard deviation over 5 runs with different random seeds and each run consists of either 3 (randomly selected out of 5) or 5 relative rotational relationships.
We can see that the proposed VRL with RPDA achieved high accuracy in recovering the relative relationships for all the tasks and outperformed other baseline methods including the benchmark STN-Afﬁne and STN-Rotate.
To gain insight into the information-shortcut problem and the proposed mitigation strategies introduced in Sec. 3.2, we performed an ablation study on the prior selection of z and RPDA.
The results are summarized in Table 1 and we make the following observations:
First, representing z as a discrete categorical (Cat.)
r.v. limits its expressiveness and improves VRL’s performance over a continuous prior (Cont.)
; Second, RPDA is a critical component necessary for VRL to learn a meaningful and independent relational property, especially when ﬂexible function approximations such as deep neural networks are used.
Additional experimental results and analysis are provided in Appendix E.
5. R ELATIONAL LEARNING WITH COUPLED RELATIONSHIPS
To further test the robustness and generalizability of VRL, we consider a scenario where the relative relationship is coupled with the absolute property.
To setup this experiment, we modiﬁed the construction of X M , denoted as X M d , so that the relative rotational relationship between a ( i ) and b ( i ) is completely determined by its digit representation (absolute property of a ( i ) and b ( i ) ): a ( i ) ∈ → 0 ◦ , a ( i ) ∈ [ ’2’, ’3’ ]
→ 72 ◦ , a ( i ) ∈ [ ’4’, ’5’ ] → 144 ◦ , a ( i ) ∈ [ ’6’, ’7’ ] → 216 ◦ , a ( i ) ∈
[Table]
[Figure] [Table]
[ ’8’, ’9’ ] → 288 ◦ (read: if a ( i ) is digit ’0’ or ’1’ then the relative rotational relationship between ( a ( i ) , b ( i ) ) is 0 ◦ ).
The question then arise: is VRL capable of learning an independent relational property even when the underlying relative relationship is coupled with the absolute property?
To test this idea, we trained VRL on X M d but validated it on a hold-out dataset of X M which has a decoupled relative relationship.
The results are shown in Fig.
4, where we can see that VRL was indeed capable of learning an independent relational property irrespective of the digit representation.
If this were not the case, we would expect to see a scatter plot Fig.
4a with heavily overlapped relative relationship labels since X M was constructed with random relative rotational relationships.
Figures 4b and 4c further demonstrate VRL generalizes well to unseen data—VRL trained on X M d learned to rotate any digit by any amount despite not having seen most of the digit-rotation pairs during training.
[Figure]
[Equation]
[Equation] [Equation] [Equation] [Equation]
5. R ELATIONAL LEARNING WITH HIGH - LEVEL PERCEPTION REASONING
Finally, we present two relational learning tasks that require high-level perception reasoning based on the following datasets:
Yale Face Database contains 165 grayscale face images of 15 human subjects under 8 facial expressions and 3 illumination conditions (Belhumeur et al., 1997).
Extended Yale Face Database B contains 16128 grayscale face images of 28 human subjects under 9 poses and 64 illumination conditions (Georghiades et al., 2001).
We ﬁrst extracted three facial expressions (happy, surprised, sad) of each subject from Yale Face
Database to form a paired dataset X F e where each data point ( a ( i ) , b ( i ) ) represents a subject with different facial expressions.
There are three undirected 1 relative expression changes between ( a ( i ) , b ( i ) ) in X F e : “happy-sad”, “happy-surprised”, “surprised-sad”.
Next, we extracted four illumination conditions (left, front, right, top) of each subject from Extended Yale Face Database B to form a paired dataset X F l where each data point ( a ( i ) , b ( i ) ) represents a subject with different illumination conditions.
There are six undirected relative illumination changes between ( a ( i ) , b ( i ) ) in X F l : “left-right”(“L-R”), “front-top”(“F-T”), “left-front”(“L-F”), “left-top”(“L-T”), “front-right”(“F-R”), “right-top”(“R-T”).
For each image, we center-cropped, resized to 64 × 64 , and normalized pixel values to be within [0 , 5] .
Due to the limited data samples in X F e (45 unique images) and X F l ( unique images), both training and validation are performed on the entire dataset.
The inference results from a trained posterior q φ ( z | a , b ) are shown in Fig.
5b and 6b where we can see that VRL is effective in learning an independent relative expression and illumination changes.
Comparing our results with existing unsupervised methods on face images (Song et al., 2007; Wu et al., 2013; Shi et al., 2018; Tapaswi et al., 2019), we remark that existing methods cluster images by its absolute property (e.g., subject identity, individual expression, illumination condition), while VRL clusters images by its relational property (e g., relative expression and illumination changes).
Further analysis of the results are provided in Appendix E.2.
[Figure]
(cid:35)
[Figure]
“F-
(cid:35)
“L-F, + :“L-T, × :“F-R (cid:50) “R-T
D ISCUSSION AND CONCLUSION
The proposed VRL method comes with both advantages and disadvantages: the main advantage of VRL is its relational learning capabilities; however, this may also be one of its disadvantages.
More speciﬁcally, VRL can learn an independent relational property even when it is coupled with the absolute property (see Sec. 5.2), i.e., VRL is oblivious to the coupling information between the two properties.
Nevertheless, such information may be of interest to the user, and in this regard, VRL only provides a partial view of the data.
In conclusion, the proposed VRL method is an efﬁcient and effective unsupervised learning method for addressing the relational learning problem where our goal is to learn an independent relational property.
By dissecting the data information into decoupled relational and absolute property , we hope VRL can bring new insight into everyday data analysis and ultimately ﬁnd applications for a wide variety of problems. A A LGORITHM
procedure VRL( X , p ( (cid:15) ) , D ) (cid:46)
If RPDA not available, D = { id( · ) | ( a , b )
= i
d( a , b ) } Initialize parameters θ, φ while not convergence of parameters ( θ, φ ) do Sample minibatch { ( a ( i ) , b ( i ) ) | i ∈ [1 ..M ] } from X . Run RPDA and obtain ( a (cid:48) ( i ) , b (cid:48) ( i ) )
=
d ( a ( i ) , b ( i ) ; r ( i ) ) , r ( i ) ∼ U ( R ) , i =1 , ..., M .
Compute gradients g = ∇ θ,φ L ( i ) RPDA (see Eq.
Update parameters θ, φ using gradients g (e.g., SGD).
end while return θ, φ end procedure B A DDITIONAL R EMARKS B. R EMARKS ON VRL-PGM The proposed VRL-PGM reﬂects our priority and compromise for using a PGM to represent the relational learning problem: we sacriﬁce some identiﬁability of the original abstract problem but obtain a rigorous and tractable model that achieves our objective of learning an independent relational property.
Here we discuss aspects of the original relational learning problem (see Section 2) that differ from the proposed VRL-PGM (see Section 3.1).
First, the original problem description in
Eq.
1 speciﬁes that the relational property z be independent of both a ’s and b ’s absolute properties; however, learning and inferencing relational property z that satisﬁes both independence constraints is a challenging problem; therefore, as a compromise, VRL-PGM only enforce the independence constraint between z and a .
Second, the original problem is inherently undirected with no cause-effect relationship between a and b , whereas VRL-PGM is based a directed acyclic graph (DAG) that artiﬁcially introduces conditional dependency between a and b .
We note that this can be viewed as the result of VRL-PGM choosing to satisfy Eq. 1(i), 1(iii), 1(iv).
It is equally valid for VRL-PGM to choose to satisfy Eq.
1(ii), 1(iii), 1(iv)
and it would lead to the same model but with a and b swapped.
In other words, the application of VRL does not require the true conditional dependency between ( a , b ) be known in advance only that it is maintained consistently throughout learning and inference.
The above-mentioned discrepancies represent the compromises we made with adopting VRL-PGM in exchange for a rigorous and tractable method for learning an independent relational property.
Finally, these compromises are not been made without consequences: information-shortcut is a direct result of not enforcing independence constraint between z and b , and deterministic-mapping can be viewed as caused by the causal relationship VRL-PGM introduced between a and b .
B. R EMARKS ON RPDA
Here we discuss the practical applicability of the proposed RPDA strategy.
More speciﬁcally, we argue that in many practical problem settings, the RPDA functions D can be designed without any knowledge of the underlying relational property.
For example, as we have explained in Sec. 3.2, in many computer vision applications, rotation invariant is a desirable property for the learned model; for example, in spectral imaging applications, oftentimes the orientation of the images are not preserved or not enforced only that they are consistent between the same paired images (Ronneberger et al., 2015).
In such problem setting, we can safely use image rotation function for constructing
D .
Another example may be: for a discrete time-series data a [ t ] , b
[ t ] that represent the input and output of a linear time-invariant (LTI) system (commonly assumed in signal processing and control theory (Oppenheim & Schafer, 2009)), and we want to learn a relational property that characterize the system’s impulse response.
We have α b [ t − τ ] = α a [ t − τ ] , ∀ α ∈ R , τ ∈ Z , and we can construct D with d ( a [ t ] , b [ t ]; α, τ ) =
( α a [ t − τ ] , α b [ t − τ ]) , α, τ ∈ R
=
R × Z .
In all of the above examples, the construction of RPDA functions D reﬂects our prior knowledge and belief of the underlying system and not based on the underlying relational property.
However, just like any data augmentation, the effectiveness of RPDA will depend on the problem setting and we advocate to start without RPDA and only apply it when suspecting information-shortcut occurs.
C D ERIVATION OF VARIATIONAL LOWER BOUND
To derive a variational lower bound for VRL-PGM, we ﬁrst write the log-evidence as log p θ ( X ) =
log p θ ( { ( a ( i ) , b ( i ) )
| i ∈
[1 ..N ] } )
= (cid:80) Ni =1 log p θ ( a ( i ) , b ( i ) ) , where each term in the summation can be expressed as:
[Equation]
The ﬁrst term on the right-hand side (RHS) is the KL-divergence from p
θ ( z | a ( i ) , b ( i ) ) to q φ ( z | a ( i ) , b ( i ) ) , which provides a measure of dissimilarity between the two distributions; the second term on the RHS continues as:
[Equation]
where in the second line we use the fact that r.v. a and z are independent.
Substitute Eq.
(6) back in (5) and rearrange terms gives us the expression for Eq. 2:
[Equation]
where
[Equation]
The term L ( i ) serves as a lower bound for the conditional log-likelihood log p θ ( b ( i ) | a ( i ) ) since
KL-divergence is non-negative.
Maximizing L ( i ) w.r.t.
φ and θ gives us both a ML estimate for p θ ( b | a , z ) (by maximizing the ﬁrst term inside the expectation in L ( i ) ) and a lower KL-divergence (the better q φ ( z | a ( i ) , b ( i ) ) approximates the true posterior p θ ( z | a ( i ) , b ( i ) ) )
as the conditional log-likelihood log p θ ( b ( i ) | a ( i ) ) does not depend on φ .
The lower bound L ( i ) can be maximized with gradient ascend methods; however, its gradients w.r.t.
φ is difﬁcult to obtain: the expectation in
L ( i ) is taken w.r.t.
the distribution q φ ( z | a ( i ) , b ( i ) ) , which is a function of φ (Paisley et al., 2012).
To obtain efﬁcient estimators for both L ( i ) and its gradients, we adopt the reparameterization trick developed in Kingma & Welling (2014) where the r.v.
z is expressed as a transformation of another r.v.
(cid:15) ∼ p ( (cid:15) )
that is independent of a , b , and φ :
z = g ( (cid:15) , a ( i ) , b
( i ) , φ ) where g is some differentiable and invertible transformation.
Given such a change of variable, the lower bound L ( i ) can be rewritten as:
[Equation]
where z = g ( (cid:15) , a ( i ) , b ( i ) , φ ) and (cid:15) ∼ p ( (cid:15) ) .
Note that the expectation in Eq.
(8) is taken w.r.t.
p ( (cid:15) ) and we can approximate L ( i ) with a Monte Carlo estimator:
[Equation]
where z ( i,l ) =
g ( (cid:15) ( i,l ) , a ( i ) , b ( i ) , φ ) and (cid:15)
( i,l ) ∼ p ( (cid:15) ) .
An estimator for L ( i ) RPDA in Eq.
(4) follows similarly as:
[Equation]
(cid:48) (
[Equation]
) ) , (10) where ( a (cid:48) ( i ) , b (cid:48) ( i ) )
=
d ( a ( i ) , b ( i ) ; r ( i ) ) , r ( i ) ∼ U ( R ) , and z ( i,l ) =
g ( (cid:15) ( i,l ) , a (cid:48) ( i ) , b (cid:48) ( i ) , φ ) , (cid:15) ( i,l ) ∼ p ( (cid:15) ) .
The lower bound for a minibatches of data X M = { ( a ( i ) , b ( i ) )
| i ∈ [1 ..M ] } can be approximated by (cid:101)
L RPDA ( θ, φ ; X M ) =
NM (cid:80) Mi
=1 (cid:101) L ( i ) RPDA .
And ﬁnally, the gradients ∇ θ,φ (cid:101)
L RPDA ( θ, φ ; X M ) =
NM (cid:80) Mi
=1 ∇ θ,φ (cid:101) L ( i ) RPDA can be computed in a straightforward manner and used to update the parameters θ and φ with stochastic optimization methods, such as
SGD.
D I MPLEMENTATION DETAILS D. VRL IMPLEMENTATION
Here we describe the details of our VRL implementation.
We parameterize both p θ ( b | a ( i ) , z
( i,l ) ) and q φ ( z | a (cid:48)
( i ) , b (cid:48)
( i ) ) using fully-connected networks (MLPs) with rectiﬁed linear non-linearities.
For binary valued data such as MNIST and Omniglot, we let p θ ( b | a ( i ) , z
( i,l ) ) be a multivariate
Bernoulli distribution whose probability parameters p ( i,l ) are computed from MLPs f pθ ( a ( i ) , z ( i,l ) ) : f [Equation] [Equation]
[Equation]
The reconstruction error in this case is the binary cross-entropy: log p θ ( b ( i ) | a ( i ) ,
z ( i,l ) )
= − (cid:80) b ( i ) log p ( i,l ) + (1 − b ( i ) )
log(1 − p ( i,l ) ) .
For real-valued data such as Yale dataset, we let p θ ( b | a ( i ) , z
( i,l ) ) be a multivariate Gaussian distribution with a ﬁxed diagonal covariance
N ( b ; µ ( i,l ) , σ 2 I ) where µ ( i,l ) is computed from MLPs f pθ ( a ( i ) , z ( i ) ) :
[Equation]
The reconstruction error in this case is the mean squared error: log p θ ( b ( i ) | a ( i ) , z
( i,l ) )
= −σ 2 (cid:107) b ( i ) − µ ( i,l ) (cid:107) 2 + const.
We experimented with both continuous and discrete (categorical) latent variable z .
For a continuous z , we let the prior p θ ( z ) be a bivariate normal distribution and the approximated posterior q φ ( z | a (cid:48)
( i ) , b (cid:48)
( i ) ) be a bivariate Gaussian distribution with a diagonal covariance
N ( z ; µ ( i ) , ( σ ( i ) ) 2 I ) where µ ( i ) and σ ( i ) are computed from MLPs f qφ ( a (cid:48)
( i ) , b (cid:48)
( i ) ) :
[Equation]
In this case the latent variable samples are drawn as: z
( i,l ) = µ ( i ) + σ ( i ) (cid:12) (cid:15) ( i,l ) , (cid:15) ( i,l ) ∼ N ( 0 , I ) .
For a discrete z , we used two categorical r.v.,
z
=
[ z 1 , z 2 ] , each having a uniform prior over ﬁve categories and let q φ ( z | a (cid:48)
( i ) , b (cid:48)
( i ) ) represents two categorical r.v. reparameterized with GumbelSoftmax distributions whose class probabilities π
= ( π 11 , . . . , π 15 ) and π 2 = ( π 21 , . . .
, π 25 ) are computed from MLPs f qφ ( a (cid:48) ( i ) , b (cid:48) ( i ) )
(Jang et al., 2017; Maddison et al., 2017):
[Equation] [Equation]
In this case the latent variable samples are drawn as: z
( i,l ) j
[Equation]
where (cid:15) ( i,l ) ∈
R 5 are i.i.d.
samples drawn from a Gumbel(0 , 1) distribution and the softmax temperture τ controls the ˛aˇrsmoothness ˛a´s of the samples.
For MNIST and Omniglot relational learning tasks, we used random image rotation for RPDA, i.e., D = { (rot( a , r ) , rot( b , r )) | r ∈ [0 , 360) } .
For Yale relational learning tasks, since we are only interested in learning an undirected relative expression and illumination changes (i.e., p θ ( z | a , b ) = p θ ( z | b , a ) ), this prompted us to construct RPDA functions D with random image rotation and swapping operations: D = { swap (rot( a , r ) , rot( b , r ))
| r ∈ [0 , 360) } where
[Equation]
Finally, the learning objective (cid:101) L ( i ) RPDA in Eq.
10 can be constructed once p θ ( b | a ( i ) , z
( i,l ) ) , p θ ( z ) , and q φ ( z | a (cid:48)
( i ) , b (cid:48)
( i ) ) are deﬁned.
For example, when we adopt a discrete latent variable z = [ z 1 , z 2 ] for
MNIST and Omniglot relational learning tasks (Sec. 5.1 and 5.2), we can derive (
cid:101) L ( i ) RPDA (with L = 1 ) as:
[Equation]
where ( a (cid:48) ( i ) , b (cid:48) ( i ) )
= (rot( a , r ( i ) ) , rot( b , r ( i ) )) , r ( i ) ∼ U ([0 , 360)) , [ π 1 , π 2 ]
= f qφ ( a (cid:48) ( i ) , b (cid:48) ( i ) ) ,
z ( i, 1) j = softmax(( (cid:15)
( i, 1) + log π j ) /τ ) , (cid:15) ( i, 1) ∼ Gumbel(0 , 1) , j = 1 , 2 , and p ( i, 1) =
f pθ ( a ( i ) , [ z ( i, 1), z
( i, 1)]) .
Similarly, (cid:101) L ( i ) RPDA (with L = 1 ) for a continuous latent variable z can be derived as:
[Equation]
16) where ( a (cid:48) ( i ) , b (cid:48) ( i ) )
= [Equation] a , r ( i ) ) , rot( b , r ( i ) )) , r ( i ) ∼ U ([0 , 360)) , [ µ ( i ) , σ ( i ) ]
= f qφ ( a (cid:48) ( i ) , b (cid:48) ( i ) ) ,
z ( i, 1) =
µ ( i ) + σ ( i ) (cid:12) (cid:15) ( i, 1) , (cid:15) ( i, 1) ∼ N ( 0 , I ) , and p ( i, 1) =
f pθ ( a ( i ) , z
( i, 1) ) .
For Yale relational learning tasks (Sec. 5.3), we adopted a continuous latent variable z with σ = 0 .
1 and (cid:101) L ( i ) RPDA (with
L = 1 ) can be derived as:
[Equation]
where ( a (cid:48) ( i ) , b (cid:48) ( i ) )
= swap(rot( a , r ( i ) ) , rot( b , r ( i ) )) ,
[Equation]
All MLPs with parameters θ and φ were jointly trained for 300k iterations (without batchnormalization, weight decay, nor dropout) to maximize (cid:101) L ( i ) RPDA in Eq.
(4) with using Adam optimizer (learning rate=0.0004, β 1 =0.9, β 1 =0.999) (Kingma & Ba, 2014).
Minibatches of size M=100 were used.
We anneal the learning rate (0.0004 base learning rate) with step decay (factor of 0.5 every 100k iterations).
When Gumbel-Softmax distributions is used, we anneal the softmax temperature τ from 1.0 to 0.5 with exponential decay (decay rate=0.00005).
D. B ASELINE METHODS IMPLEMENTATION
In our STN-Afﬁne implementation, we trained a spatial transformer (ST) to minimize the difference between b ( i ) and an afﬁned transformed a ( i ) , ˆa ( i ) (Jaderberg et al., 2015; Dong et al., 2017): a ( i ) → ST → ˆa ( i ) .
The localization network in STN-Afﬁne takes both a ( i ) , b ( i ) as input and outputs an afﬁne transformation matrix:
[Equation]
In our STN-Rotate implementation, we further restrict STN-Afﬁne to only allow rotation transformation and the localization network in this case only outputs the rotation angle ∆ θ :
[Equation]
Our VAE, AAE, GMVAE, LICM implementations are based on (Kingma & Welling, 2014;
Makhzani et al., 2016; Dilokthanakul et al., 2017; Ge, 2019; Dilokthanakul, 2017; Evtimova, 2017).
For each method, we follow the prescribed training procedures since they also work with MNIST or Omniglot dataset.
However, for a fair comparison, we also experimented with the following encoder/decoder architecture and report the best results:
[Equation]
Our NRI implementation are based on (Kipf et al., 2018; Fetaya, 2019) where we used the NRIEncoder to infer the relationhip between a ( i ) and
b ( i ) ; the inferred relationship z
( i ) , together with ( a ( i ) , b
( i ) ) , are then feed into the NRI-Decoder to predict y a ( i ) and y b ( i ) which are reconstructions for a ( i ) and
b ( i ) , respectively.
To account for the high-dimensional input data, we used two MLPs (one for each a ( i ) and b ( i ) ) to extract lower dimensional feature vectors x a ( i ) ∈ R 10 and x b ( i ) ∈
Rthat serve as input to NRI-Encoder:
[Equation]
Similarly, we insert two MLPs (with the same architecture as above) in front of NRI-Decoder to extract data feature vectors and append two MLPs after NRI-Decoder to convert the predicted feature vectors back to the high-dimensional data space:
[Equation]
All MLPs are trained jointly with NRI encoder and decoder.
E A DDITIONAL EXPERIMENTAL RESULTS AND ANALYSIS E. MNIST AND O MNIGLOT RELATIONAL LEARNING RESULTS ANALYSIS
At ﬁrst glance, the results in Fig.
3c resemble that of style-transfer, but they are fundamentally different: in style-transfer, the image b ( r,c ) is generated by applying the style of b ( r ) s to the content of a ( c ) , whereas VRL generates image b ( r,c ) by applying the relational property of ( a s , b ( r ) s ) to the image a ( c ) .
It is evident from Fig.
3c that predicted images b ( r,c ) do not share similar style to b ( r ) s , but rather the same relative rotational relationship w.r.t.
a ( c ) and a s .
A close inspection of the results in Table 1 shows that the seemingly simple relational learning tasks is in fact very challenging even for specialized methods such as STN-Afﬁne and STN-Rotate that only learn about spatial transformations.
Comparing VRL with VAE, we can see that VRL achieved better performance than VAE with more efﬁcient learning—VAE were only able to solve the simpler 3-relationship tasks by adopting a high dimensional latent space; in contrast, VRL were able to solve all tasks with a compact 2-D latent space.
Further comparing VAE with GMVAE and AAE shows that regularizing the latent space, as done in AAE and GMVAE, can degrade the performance of relational learning.
LICM failed all relational learning tasks due to the fact that the transformed and canonical distributions overlaps completely (both a ( i ) and b ( i ) are rotations of
MNIST or Omniglot images) and it is not possible for a committee of experts to distinguish the two.
NRI’s inability to solve the relational learning tasks can be attributed to the fact that it is designed to learn in a dynamical systems, but more importantly, GNN is not guaranteed to learn an independent relative relationship.
For our ﬁnal comparison, we present results from applying InfoGAN to the
MNIST relational learning task in Sec.
5.1 (Chen et al., 2016).
InfoGAN has demonstrated its ability to learn disentangled representations (represented by structured latent codes c 1 , c 2 , ..., c L ) through generative modelling.
Although inferring latent codes for a given data point is a non-trivial task for
InfoGAN, we can examine the learned latent representation by manipulating the latent codes and visually inspect the generated random samples.
We modeled the latent codes with one categorical code c 1 ∼ Cat ( K = 10 , p = 0 . 1) and two continuous codes c 2 , c 3 ∼ Unif ( − 1 , 1) .
Figure 7 shows examples of generated images from manipulating the latent codes and it is clear that none of the latent codes distinctively capture the full range of relative rotational relationships.
In summary, we
[Figure]
argue that a major challenge for applying existing methods to relational learning problem is that they learn a single representation that encodes both relational and absolute properties and it is difﬁcult to dissect the relational property from the learned representation.
E. Y ALE RELATIONAL LEARNING RESULTS ANALYSIS
We can see in Fig.
5b that VRL learned a relational property that accurately differentiates relative expression changes irrespective of the subject.
From Fig.
6b, we can see that VRL identiﬁes four distinct relative illumination changes (“L-T”, “L-F”, “R-T”, “F-R”) but collapses “F-T” and “LR” into a single cluster.
At ﬁrst glance, this result seems to suggest that VRL was not able to distinguish between “F-T” and “L-R”.
However, we argue that it is indeed possible to combine the two relative relationships without loss of information.
In fact, we can give the combined relationship an intuitive name—“Opposite direction of illumination” (“Oppo.”) which is easier to understand if we rewrite “front” as “down”.
It is easy to see that the reduced set of relative relationships R c = { “L-T”, “R-T”, “F-R”, “L-F”, “Oppo.” } discovered by VRL fully and unambiguously characterizes the data in X F l (thus no loss of information) since: 1.
any ( a ( i ) , b ( i ) ) can be characterized by one and only one relative relationship in R c ; 2.
for any given a ( i ) , each compatible relative relationship in R c applied to a ( i ) leads to a unique b ( i ) .
While we lost the identiﬁability of “F-T” and “L-R” by combining them into “Oppo.”
, it is consistent with the relational learning goal—“Oppo.”
is a relational property that does not depend on individual illumination condition (absolute property).
Although our initial expectation of learning a complete set of relative relationships is certainly reasonable and more intuitive, it is quite surprising and unexpected that VRL is capable of discovering—in a completely unsupervised manner—a non-obvious set of relative relationships that is equally valid, and yet more compact.
E. R ELATIONAL LEARNING WITH MULTIPLE RELATIONSHIPS
Here, we setup a more complex relational learning task that includes both relative rotational and scaling relationships.
We constructed a paired MNIST dataset X M
= { ( a ( i ) , b ( i ) )
| i ∈
[1 ..N ] } where each a ( i ) is a randomly rotated and scaled (by a factor × 0 .
66 or × 1 ) MNIST image and b ( i ) is another random rotation (by 0 ◦ , 72 ◦ , 144 ◦ , 216 ◦ , or 288 ◦ ) and scaling (by × 1 or × 1 . 5 ) of a ( i ) .
Note that there are a total of 10 decoupled relative relationships between ( a ( i ) , b ( i ) ) in X M(combinations of 5 rotational and 2 scaling transformations).
We trained VRL on X M 10 following the same training procedure as described in Appendix D but with larger MLPs for f pθ ( a ( i ) , z ( i ) ) and f qφ ( a (cid:48) ( i ) , b (cid:48) ( i ) )
(cf.
Eq.
11, 13) to account for the increased problem complexity:
[Equation]
The inference result is shown in Fig.
8a, where we can see that the approximated posterior accurately cluster (discriminate) data with the same (different) relative relationship together (apart).
Examples of images predicted by direct sampling in the latent space are shown in Fig.
8b.
E. R ELATIONAL LEARNING WITH CONTINUOUS RELATIONSHIPS
Lastly, we present an example with a continuous relational property.
Based on the MNIST dataset, we constructed a paired dataset X M c = { ( a ( i ) , b ( i ) )
| i ∈
[1 ..N ] } where both a ( i ) and b ( i ) are random rotation of the same MNIST image.
In this case, there is a continuous (and decoupled) relative rotational relationship between ( a ( i ) , b ( i ) ) .
We trained VRL on X M c with the same model setup as in Appendix E.3.
A scatter plot of the relational property inferred by a trained posterior is shown in Fig. 9a, and examples of images predicted by direct sampling in the latent space (denoted by markers “ × ” in Fig. 9a) are shown in Fig.
9b.
From Fig.
9 we can see that VRL learned an independent relational property that encodes a continuous relative rotational relationship; however, there is a small region in Fig.
9a with overlapping relational property that leads to an ambiguous relationship interpretation ( 120 ◦ vs. 240 ◦ ).
This ambiguity is likely caused by compressing the relative relationship down to a 2-D latent space, z ∈ R 2 , and motivates us to adopt a higherdimensional latent space, e.g., z ∈ R 3 .
Figure 10 shows inference result from repeating the previous experiment but with adopting z ∈ R 3 .
We can see that VRL learned a three-dimensional relational property that unambiguously represents the underlying continuous relative rotational relationship. − [Figure] −
[Figure]
3 z 1 −−−−    z(a)
z (1) z (2) z (3) z (4) z (5) z
(6) z (7)
z
[Figure]
[Figure]
[Figure]
−−−  2 z−−
− z       (a) z (1) z (2) z
(3) z (4) z (
[Figure]