R ELATIONAL L EARNING WITH V ARIATIONAL B AYES
Anonymous authors
Paper under double-blind review
A BSTRACT
In psychology, relational learning refers to the ability to recognize and respond to relationship among objects irrespective of the nature of those objects.
Relational learning has long been recognized as a hallmark of human cognition and a key question in artiﬁcial intelligence research.
In this work, we propose an unsupervised learning method for addressing the relational learning problem where we learn the underlying relationship between a pair of data irrespective of the nature of those data.
The central idea of the proposed method is to encapsulate the relational learning problem with a probabilistic graphical model in which we perform inference to learn about data relationships and other relational processing tasks.
I NTRODUCTION
American Psychological Association deﬁnes relational learning as (VandenBos & APA, 2007):
Deﬁnition 1.1 ( Relational learning ) .
Learning to differentiate among stimuli on the basis of relational properties rather than absolute properties.
In other words, relational learning refers to the ability to recognize and respond to relationship (called relational property ) among objects irrespective of the nature of those objects (called absolute property ).
Relational learning has long been recognized as a hallmark of human cognition, and there has been substantial research showing that adequate cognitive capacity is necessary for relational processing (Biederman, 1987;
Medin et al., 1993; Holyoak, 2012; Doumas & Hummel, 2013;
Gentner, 2016).
As a machine learning application, relational learning can provide new insight into data analysis by dissecting information in the data into relational property and absolute property.
However, in order to discover relationship patterns among raw and unknown data, relational learning is only truly useful if it can be achieved without supervised data.
A key challenge in learning relational property with machine learning-based methods is that relational property is an abstract construct; unlike absolute property, which is based on observable data and can be quantitatively measured, relational property is an abstract quantity that is difﬁcult to objectively quantify, especially when the learning is unsupervised.
In this work, we propose an unsupervised learning method—variational relation learning (VRL)—for addressing the relational learning problem.
The proposed method is completely unsupervised, which means that the learning does not require a labeled training dataset nor training examples that have the same (or different) relational property.
At its core, VRL encapsulates the relational learning problem with a probabilistic graphical model (PGM) in which we perform inference to learn about relational property and other relational processing tasks.
Furthermore, our main learning algorithm is derived from the PGM using ﬁrst principles, which gives us the ﬂexibility to use any compatible computational inference method and still retains the desired properties of the proposed method.
Our contribution in this paper is threefold.
First, we propose a PGM that encapsulates the relational learning problem.
Second, we formulate various relational processing tasks as performing inference and learning in the PGM.
Third, we propose an efﬁcient and effective learning algorithm that can be trained end-to-end and unsupervised.
P ROBLEM DEFINITION
We begin with formulating the relational learning problem as a machine learning problem: we observed a paired dataset X
=
( a ( i ) , b ( i ) )
| i ∈
[1 ..N ] } consisting of N i.i.d samples generated from a joint distribution p ( a ∈A , b ∈B ) ; our goal is to learn a relational property between a ( i ) and b ( i ) irrespective of their absolute property.
Furthermore, we want the learning to be unsupervised, e.g., we do not require a labeled dataset, such as ( a ( i ) , b ( i ) , z ( i ) )
where z ( i ) is a target variable indicating ( a ( i ) , b ( i ) )
’s
relational property, nor do we require training examples that have the same (or different) relational property.
There are two distinct features that separate our problem formulation from other unsupervised learning problem formulations:
We dissect the information in X into relational property and absolute property ; relational property characterizes the relationship between a ( i ) and b ( i ) , whereas absolute property represents speciﬁc features that independently describe a ( i ) and b ( i ) .
Our goal is to learn a relational property among X irrespective of its absolute property, i.e., we want to learn a relational property that is decoupled from the absolute property.
In addition, we are interested in two related relational processing tasks: relational discrimination 1 and relational mapping 2 (VandenBos & APA, 2007).Relational discrimination allows us to differentiate ( a ( i ) , b ( i ) ) from ( a ( j ) , b ( j ) ) based on their relational properties, while relational mapping allows us to apply the relational property of ( a ( i ) , b ( i ) ) to a different set of data, for example, deduce that b ( j ) is related to a ( j ) in the same way that b ( i ) is related to a ( i ) .
M ETHOD
Here we introduce the proposed VRL method for addressing the relational learning problem and discuss various optimization challenges unique to VRL.
3. V ARIATIONAL RELATION LEARNING
The proposed VRL method consists of two parts: ﬁrst, we encapsulate the relational learning problem with a PGM, called VRL-PGM; we then formulate various relational processing tasks as performing inference and learning in VRL-PGM.
[Figure]
The VRL-PGM model, shown in Fig.
1, generates data a , z , and b by sampling from PDFs that come from parametric families of distributions— p θ ( a ) , p θ ( z ) , p θ ( b | a , z ) —that are differentiable almost everywhere with respect to (w.r.t.)
a , z , and θ .
In practice, we observe only a set of independent realizations { ( a ( i ) , b ( i ) )
| i ∈
[1 ..N ] } while the true parameter θ ∗ and the corresponding latent variables z ( i ) are unobserved.
A key property of VRL-PGM is that random variables a
and z are independent but not conditionally independent given b , i.e., p θ ( a , z )
=
p θ ( a ) p θ ( z ) , p θ ( a , z | b ) (cid:54) =
p θ ( a | b ) p θ ( z | b ) (Bishop, 2006).
In VRL-PGM, the absolute property can be interpreted as representing the dependency between a and b , i.e., features in a that can be used to predict b , while
Deﬁnition ( Relational mapping )
The ability to apply what one knows about one set of elements to a different set of elements.
the relational property , represented by the latent variable z , can be interpreted as any additional information not found in a but can help to better predict b .
The key requirement that the learned relational property be decoupled from the absolute property is enforced by VRL-PGM’s construction where z , which represents the relational property, and a , from which the absolute property is derived, are independent.
We note that the proposed VRL-PGM is based a directed acyclic graph (DAG) that artiﬁcially introduces causal relationship between a and b as well as z and b .
VRL-PGM reﬂects our priority and compromise for using a DAG to represent the original abstract relational learning problem: we sacriﬁce some identiﬁability of the original abstract problem but we gained a rigorous and mathematical tractable PGM while achieving our primary objective of learning a decoupled (independent) relational property.
Additional discussions on the connection between VRL-PGM and the original relational learning problem is provided in appendix D.1.
Having established VRL-PGM, our primary learning objective is to approximate the unknown true likelihood function p θ ( b | a , z ) and posterior p θ ( z | a , b ) (note that by observing b , random variables z and a are no longer independent).
Learning p θ ( z | a , b ) provides us a way to infer ( a ( i ) , b ( i ) ) ’s
relational property z ( i ) ; moreover, it serves as a basis for performing relational discrimination where we compare relational properties between different pairs of data.
Learning p θ ( b | a , z ) allows us to perform relational mapping where we use the relational property of ( a ( i ) , b ( i ) ) to map from a ( j ) to b ( j ) , i.e., b ( j ) ∼ p θ ( b | a ( j ) ,
z ( i ) ) where z ( i ) ∼ p θ ( z | a ( i ) , b ( i ) ) .
3. V ARIATIONAL LOWER BOUND
We estimate the parameter for p θ ( b | a , z ) by following the maximum-likelihood (ML) principle, and approximate the true posterior p θ ( z | a , b ) with variational Bayesian approach.
More speciﬁcally, we use a variational distribution q φ ( z | a , b ) , parameterized by φ , to approximate the unknown (and often intractable) true posterior through maximizing a variational lower bound (Bishop, 2006).
To derive such a lower bound, we ﬁrst write the log-evidence as log p θ ( X ) =
log p θ ( { ( a ( i ) , b ( i ) )
| i ∈
[1 ..N ] } )
=
[Equation]
Ni =1 log p θ ( a ( i ) , b ( i ) ) , where each term in the summation can be expressed as:
[Equation]
The ﬁrst term on the RHS is the KL-divergence from p θ ( z | a ( i ) , b ( i ) ) to q φ ( z | a ( i ) , b ( i ) ) , which provides a measure of dissimilarity between the two distributions; the second term on the RHS continues as:
[Equation]
where in the second line we use the fact that random variables a and z are independent.
Substitute
Eq.
(2) back in (1) and rearrange terms gives us:
[Equation]
where
[Equation]
Since KL-divergence is non-negative, L ( θ, φ ; a ( i ) , b ( i ) ) (abbreviated as L ( i ) for notation compactness) serves as a lower bound for the conditional log-likelihood log p θ ( b ( i ) | a ( i ) ) .
Maximizing L ( i ) w.r.t.
φ and θ gives us both a ML estimate for p θ ( b | a , z ) (by maximizing the ﬁrst term inside the expectation in Eq.
(4)) and a lower KL-divergence (the better q φ ( z | a ( i ) , b
( i ) ) approximates the true posterior p θ ( z | a ( i ) , b ( i ) ) )
as the conditional log-likelihood log p θ ( b ( i ) | a ( i ) ) does not depend on φ .
The lower bound L ( i ) can be maximized with gradient ascend methods; however, its gradients w.r.t.
φ is difﬁcult to obtain: the expectation in Eq.
(4) is taken w.r.t.
the distribution q φ ( z | a ( i ) , b ( i ) ) , which is a function of φ (Paisley et al., 2012).
To obtain efﬁcient estimators for both L ( i ) and its gradients, we adopt the reparameterization trick developed in Kingma & Welling (2014) where the random variable z is expressed as a transformation of another random variable (cid:15) ∼ p ( (cid:15) ) that is independent of a , b , and φ :
z = g ( (cid:15) , a ( i ) , b
( i ) , φ ) where g is some differentiable and invertible transformation.
Given such a change of variable, the lower bound L ( i ) can be rewritten as:
[Equation]
where z = g ( (cid:15) , a ( i ) , b ( i ) , φ ) and (cid:15) ∼ p ( (cid:15) ) .
Note that the expectation in Eq.
(5) is taken w.r.t.
p ( (cid:15) ) and we can now approximate L ( i ) with a Monte Carlo estimator:
[Equation]
where z ( i,l ) =
g ( (cid:15) ( i,l ) , a ( i ) , b ( i ) , φ ) and (cid:15)
( i,l ) ∼ p ( (cid:15) ) .
The lower bound for a minibatches of data
X M = { ( a ( i ) , b ( i ) )
| i ∈ [1 ..M ] } can be approximated by (cid:101) L ( θ, φ ; X M ) =
NM (cid:80) Mi
=1 (cid:101) L ( i ) .
And ﬁnally, the gradients ∇ θ,φ (cid:101)
L ( θ, φ ; X M )=
NM (cid:80) Mi
=1 ∇ θ,φ (cid:101) L ( i ) can be computed in a straightforward manner and used to update the parameters θ and φ with stochastic optimization methods, such as
SGD.
3. O PTIMIZATION CHALLENGES
The proposed VRL method introduces unique challenges to the variational lower bound optimization problem (see Sønderby et al.
and Bowman et al.
for other known challenges).
To explain these challenges, we ﬁrst break down
VRL’s parameter updating process into the following steps (using a single datapoint as an example): (1) a datapoint ( a ( i ) , b ( i ) ) is selected; (2) sample z ( i ) ∼ q φ k
( z | a ( i ) , b ( i ) ) by using the current parameter φ k ; (3) evaluate (cid:101) L ( i ) by using φ k , θ k ; (4) calculate gradients g = ∇ θ k ,φ k (cid:101)
L ( i ) ; (5) use gradients g to update φ k , θ k and get new parameters φ k +1 , θ k +1 .
This parameter updating process can be depicted with an information ﬂow diagram shown in Fig.
2a.
Ideally, we would like every path in Fig.
2a to contribute to the evaluation of all the terms in its reachable nodes in order to obtain meaningful gradients for updating its associated parameters; however, there are two situations where this is not the case.
The ﬁrst situation, called p θ ( b ( i ) | a ( i ) , z ( i ) ) , p θ ( z ( i ) )
z ( i ) ∼ q φ ( z | a ( i ) , b ( i ) )
b ( i ) φ a ( i ) φ θ a ( i ) θ (a) p θ ( b ( i ) | a ( i ) , z ( i ) ) , p θ ( z ( i ) )
z ( i ) ∼ q φ ( z | a (cid:48) ( i ) , b (cid:48) ( i ) )
b (cid:48)
( i ) φ a (cid:48)
( i ) φ θ a ( i ) θ (b) information-shortcut , occurs when the learning of p θ ( b ( i ) | a ( i ) , z ( i ) ) rely entirely on the dashed path in Fig.
2a; more speciﬁcally, the dashed (shortcut) path directly propagates b ( i ) through z ( i ) to p θ ( b ( i ) | a ( i ) , z ( i ) ) and, as a result, the relational property z ( i ) may learn only to encode the absolute property of b ( i ) .
The second situation, called deterministic-mapping , occurs when b ( i ) procedure VRL
( X , p ( (cid:15) ) , D ) (cid:46)
If RPDA not available, D = { id( · ) | ( a , b )
= i
d( a , b ) } Initialize parameters θ, φ while not convergence of parameters ( θ, φ ) do
Sample minibatch X M = { ( a ( i ) , b ( i ) )
| i ∈ [1 ..M ] } from X .
Sample (cid:15) ( i,l ) ∼ p ( (cid:15) ) , r ( i ) ∼ U ( R ) , i =1 , ..., M, l =1 , ..., L .
Run RPDA and obtain ( a (cid:48) ( i ) , b (cid:48) ( i ) )
=
d ( a ( i ) , b ( i ) ; r ( i ) ) , i =1 , ..., M .
Compute gradients g = ∇ θ,φ (cid:101)
L (cid:0) θ, φ ; X M (cid:1) =
NM (cid:80) Mi
=1 ∇ θ,φ (cid:101) L ( i ) RPDA (see Eq.
Update parameters θ, φ using gradients g (e.g. SGD).
end while return θ, φ end procedure can be fully characterized by a ( i ) ; in this case, the learning of p θ ( b ( i ) | a ( i ) , z ( i ) ) may rely only on the dash-dotted path in Fig.
2a.
While both situations can be viewed as a overﬁtting problem, deterministic-mapping says more about the data itself and exposes a potential limitation of VRL: a decoupled relational property is primarily learned through exploring information not found in a but can help to better predict b , and when a fully characterizes b , VRL no longer need to explore information beyond a to predict b and this may prevent VRL from learning a meaningful relational property.
On the other hand, information-shortcut is caused by short-cutting the parameter updating process, which we may overcome with additional regularization techniques.
Here we propose two approaches for mitigating the information-shortcut problem by disrupting the ﬂow of information passing through the shortcut path.
In the ﬁrst approach, we restrict the ﬂow of information by constraining the expressiveness of the latent variable z ; for example, by adopting an informative prior with restrictive constraint, such as p θ ( z ) =
N ( z ; 0 , σ 2 I ) , σ (cid:28) 1 , or representing z with a discrete random variable (assuming we know a priori the underlying relational proproty are discrete).
In the second approach, we propose a novel data augmentation strategy—relationpreserving data augmentation (RPDA)—that aims to eliminate the shortcut path.
First, we deﬁne a set of relation preserving functions D = { d ( a , b ; r ) | r ∈ R (some index set) , d : A×B →A×B } where the data relationship is preserved in the following sense: p θ ( z
[Equation]
for ( a (cid:48) , b (cid:48) )
=
d ( a , b ; r ) , ∀ r .
Assuming we have access to such a D , the proposed RPDA strategy then seek to optimize a modiﬁed lower bound estimator (cid:101) L ( i ) RPDA :
[Equation]
[Equation]
) where ( a (cid:48) ( i ) , b (cid:48)
[Equation]
= d ( a ( i
[Equation]
, b ( i
[Equation]
; r ( i
[Equation]
, r ( i
[Equation]
∼
[Equation]
( R ) , and z ( i,l ) =
g ( (cid:15) ( i
[Equation]
l )
[Equation]
a (cid:48) ( i )
[Equation]
b (cid:48)
( i ) , φ ) , (cid:15) ( i,l ) ∼ p
( (cid:15) ) .
Note that due to the relation preserving property of D , we have q φ ( z
( i,l ) | a (cid:48)
( i ) , b (cid:48)
( i ) )
=
q φ ( z ( i,l ) | a ( i ) , b ( i ) ) and, therefore, (cid:101)
L ( i ) RPDA is equivalent to (cid:101) L ( i ) in Eq.
When we optimize with (cid:101) L ( i ) RPDA , the parameter updating process can be redrawn in
Fig.
2b, where now the learning of p θ ( b ( i ) | a ( i ) , z ( i ) ) can no longer rely solely on the shortcut path to propagate b (cid:48)
( i ) since it differs from b ( i ) by a non-deterministic factor r ( i ) .
In practice, it may seem unrealistic to assume that we can construct a set of RPDA functions D without extensive knowledge of the underlying relational property.
However, we can treat data augmentation as a form of regularization and construct a D that reﬂects our prior knowledge and belief of the underlying system (Ronneberger et al., 2015; Perez & Wang, 2017).
For example, if we want the learning to be rotation invariance (a common theme in computer vision applications), we can construct a D that consists of image rotation augmentations, e.g., d ( a , b ; r )=(rot( a , r ) , rot( b , r )) where rot( x , r ) rotates the image x by r ∈ R =
[0 , 360) degrees (note that both a and b are rotated by the same amount).
Additional remarks on RPDA and a detailed ablation study is provided in appendix D.2 and
C, respectively.
To summarize this section, the proposed VRL method with RPDA is described in Algorithm 1.
R ELATED WORK
Machine learning approaches for relational processing have gained increasing interest and attention in the literature.
Most of these methods focus on high-level cognitive tasks, such as visual Q&A and state prediction for complex-physics systems, and derive their relational processing capabilities from learning with clever designed neural networks (Hill et al., 2019; Santoro et al., 2017; Raposo et al., 2017; Battaglia et al., 2018; 2016; Wu et al., 2015;
Reed et al., 2015; van Steenkiste et al., 2018;
Chang et al., 2016; Fragkiadaki et al., 2015).
Our work differ from these methods in two ways: (1) our primary focus is addressing the relational learning problem where we want to learn a decoupled relational property; (2) we enforce the decoupling requirement on the learned relational property with a PGM, which gives us the ﬂexibility to use any compatible inference method or function approximation and still satisfy the decoupling requirement.
Conventional unsupervised learning methods can also be appied to our problem setting (Kingma & Welling, 2014; Goodfellow et al., 2014; Mikolov et al., 2013a;b; Song et al., 2007); however, these methods learn a single representation for the data with superimposing information about their relational and absolute property.
The difﬁculty of decoupling the relational property from the learned representation constitute a major obstacle to relational reasoning.
Other related work include methods on learning a disentangled representations with applications in style-transfer, image-to-image translation, domain adaptation, etc.
(Huang et al., 2018;
Chen et al., 2016;
Tenenbaum & Freeman, 1997; Higgins et al., 2017; Bousmalis et al., 2016; Mathieu et al., 2016; Tulyakov et al., 2017; Denton & Birodkar, 2017; Villegas et al., 2017; Donahue et al., 2017;
Shen et al., 2017)
Most of these methods strive to learn a disentangled representations of content and style (or pose for video sequence data) where content is generically deﬁned as the underling spatial structure, and style as the rendering of the structure.
In comparison, our work can be viewed as learning a disentangled representations of relational and absolute property ; however, we argue that style-content separation is fundamentally different from relational-absolute separation.
More speciﬁcally, we consider both style and content information as absolute property (both describe features of an individual data), while relational property provides new information on the (abstract) relationship between the paired data.
E XPERIMENT
In this section, we present experimental results from applying the proposed VRL method to a set of relational learning tasks designed with the MNIST dataset (LeCun & Cortes, 2010).
5. MNIST RELATIONAL LEARNING TASK
To setup a relational learning task, a paired dataset X
= { ( a ( i ) , b ( i ) )
| i ∈
[1 ..N ] } was generated by the following steps: (1) the MNIST dataset { ( x ( i ) , y ( i ) )
| i ∈ [1 ..T ] } , where x ( i ) and
y ( i ) are the digit images and their labels, was augmented with applying ﬁve evenly-spaced rotations to each of the image x
( i ) to get { ( x ( i,j ) , y
( i ) )
| i ∈ [1 ..T ] , j ∈
[1 .. 5] } (examples of augmented images are shown in Fig. 3); (2) individual datapoint ( a ( i ) , b ( i ) ) of X was chosen to be randomly rotated images of x ( i ) ,
i.e., X = { ( a ( i ) =
x ( j,k ) , b ( i ) =
x ( j,l ) ) | j ∼ U ([1 ..T ]) , k, l ∼ U ([1 .. 5]) } .
Note that there are ﬁve uniquely deﬁned rotational relationships between ( a ( i ) , b ( i ) ) in X and since they are randomly selected, the rotational relationship (relational property) is decoupled from the absolute properties of ( a ( i ) , b ( i ) ) .
Here, we use X to assess VRL’s capability of discovering the underlying relational property (rotational relationships) irrespective of their absolute property.
Additional MNIST relational learning tasks are introduces in appendix B.1.
The MNIST relational learning experiment presented above may seem contrived at ﬁrst glance but, upon deeper examination, represents a novel and unique problem setting that exempliﬁes a key relational learning challenge—learning a decoupled relational property—for existing unsupervised learning methods (see Sec. 4 for related work).
We argue that any unsupervised method that can successfully solve the above problem (or any relational learning problem) must, at a minimum, simultaneously accomplish the following two goals:
An unsupervised learning mechanism that captures and preserves the data relationships (relational property), e.g, capturing and preserving the rotational relationship between ( a ( i ) , b ( i ) ) during learning.
An unsupervised learning mechanism that decouples absolute property from the learned data relationships, e.g., learning a rotational relationship between ( a ( i ) , b ( i ) ) that does not depend on the absolute property (digit representation, rotation, etc.)
of individual a ( i ) , b ( i ) .
While most existing unsupervised methods are well-equipped to accomplish the ﬁrst goal, the second goal presents itself as a major challenge.
More speciﬁcally, to the best of our knowledge, most existing unsupervised methods seek to model all aspect of ( a ( i ) , b ( i ) ) and learn the data relationship either jointly with the absolute properties of a ( i ) , b ( i ) or as their derivatives (this includes graphbased methods that use graph-edges to represent relationships).
For such methodology, the learned relationship information necessarily entangles/couples with the absolute property; therefore, the fundamental relational learning challenge—decoupling relational property—is still present and needs to be addressed.
A key insight into why the proposed VRL framework is capable of overcoming this challenge is recognizing that VRL takes a more targeted approach and learns data relationship as a stand-alone entity (represented by the latent variable z ) that is designed, through the construction of
VRL-PGM, to be independent of a (however, we note that in VRL the learned relational property z is only independent of a but not b ; a discussion of this compromising fact and its implications is provided in appendix D.1).
5. I MPLEMENTATION
For these experiments, we adopted a two-dimensional latent variable z ∈ R 2 and let the prior p θ ( z ) be the bivariate normal distribution.
We let p θ ( b | a ( i ) , z
( i,l ) ) be a multivariate Bernoulli distribution whose probability parameters are computed from a given a ( i ) and
z ( i,l ) with an autoencoder-like neural network f dec θ (cid:0)
f enc θ ( a ( i ) ) ,
z ( i,l ) (cid:1) .
We let the approximated posterior q φ ( z | a (cid:48)
( i ) , b (cid:48)
( i ) ) be a bivariate Gaussian distribution with a diagonal covariance
N ( z ; µ ( i ) , ( σ ( i ) ) 2 I ) where µ ( i ) and σ ( i ) are the output of a neural network f qφ ( a (cid:48)
( i ) , b (cid:48)
( i ) ) .
For RPDA, we constructed a D that consists of image rotation augmentations: D = { (rot( a , r ) , rot( b , r )) | r ∈ [0 , 360) } .
Detailed experimental setup is described in appendix A.
5. R ESULTS
Relational discrimination.
We trained VRL on X and used the approximated posterior q φ ( z | a , b ) to infer the relational property of a hold-out dataset.
Figure 4a shows a scatter plot of the relational property inferred by VRL where we can see that the approximated posterior accurately cluster(discriminate) data with the same(different) rotational relationship together(apart).
Here, we compared VRL with variational autoencoder (VAE) (Kingma & Welling, 2014) and
InfoGAN (Chen et al., 2016), both of which can learn data representations of X in a completely unsupervised manner.
In the application of VAE, we adopted a 2-D latent space and use the encoder trained on
X to inferred the latent varible of a hold-out dataset; the resulting scatter plot is shown in Fig.
4b where we see that VAE failed to discriminate data based on their relational property (rotational relationship).
Next, InfoGAN has demonstrated its ability to learn disentangled representations (represented by structured latent codes c 1 , c 2 , ..., c L ) through generative modelling.
Although inferring latent codes for a given data point is a non-trivial task for InfoGAN, we can examine the learned latent representation by manipulating the latent codes and visually inspect the generated random samples.
We followed the examples in Chen et al.
and modeled the latent codes with one categorical code c 1 ∼ Cat ( K = 10 , p = 0 . 1) (model discontinuous variation in data) and two continuous codes c 2 , c 3 ∼ Unif ( − 1 , 1) (capture continuous variations in style).
Figure 5 shows
−−  2 z 1 −−   z+
+ + + + (
[Figure]
properties inferred
[Figure]
−−  4 z−−   z+
+ + + +
[Figure]
examples of generated images from manipulating the latent codes; it is clear that none of the latent codes (or a combination of them) distinctively capture the full range of relational property (rotational relationship).
Figures 4b and 5 illustrate a major challenge of using VAE and InfoGAN (and other related methods) for relational learning: these methods learn a single representation that encodes both relational and absolute property of X and it is difﬁcult to dissect the relational property from the learned representation.
Detailed experimental setup for both VAE and InfoGAN is described in appendix A.
[Figure]
Relational mapping.
We evaluated VRL’s learned likelihood function p θ ( b | a , z ) by visualizing the predicted images given a and z .
We chose a from a hold-out dataset and z from: (1) direct sampling in the latent space; (2) relational property inferred from a source datapoint ( a s , b s ) .
Figure 6a shows predicted images with z sampled from the latent space shown in Fig.
4a.
Figure 6b shows examples of relational mappings from a ( c ) to b ( r,c ) by applying the relational property inferred from a source datapoint ( a s , b ( r ) s ) .
At ﬁrst glance, the results in Fig.
6b resemble that of style-transfer, but they are fundamentally different: in style-transfer, the image b ( r,c ) is generated by applying the style of b ( r ) s to the content of a ( c ) , whereas VRL generates image b ( r,c ) by applying the relationship of ( a s , b ( r ) s ) to the image a ( c ) .
It is evident from Fig.
6b that predicted images b ( r,c ) do not share similar style to b ( r ) s , but rather the same rotational relationship w.r.t.
a ( c ) and a s .
Additional experimental results and discussion are provided in appendix B and D.3, respectively.
[Figure]
[Equation] [Equation]
[Equation] [Equation]
[Equation]
D ISCUSSION AND CONCLUSION
A core component of the proposed VRL method is approximating the intractable posterior with variational inference (VI) methods.
There is a vast literature on the subject of VI that we can leverage to further improve and extend VRL; for example, prior works have proposed ﬂexible and complex approximated posterior distributions that we can use to learn a rich posterior approximations for characterizing the relational property (Rezende & Mohamed, 2015; Dinh et al., 2014).
Another interesting idea to explore is learning a generative model for VRL: in this work, the primary learning objective is maximizing a variational lower bound derived from VRL-PGM; however, with the advent of computationally efﬁcient methods for learning a generative model, it would be interesting to include another learning objective that directly models the data generating aspect of VRL-PGM (Goodfellow et al., 2014; Larsen et al., 2015).
The proposed method comes with both advantages and disadvantages: the main advantage of VRL lies in its relational learning capabilities; however, this may also be one of its disadvantages.
More speciﬁcally, VRL can learn a decoupled relational property even when it is coupled with the absolute property, i.e., VRL is oblivious to the coupling information between the two properties (an example with coupled relational property is provided in appendix B.1.1).
Nevertheless, such information may be of interest to the user, and in this regard, VRL provides only a partial view of the data.
In conclusion, the proposed VRL method is an efﬁcient and effective unsupervised learning method for addressing the relational learning problem where our goal is to learn a decoupled relational property.
By dissecting the data information into decoupled relational and absolute property , we hope VRL can bring new insight into everyday data analysis and ultimately ﬁnd applications for a wide variety of problems.
A E XPERIMENTAL SETUP
Recall that we adopted a two-dimensional latent variable z ∈ R 2 and let the prior p θ ( z ) be the bivariate normal distribution.
For binary valued data (e.g., MNIST dataset), we let the likelihood function p θ ( b ∈ B | a ( i ) ,
z
( i,l ) ) of VRL be a multivariate Bernoulli distribution whose probability parameters p ( i,l ) are computed from a given a ( i ) and
z ( i,l ) with an autoencoder-like neural network f
[Equation]
[Equation]
where Conv ( · ) is a strided (stride 2) convolutional layer and Conv T ( · ) is a transposed convolutional layer.
We used batch-normalization after most layers and leaky rectiﬁed linear units (with leaky rate 0.01) as nonlinear activation function.
We let the approximated posterior q φ ( z | a (cid:48)
( i ) , b (cid:48)
( i ) ) of VRL be a bivariate Gaussian distribution with a diagonal covariance
N ( z ; µ ( i ) , ( σ ( i ) ) 2 I ) where µ ( i ) and σ ( i ) are the output of a neural network f qφ ( a (cid:48)
( i ) , b
[Equation]
i ) ) :
f
[Equation]
( i ) , b (cid:48)
( i ) ]
→ Conv (3 x 3 x 8) → Conv (3 x 3 x 32) → Conv (3 x 3 x 128) →
FC (4) →
[ µ ( i ) , σ ( i ) ] .
We sampled from the posterior z ( i,l ) ∼ N ( z ; µ ( i ) , ( σ ( i ) ) 2 I ) using z ( i,l ) =
g ( (cid:15) ( i,l ) , a (cid:48) ( i ) , b (cid:48) ( i ) , φ ) = µ ( i ) + σ ( i ) (cid:12) (cid:15) ( i,l ) where (cid:15) ( i,l ) ∼ p ( (cid:15) )
=
N ( 0 , I ) and set L = 1 .
We used image rotation augmentations: D = { (rot( a , r ) , rot( b , r )) | r ∈ [0 , 360) } for constructing RPDA.
In this case, the learning objective (cid:101) L ( i ) RPDA in Eq.
(7) can further be derived as (see next section for the derivation):
[Equation]
where ( µ ( i ) , σ ( i ) )
=
f qφ ( a (cid:48) ( i ) , b (cid:48) (
[Equation] [Equation]
= f
dec
[Equation]
f enc
[Equation]
(cid:17)
[Equation]
) , (cid:15) ( i, 1) ∼ N ( 0 , I ) , ( a (cid:48) ( i ) , b (cid:48) ( i ) )
= (rot( a ( i ) , r ( i ) ) , rot( b ( i ) , r ( i ) )) , r ( i ) ∼ U ([0 , 360)) .
Parameters θ and φ were jointly trained to maximize Eq.
(8) using Adam optimizer (learning rate = 0.0001, β 1=0 .
9 , β 2=0 . 999 ) (Kingma & Ba, 2014); minibatches of size
M =100 were used.
In our VAE implementation we used the following network architecture for encoder, Enc ( a ( i ) , b ( i ) ) , and decoder, Dec ( z ( i,l ) ) :
Enc :[ a ( i ) , b ( i ) ]
→ Conv (3 x 3 x 16) → Conv (3 x 3 x 64) → Conv (3 x 3 x 256) → FC (4) → [ µ ( i ) ∈ R 2 , σ ( i ) ∈ R 2 ]
Dec :[ z ( i,l ) ∈ R 2 ] → FC → Conv T (3 x 3 x 256) → Conv T (3 x 3 x 64) → Conv T (3 x 3 x 16)
→ Conv (1 x 1 x 1) Sigmoid −−−−→ [0 , 1] dim ( B ) .
We used Adam optimizer with the same hyperparameter setup as before.
For InfoGAN implementation, we followed the setup described in Chen et al.
(2016) except that each datapoint consists a paired MNIST image ( a ( i ) , b ( i ) ) .
A. D ERIVATION OF TRAINING OBJECTIVE FUNCTION
Our derivation for Eq.
(8) largely follow the work of Kingma & Welling (2014); here for completeness we outline the key steps.
First, Eq.
(4) in Section 3.2 can equivalently be expressed as:
[Equation]
where z = g ( (cid:15) , a ( i ) , b ( i ) , φ ) , (cid:15) ∼ p ( (cid:15) )
[Equation]
and g is some differentiable and invertible transformation.
We can approximate L ( i ) in Eq.
(9) with a Monte Carlo estimator (cid:101) L ( i ) :
[Equation]
where z ( i,l ) =
g ( (cid:15) ( i,l ) , a ( i ) , b ( i ) , φ ) and (cid:15)
( i
[Equation]
l ) ∼ p ( (cid:15) ) .
Based on the RPDA functions D = { d ( a , b ; r ) | r ∈ R } and its relation preserving assumption that q φ ( z | a ( i ) , b ( i ) )
=
q φ ( z | a (cid:48)
( i ) , b (cid:48)
( i ) ) for ( a (cid:48) ( i ) , b (cid:48)
( i ) )
=
d ( a ( i ) , b ( i ) ; r ) , ∀ r (see Section 3.3), we can express (cid:101) L ( i ) in Eq.
(10) equivalently as (cid:101) L ( i ) RPDA :
[Equation]
where
[Equation]
The ﬁrst term in Eq.
(11) is the KL-divergence from p θ ( z ) to q φ ( z | a (cid:48)
( i ) , b (cid:48)
( i ) ) , which can be computed analytically given that we assume p θ ( z ) =
N ( 0 , I ) and q φ ( z | a (cid:48) ( i ) , b (cid:48) ( i ) ) =
N ( z ; µ ( i ) , ( σ ( i ) ) 2 I )
[Equation]
[Equation]
The likelihood function p θ ( b ( i ) | a ( i ) , z
( i,l ) ) is deﬁned as a multivariate Bernoulli distribution whose probability parameters p ( i,l ) are computed from the neural network f dec θ (cid:0) f enc θ ( a ( i ) ) ,
z ( i,l ) (cid:1) and we have:
[Equation]
By substituting Eq.
(12) and (13) back in (11) and recall that z ( i,l )
=
g ( (cid:15) ( i,l ) , a (cid:48) ( i ) , b (cid:48) ( i ) , φ ) = µ ( i ) + σ ( i ) (cid:12) (cid:15) ( i,l ) , p ( (cid:15) )
=
N ( 0 , I ) , D = { (rot( a , r ) , rot( b , r )) | r ∈ [0 , 360) } , L =1 , we arrive at Eq.
B A DDITIONAL EXPERIMENTAL RESULTS B. A DDITIONAL MNIST RELATIONAL LEARNING EXPERIMENTS
Here, we provide additional experimental results based on the paired MNIST dataset constructed in
Section 5.
B.1. MNIST EXAMPLE WITH COUPLED RELATIONAL PROPERTY
First, to further test the robustness of the proposed method, we considered a scenarios where the underlying relational property is coupled with the absolute property.
An example dataset X 2 was constructed with the rotational relationship between each of the datapoint completely determined by its digit label:
X 2 = { ( a ( i ) =
x ( j,k ) , b ( i ) =
x ( j,l +1) )
| j ∼ U ([1 ..T ]) , k ∼ U ([1 .. 5]) , l =
( k + (cid:98) y
( j ) / 2 (cid:99) )
mod 5 } .
In this case, it is possible to infer the relational property solely based on the image representation of the digit (absolute property of a ( i ) ), for example, a ( i ) ∈
→ 0 ◦ (read: if a ( i ) is recognized as either digit 0 or 1 , the rotational relationship between ( a ( i ) , b ( i ) ) is 0 ◦ ) , a ( i ) ∈ [ ’2’, ’3’ ]
→ 72 ◦ , a ( i ) ∈ [ ’4’, ’5’ ] → 144 ◦ , a ( i ) ∈ [ ’6’, ’7’ ] → 216 ◦ , a ( i ) ∈ [ ’8’, ’9’ ] → 288 ◦ .
The question then arise: is VRL capable of learning a decoupled relational property even when it is coupled with the absolute property?
To test this idea, we trained VRL (with the same setup as that described in appendix A) on X 2 but validated it on a hold-out dataset with a decoupled relational property (much like how X was constructed).
The resulting scatter plot is shown in Fig.
7, where we can see that the approximated posterior q φ ( z | a , b ) accurately cluster(discriminate) data with the same(different) rotational relationship together(apart).
This result shows that VRL was indeed capable of learning a decoupled relational property irrespective of the digit representation of a ( i ) .
If this were not the case, we would expect to see a scatter plot with heavily overlapped relationship labels since the validation dataset was constructed with random rotational relationships.
Examples
[Figure]
(cid:35)
[Equation]
: 72 ◦ , + : 144 ◦ , × :216 ◦ , ♦ :288 ◦ ).
of images predicted by the learned likelihood function p θ ( b | a , z ) are shown in Fig.
8, where
Fig.
8a shows predicted images based on direct sampling in the latent space (shown in Fig. 7) and
Fig.
8b shows examples of relational mappings.
Figures 8a and 8b further corroborate our claim that
[Figure]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation]
VRL is capable of learning a decoupled relational property and generalizing it to unseen data, e.g.,
VRL trained on X 2 learned to rotate any digit by any amount despite not having seen most of the digit-rotation pairs during training.
Note that this experiment is an example of deterministic-mapping discussed in Section 3.3, nevertheless VRL was able to utilize all paths in Fig.
2b effectively to learn about the data.
B.1. MNIST EXAMPLE WITH MULTIPLE RELATIONAL PROPERTY
Next, we setup a more complex relational learning task that includes both rotational and resizing relationships between ( a ( i ) , b ( i ) ) : each MNIST image x ( i ) was augmented with ﬁve evenly-spaced rotations and three different resizing transformations to get { ( x ( i,j,k ) , y ( i ) )
| i ∈ [1 ..T ] , j ∈
[1 .. 5] , k ∈ [1 .. 3] } .
Examples of augmented images are shown in Fig.
9, where top, middle, and bottom row images are resized by a factor of × 0 . 66 , × 1 , and × 1 . 5 , respectively.
We considered a relational learning task where each datapoint ( a ( i ) , b ( i ) ) has a decoupled rotational and/or resizing relational property; more speciﬁcally, we constructed a dataset X 3 in the following way:
[Equation]
[1 .. 5]) , m ∼ U ([1 , 2]) , n ∼ U ( V ) } where
[Equation]
] , if m =
Note that in this case, b ( i ) is either the same size as a ( i ) or is × 1 . 5 larger than a ( i ) , and there are a total of 10 different relationships between ( a ( i ) , b ( i ) ) in X 3 (combinations of 5 rotational andresizing transformations).
We trained VRL on X 3 with the same model setup as that described in appendix A and used the trained model to infer the relational property of a hold-out dataset.
The inference result is shown in Fig.
10, where we can see that the approximated posterior q φ ( z | a , b ) accurately cluster(discriminate) data with the same(different) relationship together(apart).
Examples of images predicted by the learned likelihood function p θ ( b | a , z ) are shown in Fig.
11, where
Fig.
11a shows predicted images based on direct sampling in the latent space (shown in Fig. 10), and Fig.
11b shows examples of relational mappings.
These results are consistent with the ﬁndings presented and discussed in Section 5.3.
B.1. MNIST EXAMPLE WITH CONTINUOUS RELATIONAL PROPERTY
Finally, we present an example with a continuous relational property.
Based on the MNIST dataset { ( x ( i ) , y ( i ) )
| i ∈ [1 ..T ] } , we constructed a paired dataset X 4 in the following way:
[Equation]
where rot( x ( j ) , r ( i ) ) rotates the image x ( j ) by r ( i ) ∈ [0 , 360) degrees.
In this case, b ( i ) is a random rotation of a ( i ) and there is a continuous (and decoupled) rotational relationship between ( a ( i ) , b ( i ) ) in X 4 .
We trained VRL on X 4 with the same model setup as that described in appendix A and used the trained model to infer the relational property of a hold-out dataset.
A scatter plot of the relational property infered by the approximated posterior q φ ( z | a , b ) is shown in Fig.
12, and examples of images predicted by the learned likelihood function p θ ( b | a , z ) are shown in Fig.
13, where Fig. 13a shows predicted images based on direct sampling in the latent space (denoted by markers “ × ” in
[Figure]
z+0+ +144+ +288+0,x1.
+72, x1.5 +144, x1.
+216, x1.5 +288, x1.
(cid:35) ( blue ):0 ◦ , (cid:79) :72 ◦ , +:144 ◦ ,
[Figure] [Figure]
, ♦ :
[Figure]
(cid:50) :
[Figure]
, × 1 . 5 , (cid:3) :72 ◦ , × 1 . 5 , (cid:1) :144 ◦ , × 1 . 5 , (cid:52) :216 ◦ , × 1 . 5 , (cid:35) ( cyan ):288 ◦ , × 1 . 5 ).
[Figure]
[Equation]
[Figure]
applying
Fig. 12), and Fig.
13b shows examples of relational mappings.
From Fig.
12 and 13 we can see that
VRL learned a decoupled relational property that encodes a continuous data (rotational) relationship; however, there is a small region in Fig.
12 with overlapping relational property that leads to an ambiguous interpretation of the rotational relationship ( 120 ◦ vs. 240 ◦ ).
This ambiguity is likely caused by compressing the continuous data (rotational) relationship down to a two-dimensional latent space, z ∈ R 2 , and motivates us to adopt a higher-dimensional latent space, e.g., z ∈ R 3 .
Figureshows inference result from repeating the previous experiment but with setting z ∈ R 3 ; we can see that VRL learned a three-dimensional relational property that unambiguously represents the underlying continuous data (rotational) relationship.
[Figure]
[Figure]
B. Y ALE FACE RELATIONAL LEARNING EXPERIMENTS B.2. Y ALE FACE EXAMPLE WITH LEARNING EMOTION CHANGE
Here we present another relational learning example based on the Yale face dataset (Belhumeur et al., 1997).
The Yale face dataset consists of 15 subjects, each with 8 facial expressions andlighting conﬁgurations.
We extracted three facial expressions (happy, surprised, sad) of each subject to form { x ( i,j ) | i ∈ [1 .. 15] , j ∈ [1 .. 3] } .
Examples of face images are shown in Fig.
15a, where we center-cropped, resized to 64 × 64 , and normalized pixel values to be within [0 , 5] .
We constructed a dataset X F e where each datapoint ( a ( i ) , b ( i ) ) represents a subject with different facial expressions ( emotions ):
[Equation]
Our initial intention was to apply VRL to X F e to learn about the “emotional change” between ( a ( i ) , b ( i ) ) irrespective of the subject; however, because X F is an extremely limited dataset that z 1 − 2 − 1 0 z− 2 − 1 0 1z− 2 −
z
1 − 2 − 1 0 1z 2 − 2 − 1 0 z− 2 −0 z 1 − 2 − 1 0 1 2 z 2 − 2 − 1 0 1z− 2 −0 z 1 − 2 − 1 0 z− 2 − 1 0 1z− 2 −0 z− 2 − 1 0 1z 2 − 2 − 1 0 z− 2 −0 z− 2 − 1 02 z− 2 − 1 0 1z− 2 −
[Figure]
consists of only 45 different images (15 subjects, each with 3 facial expressions) [Figure] we settled for a less ambitious goal of learning an undirected emotional change, i.e., “happy → sad” = “sad → happy” and simply denoted as “happy-sad”.
In this case, there are three different emotional changes between ( a ( i ) , b ( i ) ) in X F : “happy-sad”, “happy-surprised”, “surprised-sad”.
To setup a VRL model for training, we adopted a two-dimensional latent variable z ∈ R 2 and let the prior p θ ( z ) be the bivariate normal distribution.
Since this is a real-valued dataset, we let p θ ( b ( i ) | a ( i ) , z
( i,l ) ) be a multivariate Gaussian distribution with a ﬁxed diagonal covariance
N ( b ( i ) ; µ ( i,l ) b , σ 2 I ) where σ = 0 .
1 and µ ( i,l ) b is computed from a given a ( i ) and
z ( i,l ) with an autoencoder-like neural network
[Equation]
and we have:
[Equation]
Again
[Equation]
we let the approximated posterior q φ ( z | a (cid:48)
( i ) , b (cid:48)
( i ) ) be a bivariate Gaussian distribution with a diagonal covariance
N ( z ; µ ( i ) , ( σ ( i ) ) 2 I ) where µ ( i ) and σ ( i ) are the output of a neural network f qφ
[Equation]
, b
[Equation]
[Equation]
Next, based on the premise that we are only interested in learning an undirected emotional change
[Equation]
we have p θ ( z | a , b ) =
p θ ( z | b , a ) and this prompted us to construct RPDA functions D with random image rotation and swapping operations: D = { swap (rot( a , r ) , rot( b , r )) | r ∈ [0 , 360) } where
[Equation]
Finally, we combine the above settings with z ( i,l ) = µ ( i ) + σ ( i ) (cid:12) (cid:15)
( i,l ) , p ( (cid:15) )
=
N ( 0 , I ) , L = 1 , and substituting Eq.
(12) and (14) back in (11) to derive the following lower bound estimator:
[Equation]
where ( µ ( i ) , σ ( i ) )
=
f qφ ( a (cid:48) ( i ) , b (cid:48) ( i ) ) ,
[Equation]
The rest of the training setup remains the same as that described in appendix A.
We trained VRL on
X F e and then used the approximated posterior q φ ( z | a , b ) to infer the relational property of X F e .
The inference result is shown in Fig.
15b, where we can see that VRL learned a relational property that accurately differentiates emotional changes irrespective of the subject.
B.2. Y ALE FACE EXAMPLE WITH LEARNING ILLUMINATION CONDITION CHANGE
Next, we present an experiment that learns relationship on “illumination condition changes” among the Extended Yale Face Database B (Georghiades et al., 2001).
The Extended Yale Face Database B contains 16128 images of 28 human subjects under 9 poses and 64 illumination conditions.
To setup a relational learning example, we extracted four illumination conditions (source of illumination: left, front, right, top) of each subject to form { x ( i,j ) | i ∈ [1 .. 28] , j ∈ [1 .. 4] } .
Examples of face images are shown in Fig.
16a, where we center-cropped, resized to 64 × 64 , and normalized pixel values to be within [0 , 5] .
We constructed a dataset X F l where each datapoint ( a ( i ) , b ( i ) ) represents a subject with different illumination conditions (lightings):
[Equation]
Like the “learning undirected emotional changes” example presented in appendix B.2.1, we apply
VRL to X F l to learn about the “undirected lighting changes” between ( a ( i ) , b ( i ) ) irrespective of the subject, i.e., “left → right” = “right → left” and simply denoted as “left-right”.
In this case, there are six different lighting changes between ( a ( i ) , b ( i ) ) in X F l : “left-right”(“L-R”), “front-top”(“F-T”), “left-front”(“L-F”), “left-top”(“L-T”), “front-right”(“F-R”), “right-top”(“R-T”).
We trained VRL on X F l with the exact same model and training setup as used in appendix B.2.1 and then used [Figure]
the approximated posterior q φ ( z | a , b ) to infer the relational property of X F l .
The inference result is shown in Fig.
16b where we can see that while VRL correctly identiﬁes four relationships (“L-T”, “R-T”, “F-R”, “L-F” represented by four distinct clusters in Fig. 16b), it collapses “F-T” and “L-R” into a single indistinguishable cluster .
At ﬁrst glance, this counterintuitive result— collapsing of “F-T” and “L-R”—seems to indicate that VRL was not able to learn meaningful representations for “F-T” and “L-R”; however, there is an elegant and logical explanation that justiﬁes this unexpected result. Upon closer examination, we can see that it is indeed possible to combine the relationships “F-T”, “L-R” without loss of information. In fact, we can give the combined relationship a meaningful name—“Opposite illumination direction” or “Oppo.” for short (in this interpretation it is more intuitive to write “front” as “down”).
With the newly formed compressed relationships R c = { “L-T”, “R-T”, “F-R”, “L-F”, “Oppo.” } , it is easy to see that R c is a valid set of relationships that fully characterizes the data (thus no loss of information) since: 1. any ( a ( i ) , b ( i ) ) can be characterized by one and only one relationship in R c ; 2. for a given a ( i ) , each compatible relationship in R c applied to a ( i ) leads to a unique b ( i ) . And while we lost the identiﬁability of “F-T” and “L-R”, combining them into “Oppo.” is consistent with the relational learning goal— relationship “Oppo.” is decoupled from the absolute property (illumination direction) of individual data. Finally, although the initial expection of learning a complete set of relationships is certainly logical and straightforward, it is quite surprising and unexpected that VRL is capable of discovering, in a completely unsupervised manner, an equally valid set of relationships that is more compact.
To summarize the Yale face experiments presented appendix B.2, we make the following two observations. First, when comparing our results with existing unsupervised leanring methods on face images (Song et al., 2007), we can see that existing methods cluster face images by its absolute property (e.g., subject identity, individual emotion, individual illumination condition, etc.) while the proposed VRL method clusters images by reltaionships (face images with different subject identity groups together in Fig. 15b and 16b). Second, it is worth noting that VRL does not learn the facial expression or illumination condition for each of ( a ( i ) , b ( i ) ) independently and then classify them based on their difference, but instead directly learns the emotional/lighting relationship between ( a ( i ) , b ( i ) ) .
C A BLATION STUDY ON RPDA
In this work, we introduced relation-preserving data augmentation (RPDA) as a practical data augmentation strategy for overcoming the information-shortcut problem—an unique overﬁtting problem introduced by VRL (see Section 3.3).
In order to evaluate the contribution of RPDA, we performed ablation study on the paired MNIST dataset X constructed in see Section 5.1.
First, recall that in Section 5.2 we constructed RPDA functions D with image rotation augmentations and optimized the following lower bound estimator (cf.
Eq.
[Equation]
where
[Equation]
In the ﬁrst ablation study, we experimented with removing RPDA from the VRL training, which amounts to optimizing with the original lower bound estimator (cid:101) L ( i ) in Eq.
With the rest of the training setup remained unchanged (see appendix A), we trained VRL on X without RPDA and used the trained model to infer the relational property of a hold-out dataset; the inference result is shown in Fig.
17a.
In the second ablation study, we repeated the VRL training, but instead of following the proposed RPDA strategy, we applied RPDA functions D in a conventional data augmentation routine.
More speciﬁcally, we optimized the lower bound (cid:101) L ( i ) in Eq.
(6) over a minibatch of data augmented with D , X (cid:48)
M = { ( a (cid:48) ( i ) , b (cid:48) ( i ) ) | i ∈ [1 ..M ] } , which leads to optimizing with the following lower bound estimator:
[Equation]
where
[Equation]
With the rest of the training setup remained unchanged (see appendix A), we trained VRL on X with maximizing (cid:101) L ( i ) DA and used the trained model to infer the relational property of a hold-out dataset; the inference result is shown in Fig.
17b.
Comparing the results from both ablation studies, shown in −−  2 z−−   z+ + + + + (a) −−  2 z−− [Equation] z+ + + + + (b)
[Equation]
: 72 ◦ , + : 144 ◦ , × : 216 ◦ , ♦ : 288 ◦ ): (a) relational property inferred by a VRL model that was trained without RPDA; (b) relational property inferred by a VRL model that was trained with applying RPDA functions D in a conventional data augmentation routine.
Fig.
17, with inference result from VRL training with RPDA, shown in Fig.
4a, we can make the following observations: ﬁrst, RPDA is a critical component necessary for VRL to learn a meaningful and decoupled relational property, especially when ﬂexible function approximations such as deep neural networks are used; second, by looking at the difference between (cid:101) L ( i ) DA and (cid:101)
L ( i ) RPDA (observe that (cid:101) L ( i ) DA applies RPDA functions D to every term in Eq.
(17), whereas (cid:101) L ( i ) RPDA applies D only to selected terms in Eq.
(16) according to the RPDA strategy), we can draw the conclusion that the improvements brought by RPDA (and its key innovation) comes not from what data augmentation functions are applied but how they are applied.
D A DDITIONAL R EMARKS D. R EMARKS ON VRL-PGM
The central idea of the proposed VRL method is to encapsulate the relational learning problem with a probabilistic graphical model—VRL-PGM—and then formulate various relational processing tasks as performing inference and learning in the graphical model.
Here we discuss aspects of the original relational learning problem (see Section 2) that differ from the proposed VRL-PGM (see Section 3.1).
First, the original problem speciﬁes that the relational property be decoupled from both a ’s and b ’s absolute properties; however, the latent variable z that is used to represent relational property in
VRL-PGM is only independent of a but not b .
Second, we note that the original problem is inherently undirected with no cause-effect relationship between a and b , whereas VRL-PGM is based a directed acyclic graph (DAG) that artiﬁcially introduces conditional dependency between a and b .
However, we argue that the application of VRL does not require the true conditional dependency between ( a , b ) be known in advance only that it is maintained consistently throughout learning and inference, i.e.,
VRL can be applied in the same way to learn about the relational property between ( b , a ) , where we swap a and b .
We further remark that the unique optimization challenges—deterministic-mapping and information-shortcut—introduced by VRL (discussed in detail in Section 3.3) can be viewed as the consequence of the compromises we made with using DAG for VRL.
More speciﬁcally, deterministic-mapping can viewd as caused by the causal relationship VRL-PGM introduced between a and b whereas information-shortcut can be viewed as caused by the causal relationship VRL-PGM introduced between z and b .
Next, we note that the independence between latent variable z and a is an inherent property of the proposed VRL-PGM and does not need to be enforced explicitly (Bishop, 2006).
We can also understand this theoretical property through the following intuitive explanation: We derived our main learning objective Eq. 4 (variational lower bound) from ﬁrst principle without any heuristics and/or assumptions not already provided by VRL-PGM.
We can further dissect the various terms in Eq.and observe that the learning is heavily guided by maximizing the likelihood term log p θ ( b ( i ) | a ( i ) , z ) since this is the only term that is constrained by the data a ( i ) and
b ( i ) while other terms in Eq.
4 can be viewed as regularization on the unobserved z .
Next, in maximizing log p θ ( b ( i ) | a ( i ) , z ) we are only learning to predict b ( i ) from a ( i ) and
z ( i ) ; and since a ( i ) is already been conditioned on, there is no incentive for z ( i ) to learn redundant (dependent) information from a ( i ) .
This effect is further “encouraged” when the learning objective Eq.
4 also includes additional regularization terms on z to learn a compact representation that “squeeze-out” any redundant information that does not help with predicting b ( i ) .
In the information ﬂow diagram shown in Fig.
2a one may argue that since a ( i ) also propagate information through the latent variable z
it may introduce dependency between a and z (note that in the posterior p θ ( z | a , b ) , z and a are no longer independent when b is observed); however, the information propagated from a ( i ) through z ( i ) is mainly used to maximize the likelihood term for predicting b ( i ) , and since there is already a direct propagation path from a ( i ) to b ( i ) (as part of the likelihood function p θ ( b ( i ) | a ( i ) , z ) ) there is, again through regularization on z , no incentive for z to carry redundant information from z (only decoupled relationship information derived from a and b ).
In short, while there is no explicit penalty on learning dependency between z and a in our learning objective Eq. 4, the independence is naturally encouraged through the interplay between different terms in the learning objective.
We want to emphasize that the goal of relational learning is to learn a relational property (represented by latent variable z ) that is independent from the absolute property of a ; relational property z will necessarily derive relationship information from both a and b (through posterior inference p θ ( z | a , b ) ) but should not depend on the absolute property of a (any information that describe a ( i ) alone).
Moreover, we note that when applying VRL-PGM to the relational learning problem setting, only a is observed and constrained by the data while the latent variable z is unobserved free to vary, so there is no contradiction to the independence assumption.
Lastly, this theoretical property is backed up by our various experimental results where we clearly show that VRL can indeed learn a decoupled (independent) relational property z from the absolute property of a .
Howerver, if one would like to take a more cautious approach and explicitly safeguard against introducing dependency between z and a , the proposed VRL-PGM provides a straightforward way for achieving this goal: we can append any non-positive function that measures the dependency between a and z with maximum attained when they are independent to Eq.
(4) without invalidating the lower bound.
For example, we can add the negative mutual information between z and a , − I
( z , a ) =
− D KL ( p θ ( z , a ) (cid:107) p θ ( z ) p θ ( a )) , to Eq.
(4) and obtain:
[Equation]
And since I ( z , a ) ≥
and I ( z , a ) =
if and only if z and a are independent, the addition of − I ( z , a ) in Eq.
(18) not only does not invalidate the original bound in Eq.
(4) but also does not sacriﬁce the quality of the lower bound ( z and a are independent in VRL-PGM).
We will leave the investigation of this idea to a future work.
D. R EMARKS ON RPDA
Here we would like to comment on the practical applicability of the proposed RPDA stratey.
More speciﬁcally, we would like to convey the idea that in many practical problem settings, the RPDA functions D can be designed without any knowledge of the underlying relational property.
For example, as we have explained in Sec. 3.3, in many computer vision applications, rotation invariant is a desirable property for the learned model; for example, in spectral imaging applications, oftentimes the orientation of the images are not preserved or not enforced (only that they are consistent between the same paired images) (Ronneberger et al., 2015).
In such problem settings, we can safely use image rotations function for constructing D .
Another example may be: for a discrete time-series data a [ t ] , b
[ t ] that represent the input and output of a linear time-invariant (LTI) system (commonly assumed in signal processing and control theory (Oppenheim & Schafer, 2009)), and we want to learn a relational property that characterize the system’s impulse response.
We have α b [ t − τ ] = α a [ t − τ ] , ∀ α ∈ R , τ ∈ Z , and we can construct D with d ( a [ t ] , b [ t ]; α, τ ) =
( α a [ t − τ ] , α b [ t − τ ]) , α, τ ∈ R
=
R × Z .
In all of the above examples, the construction of RPDA functions D reﬂects our prior knowledge and belief of the underlying system and not based on the underlying relational property of the data; therefore, in many instances, RPDA can be designed without any knowledge of the underlying relational property.
However, we would also note that RPDA is not central to the theory of the proposed method—VRL can be applied without RPDA—but rather a practical data augmentation strategy for addressing a unique optimization challenge (information-shortcut) of VRL learning.
In all of our experimental results, we ﬁnd that RPDA to be very effective and crucial in overcoming the information-shortcut problem (as illustrated in appendix C), but just like any data augmentations, this is problem dependent and we advocate to start without RPDA and only apply it when suspecting information-shortcut occurs.
D. R EMARKS ON EXPERIMENTAL RESULTS
Here we would like to give an overall summary and make general remarks on the experimental results we have presented in this work (included in Sec. 5 and appendix B).
First, we proposed four
MNIST experiments that represent a diverse set of relational learning problems: decoupled relational property (Sec. 5.1), coupled relational property (appendix B.1.1), multiple relational properties (appendix B.1.2), and continuous relational property (appendix B.1.3).
Although these experiments are introduced with well-controlled ground-truth relationships (so that we can easily validate and interpret the results), the application of the proposed VRL is completely unsupervised without any prior knowledge of the underlying relational property.
Furthermore, we deliberately design and successfully solve all four MNIST tasks using the exact same model and training setup despite each experiment represents a very different relational learning scenario (discrete vs. continuous, coupled vs. decoupled).
In addition, we presented two Yale face tasks with high-level perception relationships: change of emotions (appendix B.2.1), and change of illumination conditions (appendix B.2.2).
Again, the application of VRL to the two Yale face tasks is completely unsupervised and we deliberately design and successfully solve these two tasks using the exact same model and training setup.
When comparing our model and training setups between the MNIST and Yale face experiments, their differences can all be attributed to the need for accommodating different data types, e.g., increasing nework size for larger face images (vs. smaller MNIST images), modifying learning objective to adapt real-valued Yale dataset (vs. binay valued MNIST dataset), and not for accomodating different relationships.
Taking this into consideration, our results shows that we have solved both class of problems with the exact same principled method despite each class of problems represents a very different kind of relationships (the relationships in MNIST are geometric whereas the relationahips in Yale are high-level perception, e.g., sentiment, external environmental factors).
We believe our results further demonstrates that the proposed theoretical VRL framework is robust, and the practical application of VRL is stable and generalizable to different relationships.