Sample Space Truncation on Boltzmann Machines
Abstract
We present a lightweight variant of Boltzmann machines via sample space truncation, called a truncated Boltzmann machine (TBM), which has not been investigated before while can be naturally introduced from the log-linear model viewpoint.
TBMs can alleviate the massive computational cost of exact training of
Boltzmann machines that requires exponential time evaluation of expected values and the partition function of the model distribution.
To analyze the learnability of TBMs, we theoretically provide bias-variance decomposition of the log-linear model using dually ﬂat structure of statistical manifolds
Introduction
A Boltzmann machine (BM) (Ackley et al., 1985), an instance of the log-linear model (Agresti, 2012) and a representative of Markov Random Fields (MRFs) (Kindermann and Snell, 1980), models a joint distribution of multiple binary variables.
BMs are a fundamental of deep learning, and have been used for various tasks such as density estimation, missing value imputation, and sampling.
However, since the number of possible conﬁgurations is exponential to the number of variables, its exact training is usually intractable.
In particular, exact training of BMs requires exponential time evaluation of the expected values and the partition function of the model distribution.
To overcome this issue, approximation techniques have been proposed, for example, Gibbs sampling (Casella and
George, 1992; Geman and Geman, 1984) and contrastive divergence (Hinton, 2002; Tieleman, 2008) for evaluation of the expected values and annealed importance sampling (AIS) (Salakhutdinov and
Murray, 2008) and tracking (Desjardins et al., 2011) for computing the partition function, while exact training of BMs is still a challenging problem.
We tackle this problem by truncating the sample space.
We propose truncated Boltzmann machines (TBMs), which is a lightweight energy-based model on a sample space adaptively constructed from a given dataset.
Its training is efﬁcient; the time complexity is linear to the number of variables and the sample size and quadratic to the number of parameters, hence no approximation technique is needed.
Moreover, since TBMs do not have any hidden variables, their training is formulated as convex optimization.
Furthermore, we theoretically perform bias-variance decomposition of the
Kullback–Leibler (KL) divergence using information geometry (Amari, 2016).
Since TBMs belong to a class of the log-linear model on partially ordered sets (Sugiyama et al., 2016, 2017), the resulting statistical manifold has the dually ﬂat structure (Amari, 2001), which allows us to apply Pythagorean theorem to achieve bias-variance decomposition.
Log-Linear Model
First we introduce the log-linear model, which models discrete distributions.
We assume that the domain S of distributions is the set f 0 ; 1 ; 2 ; : : : ; n g without loss of generality, hence a discrete distribution P over S can be treated as a probability vector p =
( p 1 ; p 2 ; : : : ; p n ) 2 (0 ; 1) n as the degree of freedom is n .
Each probability vector p should satisfy p T 1 < 1 so that p T 1 + p
= 1 .
To simplify the notation, we always treat each vector as a column vector.
Uppercase letters P; Q; R; : : : denote distributions and lowercase letters p; q; r; : : : denote the corresponding probability mass functions.
We use the notation [ n ] = f 1 ; 2 ; : : : ; n g .
In the log-linear model (Agresti, 2012), a discrete distribution P over S represented as a probability vector p 2 (0 ; 1) n , is modeled via an n -dimensional parameter vector (cid:18) =
( (cid:18) 1 ; (cid:18) 2 ; : : : ; (cid:18) n ) 2 R n and a model matrix F 2 R n (cid:2) n , which encodes a relationship between elements and is usually given beforehand.
The assumption here is that F must be non-singular to construct a regular statistical model (Agresti, 2012).
The general form of the log-linear model is given as
[Equation]
where log is an element-wise operation,
[Equation]
Equation (1) is often used as a general form of the log-linear model (Coull and Agresti, 2003).
For example, when the model matrix F is the n (cid:2) n identity matrix, we obtain the standard discrete distribution, where there is no interaction between elements in S .
In such a case, (cid:18) i = log( p i =p 0 ) for each
i 2 [ n ] .
We also introduce an expectation parameter (cid:17) 2 (0 ; 1) n deﬁned as (cid:17) =
F T p .
In information geometry, it is well known that ( (cid:18) ; (cid:17) ) gives the dual coordinate system of the set of ( n + 1) -dimensional discrete distributions, or a dually ﬂat manifold (Amari, 2001, 2016), where (cid:18) and (cid:17) are orthogonal with each other.
The Boltzmann machine (BM) (Ackley et al., 1985), a well known energy-based model that treats combinatorial interaction between binary variables, is realized as a special case of the log-linear model.
For a d -dimensional binary vector b 2 f 0 ; 1 g d , the fully connected BM, modeled as a graph
G =
( V; E ) with V = f 1 ; 2 ; : : : ; d g and E = V (cid:2) V , is given as
[Equation]
with a parameter vector (cid:21) =
( (cid:21) ?
; (cid:21) f 1 g ; : : : ; (cid:21) f d g ; (cid:21) f 1 ; 2 g ; : : : ; (cid:21) f
d (cid:0) 1 ;d g ) .
Higher order BMs are deﬁned similarly (Sejnowski, 1986).
To clarify the relationship between Equation (2) and the general form in Equation (1), ﬁrst we prepare a one-to-one indexing mapping (cid:19) :
S = f 0 ; 1 ; : : : ; 2 d g ! f 0 ; 1 g d that gives a natural number to each d -dimensional binary vector b 2 f 0 ; 1 g d , and we assume that (cid:19) (0) =
Thus each probability p i in Equation (1) corresponds to p ( (cid:19)
( i )) in Equation (2).
In addition, we use an auxiliary function (cid:27) : f 0 ; 1 g d ! 2 f 1 ;:::;d g that returns indices of “1” of an input binary vector, e.g., (cid:27) ((0 ; 1 ; 1 ; 0))
=
f 2 ; 3 g .
Then one can see that the log-linear model in Equation (1) coincides with (2) if we let p
= p ( (0 ; : : : ; 0) ) , ( (cid:18) )
= (cid:0) (cid:21) ? , and f ij =
[Equation]
Therefore combinatorial structure of the binary log-linear model in Equation (2), which is realized as the inclusion relationship (cid:27) ( b ) (cid:18) (cid:27) ( b ′ ) for a pair of binary vectors b ; b ′ 2 f 0 ; 1 g d , is encoded as a the binary value of the model matrix F .
In addition, the binary log-linear model in Equation (2) implicitly perform regularization using the restriction to (cid:18) such that (cid:18) i =
if j (cid:27) ( (cid:19) ( i ))
j > 2 .
Truncated Boltzmann Machines
Here we formulate TBMs.
As we have shown in the previous section, BMs model joint distributions over d binary variables, where the sample space S is always ﬁxed to f 0 ; 1 g d , resulting in the exponentially larger sample space j S j = 2 d .
This is problematic as the computational cost of computing (cid:17) and ( (cid:18) ) is O (2 d ) , which is required to compute the gradient of the KL divergence in each iteration of gradient descent to train BMs.
An interesting observation of the log-linear model is that the ﬁxed sample space f 0 ; 1 g d is not necessarily, and it is possible to truncate unnecessary states from the sample space S .
Assume that a sample D (cid:18) [ n ] is given as data and a parameter domain B (cid:18) [ n ] is ﬁxed.
From the deﬁnition of log-linear model in Equation (1), the minimum requirement of S is obtained as
[Equation]
This is why the model matrix F becomes singular if we choose S ′ (cid:26) S as a sample space.
In contrast, Sugiyama et al.
(2017) showed that F is always regular if S is a partially ordered set and each entry is given as f ij = 1 j (cid:20) i ,
where i (cid:20) j () (cid:27) ( (cid:19) ( j )) (cid:18) (cid:27) ( (cid:19) ( i )) in our case.
Our proposal is to set S in Equation (4) as the sample space of the log-linear model; that is, truncate unnecessary space f 0 ; 1 g d n S , and learn the distribution on the space, where the model matrix is also given by Equation (3).
Surprisingly, this truncated sample space has not been investigated before in the literature of Boltzmann machines or Ising models.
Learning is achieved by convex optimization, where the objective is to minimize the KL divergence from the empirical distribution ^ P given by D to the model distribution R .
An algorithm for the ﬁrst-order optimization (gradient descent) is summarized in Algorithm 1 in Appendix.
Since we have ∇ D KL ( ^ P; R ) = (cid:17) R (cid:0) (cid:17) ^ P in each iteration, an update formula of a subvector (cid:18) RB = ( (cid:18) i ) i 2 B of (cid:18) R with respect to B of a current distribution R is given as (cid:18) RB (cid:18) RB (cid:0) " ( (cid:17) RB (cid:0) (cid:17) ^ PB ) , where " 2 R is a learning rate.
The time complexity of each update in the ﬁrst order optimization is
O ( j S jj B j ) =
O ( j D jj B j + j B j 2 ) in the worst case as we need to compute (cid:17) R = F exp( F TB (cid:18) RB (cid:0) R ) to obtain the next gradient from the updated (cid:18) R , where F B is the j D [ B j (cid:2)
j B j matrix composed of columns of F with respect to B .
In the following, we empirically examine the efﬁciency and the effectiveness of TBMs compared with two representative models: fully visible Boltzmann machines (BMs) and restricted Boltzmann machines (RBMs).
Environment and Implementation.
We used Amazon Linux AMI release 2018.03 and ran all experiments on 2.50 GHz Intel Xeon Platinum 8175M CPU and 384 GB of memory.
All methods,
TBMs, BMs, and RBMs, were implemented in C++ and compiled with gcc 4.8.5.
All experiments are conducted on R version 3.6.0.
We used contrastive divergence with a single step of alternating
Gibbs sampling (persistent CD-1) (Hinton, 2002; Tieleman, 2008) in learning of BMs and RBMs.
RBMs and BMs use mini-batch with the size 20 in training.
Throughout our experiments, the number of iterations were up to 10 4 for all methods to ensure the convergence.
Datasets.
We have collected two dataset, MNIST (LeCun et al., 2007), which is a collection of images of handwritten digits and popularly used as a benchmark dataset of image classiﬁcation, and ret-1 (Zhang et al., 2014), which is a neural spiking dataset and originally obtained by Lefebvre et al.
In the MNIST dataset, each 28 (cid:2) 28 image is represented as a 784 -dimensional vector.
Since each image v 2 R 784 is not originally a binary vector, we binarized each pixel such that b ( v i )
=if v i > 0 and b ( v i ) = {+
+} otherwise.
Protocol.
We train each model (TBM, RBM, and BM) on a dataset randomly sampled form each class of our datasets.
To measure the quality of learned distributions, we extract feature representations from each trained model and apply classiﬁcation to them.
More precisely, ﬁrst we collect 50 samples with the sample size N = 100 from each class of a dataset, hence in total there are 500 samples in MNIST and 350 samples in ret-1.
Then we train each model on each sample.
For the TBM and the BM, let ( (cid:21)
f 1 g ; : : : ; (cid:21) f
d g ) be bias parameters and ( (cid:21) f 1 ; 1 g ; : : : ; (cid:21) f
d (cid:0) 1 ;d g ) be weight parameters in Equation (2).
To obtain a feature vector representation of a sample from each trained model, we construct f = ( f 1 ; : : : ; f d ) such that each f i = j (cid:21)
f i g j + ∑ j
[ d ] j (cid:21) f i;j g j since the absolute value of each parameter represents the intensity of the corresponding variable (or variable interaction) in the model.
Similarly, for RBMs with weight parameters ( (cid:21) f 1 ; 1 g ; : : : ; (cid:21) f d;h g ) with h hidden variables, we obtain f by f i =
j (cid:21)
f
i g j + ∑ j
[ h ] j (cid:21) f i;j g j .
We varied the number of weight parameters and examined the scalability and performance sensitivity.
To do this, weight parameters are randomly chosen and (cid:21) f
i;j g
=
if it is not selected.
After obtaining such feature representations for all samples, we perform classiﬁcation for them.
We use k NN ( k = 10 ) and obtained the accuracy via leave-one-out cross validation.
Therefore, high classiﬁcation accuracy means that the quality of the learned representation of a distribution is high.
Results & Discussion Results of running time and classiﬁcation accuracy are plotted in Figure 1(a) and (b).
We present average running time of learning for each sample in Figure 1(a).
It is clear that TBMs are much faster than the BMs and RBMs.
In particular, if the number of parameters is less than 1,000, it is two and four orders of magnitude faster than RBMs and BMs, respectively.
The efﬁciency of TBMs becomes smaller if we use massive parameters ( (cid:24) 10 5 ), which is predicted from the complexity analysis that TBMs requires the quadratic complexity and BMs and RBMs require linear complexity, while it is still an order of magnitude faster than BMs.
This result means that our sample space truncation enables us to perform efﬁcient distribution learning without any approximation techniques such as contrastive divergence and mini-batch selection.
Number
[Figure]
A cc u r a c y
[Figure]
R unn i n g t
i m e ( s )  104 0.
Number of weight parameters Number
[Figure]
Number of weight parameters Number of weight parameters 0.
MNIST retMNIST ret-
[Figure]
Figure 1(b) shows that the classiﬁcation accuracy of TBMs is almost 1 if we use the reasonable amount of parameters (higher than 100) in both datasets.
Hence the loss of information due to sample space truncation can remain at low level, which shows a good trade-off between the effectiveness and the efﬁciency of TBMs.
In addition, we illustrate distribution of feature representations in
Figure 3 in Appendix by projecting them onto 2D space by t-SNE (van der Maaten and Hinton, 2008).
Classes are clearly separated, which also indicates the effectiveness of TBMs despite the fact that it achieves massive speedup.
3.1 Bias-Variance Decomposition
Using the geometric structure of the log-linear model, we analyze its learnability by providing biasvariance decomposition of the KL divergence, which is a fundamental analysis of learning models.
In a dually ﬂat manifold, learning of a distribution on S is realized as e -projection, which coincides with maximum likelihood estimation (MLE) or KL divergence minimization (Amari, 2016, Chapter 2.8.3).
In the following, for each distribution P , let (cid:18) P and (cid:17) P be its (cid:18) - and (cid:17) -coordinates, respectively, to clarify that they represent the same distribution P .
Let S model be an e -ﬂat model submanifold given as
[Equation]
with a predetermined parameter domain B 2 [ n ] .
Since S model is determined by the linear constraint on the (cid:18) -coordinate, it is a convex set with respect to (cid:18) -coordinate.
The parameter domain B determines the representation power of the model; overﬁtting occurs if B is too large and underﬁtting occurs if B is too small.
More precisely, for a series of parameter domains ∅ = B 0 (cid:18) B 1 (cid:18) B 2 (cid:18) (cid:1) (cid:1) (cid:1) (cid:18) B h
=
[ n ] , we obtain the hierarchy of manifolds: f P 0 g = S 0 (cid:18) S 1 (cid:18) S 2 (cid:18) (cid:1) (cid:1) (cid:1) (cid:18) S h
= S , where each S j = f P 2 S j (cid:18) Pi =
for all i 2 S n B j g and P 0 is the uniform distribution such that (cid:18) =
For an empirical distribution ^ P , we obtain an m -ﬂat data submanifold S data given as
[Equation]
Similarly to S model , S data is a convex set with respect to (cid:17) -coordinate.
The learning procedure is formulated as a projection of some initial distribution R 2 S model onto the data submanifold S data , which means that it is convex optimization and there exists a unique global solution; that is, the set S model \ S data is always a singleton, thanks to the dually ﬂat structure (Amari, 2009).
Here we perform bias-variance decomposition of the KL-divergence.
Our idea is to decompose the expectation of the KL divergence E [ D KL ( P (cid:3) , ^ P B )] from the true (unknown) distribution P (cid:3) to the
MLE (maximum likelihood estimation) ^ P B of an empirical distribution ^ P with a ﬁxed parameter domain B , and decompose it using the information geometric property.
More precisely, let S (cid:3) data be the m -ﬂat submanifold that is obtained by replacing ^ P with P (cid:3) in S data .
Since P (cid:3) 2 S (cid:3) data and ^ P B 2 S model , we can apply orthogonal decomposition (Amari, 2001) to the KL divergence
D KL ( P (cid:3) ; ^ P B ) , also known as (generalized) Pythagorean theorem, and obtain the following:
[Equation]
The ﬁrst term corresponds to the bias of the model and measures its representation power, which becomes smaller when the model manifold gets larger.
For the variance term, we can lower bound it as j B j = 2 N using (Barron et al., 1998), which suggests that the variance is independent of the model structure F even if we have combinatorial structure of variables like BMs or TBMs in it.
A Information Geometry of Log-Linear Model
We study information geometric structure of the log-linear model.
Although more general results are already known in analysis of the exponential family (Amari, 2016) or for graphical models (Wainwright and Jordan, 2008) we explicitly describe it for the better understanding of the geometric structure of the log-linear model
As we will show in this section the set of distributions represented by a log-linear model always forms a dually ﬂat manifold , which is a statistical manifold with a pair of coordinate systems mainly studied in the area of information geometry (Amari, 2016).
A dually ﬂat manifold is generated by a convex function which is the partition function ( (cid:18) ) in our case.
From the condition p T 1 + p 0 = p T 1 + exp( (cid:0) ( (cid:18) ))
= 1 , ( (cid:18)
[Equation]
=
[Equation]
This is the well-known LogSumExp function and therefore is convex.
Let us apply Legendre transformation to ( (cid:18) ) , which is deﬁned as φ ( (cid:17) )
=
[Equation] ( (cid:18) )) , We have its closed form solution as follows:
Proposition 1 (Legendre dual) .
The Legendre dual φ ( (cid:17) ) of ( (cid:18) )
= (cid:0) log p 0 is given as
[Equation] where p ′ =
( p 0 ; p ) =
( p 0 ; p 1 ; : : : ; p n ) .
Proof.
From the Legendre transformation given as
[Equation]
we have
[Equation]
Since it follows that max q ′ log q ′ T p ′
= log p ′ T p ′ , Equation (8) holds.
Note that we have φ ( (cid:17) )
=
log( F (cid:0) 1 (cid:17) ′ )
F (cid:0) 1 (cid:17) ′ with (cid:17) ′
= (1 ; (cid:17) ) , hence we can see that φ ( (cid:17) ) is a function of (cid:17) .
Now we consider the geometric structure of the statistical manifold S =
f P j p 2 (0 ; 1) n ; p T 1 < 1 g , which is the set of ( n + 1) -dimensional discrete distributions.
In information geometry, it is known that a pair of coordinate systems is obtained as the gradient of ( (cid:18) ) and φ ( (cid:17) )) , respectively (Rockafellar, 1970; Banerjee et al., 2005).
As we show in the following proposition, the coordinates interestingly coincide with (cid:18) and (cid:17) .
Proposition 2 (dual coordinate) .
For the partition function ( (cid:18) ) of the log-linear model in Equation (1) and its Legendre dual φ ( (cid:17) ) given in Equation (8) , the dual coordinate system of the statistical manifold S is obtained as
[Equation]
Proof.
First we prove that the parameter vector (cid:18) coincides with ∇ φ ( (cid:17) ) .
Since we have
[Equation]
where f (cid:0) 1 ij is the ( i; j ) entry of F (cid:0) 1 , it follows that @φ ( (cid:17) )
@(cid:17) i = @
@(cid:17) i @
∑ j 2 [ n ]
[Equation]
= ∑ j 2 [ n ] f (cid:0) 1 ij log p j + ∑ j 2 [ n ] f (cid:0) 1 ij ( (cid:18) ) =
(cid:18) i :
The last equality comes from (cid:18) =
F (cid:0) 1 log p + F (cid:0) 1 .
Next, we prove (cid:17) =
F p .
From Equation (7), @ ( (cid:18) )
@(cid:18) i =
[Equation]
exp( ( (cid:18)
[Equation]
[Equation]
(cid:17) i :
Hence Equation (10) follows.
The resulting manifold S with the pair of coordinate systems ( (cid:18) ; (cid:17) ) is said to be dually ﬂat .
In any dually ﬂat manifold, the Fisher information matrix coincides with Riemannian metric, which is deﬁned as ∇∇ ( (cid:18) ) and ∇∇ φ ( (cid:17) ) for (cid:18) - and (cid:17) -coordinates, respectively.
We ﬁrstly obtain the closed form solution of the Riemannian metric in the general case of the log-linear model.
Theorem 1 (Riemannian metric) .
Riemannian metric for (cid:18) - and (cid:17) -coordinates g ij ( (cid:18) ) and g ij ( (cid:17) ) of the log-linear model are obtained as
[Equation]
Proof.
In the log-linear model, the Riemannian metric for the (cid:18) -coordinate is obtained as [Equation]
@(cid:18) i @(cid:18) j ( (cid:18) ) =
@(cid:17) i @(cid:18) j
= @ @(cid:18)
j ∑ k 2 [ n ] f ki p k = @
@(cid:18)
j ∑ k 2 [ n ] f ki exp @∑ l 2 [ n ] f kl (cid:18) l (cid:0) ( (cid:18) )
A =
∑ k 2 [ n ] f ki exp @∑ l 2 [ n ] f kl (cid:18) l (cid:0) ( (cid:18) )
A ( f kj (cid:0) (cid:17) j ) =
∑ k 2 [ n ] f ki f kj p k (cid:0) (cid:17) j ∑ k 2 [ n ] f ki p k
=
∑ k 2 [ n ] f ki f kj p k (cid:0) (cid:17) i (cid:17)
j :
(cid:31)(cid:30)(cid:29) (cid:31)(cid:30)(cid:28) (cid:31)(cid:30)(cid:27) (cid:31)(cid:30)(cid:26) (cid:31)(cid:30)(cid:25) (cid:31)(cid:30)
[Figure]
(cid:27) (cid:31)(cid:30)
[Figure]
(cid:25) (cid:31)(cid:30)
[Figure]
(cid:24) (cid:31)(cid:30)
[Figure]
(cid:23)(cid:22)
[Figure]
The Riemannian metric for (cid:17) -coordinate is obtained as
[Equation]
; where we have @
[Equation]
j ∑
[Equation]
∑
[Equation]
[Equation]
Hence it follows that
[Equation]
yielding Equation (12).
Moreover, two coordinates (cid:18) and (cid:17) are orthogonal with each other, that is,
[Equation]
where (cid:14) ij is the Kronecker delta and (cid:14) ij = 1 if i = j and (cid:14) ij = 0 otherwise.
Equivalently, we have
[Equation]
B Empirical Validation of Variance
We empirically demonstrate the tightness of the lower bound j B j = 2 N of the variance.
To obtain the variance var( P (cid:3) B )
=
E [ D KL ( P (cid:3) B ; ^ P B )] , ﬁrst we ﬁx a true distribution P (cid:3) generated from the uniform distribution with its sample space S with j S j = 1 ; 000 and get P (cid:3) B estimated by a TBM with (cid:27) = 0 :and k = 2 , which gives a reasonable amount of parameters B .
Then the lower bound is obtained as j B j = 2 N .
In each trial, we repeat 100 times generating a sample D with the size N from P (cid:3) and generated ^ P B with ﬁxing S and B to directly estimate the variance ( (cid:6) its standard deviation).
In
Figure 2( a ) the sample size N is varied from 100 to 1,000,000 with ﬁxing the number of variables n = j V j = 50 while in Figure 2( b ) n is varied from 10 to 1,000 with ﬁxing N = j D j = 100,000.
These plots clearly show that our lower bound is tight enough across all settings.
The lower bound exceeds the actual variance in some cases, which is due to the approximation error of the Taylor series expansion or ﬂuctuation of random sampling.
[Figure]