Paper:
Restricted Boltzmann Machine (RBM) is a bipartite graphical model that is used as the building block in energy-based deep generative models. Due to its numerical stability and quantifiability of its likelihood, RBM is commonly used with Bernoulli units. Here, we consider an alternative member of the exponential family RBM with leaky rectified linear units – called leaky RBM. We first study the joint and marginal distributions of the leaky RBM under different leakiness, which leads to interesting interpretation of the leaky RBM model as truncated Gaussian distribution. We then propose a simple yet efficient method for sampling from this model, where the basic idea is to anneal the leakiness rather than the energy; – i.e., start from a fully Gaussian/Linear unit and gradually decrease the leakiness over iterations. This serves as an alternative to the annealing of the temperature parameter and enables numerical estimation of the likelihood that are more efficient and far more accurate than the commonly used annealed importance sampling (AIS). We further demonstrate that the proposed sampling algorithm enjoys relatively faster mixing than contrastive divergence algorithm, which improves the training procedure without any additional computational cost.
1 INTRODUCTION
In this paper, we are interested in deep generative models. One may naively classify these models into a family of directed deep generative models trainable by back-propagation (e.g., Kingma & Welling, 2013; Goodfellow et al., 2014), and deep energy-based models, such as deep belief network (Hinton et al., 2006) and deep Boltzmann machine (Salakhutdinov & Hinton, 2009). The building block of deep energy-based models is a bipartite graphical model called restricted Boltzmann machine (RBM). The RBM model consists of two layers, visible and hidden. The resulting graphical model which can account for higher-order interactions of the visible units (visible layer) using the hidden units (hidden layer). It also makes the inference easier that there are no interactions between the variables in each layer.
The conventional RBM uses Bernoulli units for both the hidden and visible units (Smolensky, 1986). One extension is using Gaussian visible units to model general natural images (Freund & Haussler, 1994). For hidden units, we can also generalize Bernoulli units to the exponential family (Welling et al., 2004; Ravanbakhsh et al., 2016).
Nair & Hinton (2010) propose a variation using Rectified Linear Unit (ReLU) for the hidden layer with a heuristic sampling procedure, which has promising performance in terms of reconstruction error and classification accuracy. Unfortunately, due to its lack of strict monotonicity, ReLU RBM does not fit within the framework of exponential family RBMs (Ravanbakhsh et al., 2016). Instead we study leaky-ReLU RBM (leaky RBM) in this work and address two important issues i) a better training (sampling) algorithm for ReLU RBM and; ii) a better quantification of leaky RBM –i.e., evaluation of its performance in terms of likelihood.
We study some of the fundamental properties of leaky RBM, including its joint and marginal distributions (Section 2). By analyzing these distributions, we show that the leaky RBM is a union of
truncated Gaussian distributions. In this paper, we show that training leaky RBM involves underlying positive definite constraints. Because of this, the training can diverge if these constrains are not satisfied. This is an issue that was previously ignored in ReLU RBM, as it was mainly used for pre-training rather than generative modeling.
Our contribution in this paper is three-fold: I) we systematically identify and address model constraints in leaky RBM (Section 3); II) for the training of leaky RBM, we propose a meta algorithm for sampling, which anneals leakiness during the Gibbs sampling procedure (Section 3) and empirically show that it can boost contrastive divergence with faster mixing (Section 5); III) We demonstrate the power of the proposed sampling algorithm on estimating the partition function. In particular, comparison on several benchmark datasets shows that the proposed method outperforms the conventional AIS (Salakhutdinov & Murray, 2008) in terms of efficiency and accuracy (Section 4). Moreover, we provide an incentive for using leaky RBM by showing that the leaky ReLU hidden units perform better than the Bernoulli units in terms of the model log-likelihood (Section 4).
2 RESTRICTED BOLTZMANN MACHINE AND RELU
The Boltzmann distribution is defined as p(x) = e−E(x)/Z where Z = ∑ x e −E(x) is the partition function. Restricted Boltzmann Machine (RBM) is a Boltzmann distribution with a bipartite structure It is also the building block for many deep models (e.g., Hinton et al., 2006; Salakhutdinov & Hinton, 2009; Lee et al., 2009), which are widely used in numerous applications (Bengio, 2009). The conventional Bernoulli RBM, models the joint probability p(v, h) for the visible units v ∈ [0, 1]I and the hidden units h ∈ [0, 1]J as p(v, h) ∝ exp(−E(v, h)), where E(v, h) = a>v − v>Wh + b>h. The parameters are a ∈ RI , b ∈ RJ and W ∈ RI×J . We can derive the conditional probabilities as
p(vi = 1|h) = σ  J∑ j=1 Wijhj + ai  and p(hj = 1|v) = σ( I∑ i=1 Wijvi + bj ) , (1)
where σ(x) = (1 + e−x)−1 is the sigmoid function.
One extension of Bernoulli RBM is replacing the binary visible units by linear units v ∈ RI with independent Gaussian noise. The energy function in this case is given by
E(v, h) = I∑ i=1 (vi − ai)2 2σ2i − I∑ i=1 J∑ j=1 vi σi Wijhj + b >h.
To simplify the notation, we assume a normalized data so that ai and σi is no longer required. The energy function is accordingly simplified to E(v, h) = ‖v‖ 2
2 − v >Wh + b>h (Note that the
elimination does not influence the discussion and one can easily extend all the results in this paper to the model that includes ai and σi.).
The conditional distributions are as follows:
p(vi|h) = N  J∑ j=1 Wijhj , 1  and p(hj = 1|v) = σ( I∑ i=1 Wijvi + bj ) , (2)
where N (µ, V ) is a Gaussian distribution with mean µ and variance V . To simplify the notation, in the following we define ηj = ∑I i=1Wijvi + bj – that is ηj is the input to the j
th hidden layer neuron and similarly define νi = ∑J j=1Wijhj + ai. Using this notation the conditionals in the (2) are p(vi|νi) = N (νi, 1) and p(hj = 1|ηj) = σ(ηj).
2.1 RELU RBM WITH CONTINUOUS VISIBLE UNITS
From (1) and (2), we can see that the mean of the p(hj |v) is the nonlinearity of the hidden unit at ηj = ∑I i=1Wijvi + bj – e.g., mean of the Bernoulli unit is the sigmoid function. From this perspective, we can extend the sigmoid function to other functions and thus allow RBM to have more expressive power (Ravanbakhsh et al., 2016). In particular, it would be interesting to use rectified linear unit (ReLU) nonlinearity, f(ηj) = max(0, ηj), for generative modeling.
Nair & Hinton (2010) use an RBM with visible Gaussian unit and ReLU hidden activation functions for pretraining. They suggest sampling from max(0, ηj+N (0, σ(ηj)) for conditional sampling from the hidden units (compare to (2)). However, this sampling heuristic does not suggest the parametric form of the joint ReLU-Gaussian distribution. This also means we cannot evaluate it using methods such as Annealed Importance Sampling that require access to this parametric form. In fact, only strictly monotonic activation functions can derive feasible joint and conditional distributions in the exponential familly RBM and ReLU is not strictly monotonic Ravanbakhsh et al. (2016). Similar activation functions that are monotonic are Softplus, f(ηj) = log(1 + eηj ) and leaky ReLU (Maas et al., 2013), defined as f(ηj) = max(cηj , ηj), where c ∈ (0, 1) is the leakiness parameter. In contrast to the ReLU RBM the joint parametric form of these two distributions are available. However, the energy (logarithm of the joint probability) in the case of Softplus activation function contains a polylogarithmic term that requires evaluation of an infinite series; see Table 1 in Ravanbakhsh et al. (2016). For this reason, here we focus on Leaky-ReLU activation function.
By Ravanbakhsh et al. (2016), the conditional probability of the activation, assuming the nonlinearity f(ηj), is generally defined as p(hj |v) = exp (−Df (ηj‖hj) + g(hj)), where Df (ηj‖hj) is the Bregman Divergence associated with f, and g(hj) is the base (or carrier) measure in the exponential family which ensures the distribution is well-defined. The Bergman divergence, for strictly monotonic function f , is Df (ηj‖hj) = −ηjhj + F (ηj) + F ∗(hj), where F with ddηj F (ηj) = f(ηj) is the anti-derivative (integral) of f and F ∗ is the anti-derivative of f−1 (i.e., f−1(f(η)) = η); Note that due to the strict monotonicity of f , f−1 is well-defined, and F and F ∗ are commonly referred to as conjugate duals.
Considering the leaky ReLU activation function f(η) = max(cη, η), using this formalism, the conditional distributions of hidden units in the leaky RBM simplifies to (see Appendix A.1 for details)
p(hj |v) = { N (ηj , 1), if ηj > 0 N (cηj , c), if ηj ≤ 0.
(3)
Since the visible units uses the identity function, the corresponding conditional distribution is a Gaussian1
p(vi|h) = N  J∑ j=1 Wijhj , 1  , (4) Having these two conditional distributions is enough for training a leaky RBM model using contrastive divergence (Hinton, 2002) or some other alternatives (e.g., Tieleman, 2008; Tieleman & Hinton, 2009).
3 TRAINING AND SAMPLING FROM LEAKY RBM
Given the conditional distributions p(v|h) and p(h|v), the joint distribution p(v, h) from the general treatment for MRF model is (Yang et al., 2012; Ravanbakhsh et al., 2016)
p(v, h) ∝ exp v>Wh− I∑ i=1 (F̃ ∗(vi) + g(vi))− J∑ j=1 (F ∗(hj) + g(hj))  , (5) where F̃ ∗(vi) and F ∗(hj) are anti-derivatives of the inverses of the activation functions f̃(vi) and f(hj) for visible units vi and hidden units hj , respectively (see Section 2.1). Assuming f(ηj) = max(cηj , c) and f̃(νi) = νi in leaky-ReLU RBM, the joint distribution above becomes (see Appendix A.2 for details)
p(v, h) ∝ exp v>Wh− ‖v‖2 2 − ∑ ηj>0 ( h2j 2 + log √ 2π ) − ∑ ηj≤0 ( h2j 2c + log √ 2cπ ) + b>h  , 1which can also be written as p(vi|h) = exp ( −Df̃ (νi‖vi) + g(vi) ) , where νi = ∑ j=1Wijhj and
f̃(νi) = νi and Df̃ (νi‖vi) = (νi − vi) 2 and g(vi) = − log
√ 2π.
and the corresponding visible marginal distribution is
p(v) ∝ exp −1 2 v> I −∑ ηj>0 WjW > j − c ∑ ηj≤0 WjW > j  v + ∑ ηj>0 bjW > j v + c ∑ ηj≤0 bjW > j v  . (6)
where Wj is the j-th column of W .
3.1 LEAKY RBM AS UNION OF TRUNCATED GAUSSIAN DISTRIBUTIONS
From (6) we see that the marginal probability is determined by the affine constraints ηj > 0 or ηj ≤ 0 for all hidden units j. By combinatorics, these constraints divide RI (the visible domain) into at most M = ∑I i=1 ( J i ) convex regions R1, · · ·RM . An example with I = 2 and J = 3 is shown in Figure 1. If I > J , then we have at most 2J regions.
We discuss the two types of these regions. For bounded regions, such as R1 in Figure 1, the integration of (6) is also bounded, which results in a valid distribution. Before we discuss the unbounded cases, we define Ω = I − ∑J j=1 αjWjW > j , where αj = 1ηj>0 + c1ηj≤0. For the unbounded region, if Ω ∈ RI×I is a positive definite (PD) matrix, then the probability density is proportional to a multivariate Gaussian distribution with mean µ = Ω−1 (∑J j=1 αjbjWj ) and precision matrix Ω (covariance matrix Ω−1) but over an affine-constrained region. Therefore, the distribution of each unbounded region can be treated as a truncated Gaussian distribution. The marginal distrubution can be treated as a union of truncated Gaussain distribution. Note that leaky RBM is different from Su et al. (2017), which use single truncated Gaussian distribution to model joint (conditional) distributions and require approximated and more complicated sampling algorithms for truncated Gaussian distribution, while leaky RBM only requires to sample from Gaussian distributions.
On the other hand, if Ω is not PD, and the region Ri contains the eigenvectors with negative eigenvalues of Ω, the integration of (6) over Ri is divergent (infinite), which can not result in a valid probability distribution. In practice, with this type of parameter, when we do Gibbs sampling on the conditional distributions, the sampling will diverge. However, it is unfeasible to check exponentially many regions for each gradient update. Theorem 1. If I −WW> is positive definite, then I − ∑ j αjWjW > j is also positive definite, for all αj ∈ [0, 1].
The proof is shown in Appendix 1. From Theorem 1 we can see that if the constraint I −WW> is PD, then one can guarantee that the distribution of every region is a valid truncated Gaussian distribution. Therefore, we introduce the following projection step for each W after the gradient update.
argmin W̃
‖W − W̃‖2F
s.t. I − W̃W̃> 0 (7)
Theorem 2. The above projection step (7) can be done by shrinking the singular values to be less than 1.
The proof is shown in Appendix C. The training algorithm of the leaky RBM is shown in Algorithm 1. By using the projection step (7), we could treat the leaky RBM as the union of truncated Gaussian distributions, which uses weight vectors to divide the space of visible units into several regions and use a truncated Gaussian distribution to model each region. Note that the leaky RBM model is different from Su et al. (2016), which uses a truncated Gaussian distribution to model the conditional distribution p(h|v) instead of the marginal distribution. The empirical study about the divergent values and the necessity of the projection step is shown in Appendix D. Without the projection step, when we run Gibbs sampling for several iterations from the model, the sampled values will diverge because the model does not have a valid marginal distribution p(v). It also implies that we cannot train leaky RBM with larger CD steps without projection, which would result in divergent gradients. The detailed discussion is shown in Appendix D.
Algorithm 1 Training Leaky RBM for t = 1, . . . , T do
Estimate gradient gθ by CD or other algorithms with (13) and (4), where θ = {W,a, b}. θ(t) ← θ(t−1) + ηgθ. Project W (t) by (7).
end for
3.2 SAMPLING FROM LEAKY-RELU RBM
Gibbs sampling is the core procedure for RBM, including training, inference, and estimating the partition function (Fischer & Igel, 2012; Tieleman, 2008; Salakhutdinov & Murray, 2008). For every task, we start from randomly initializing v by an arbitrary distribution q, and iteratively sample from the conditional distributions. Gibbs sampling guarantees the procedure result in the stationary distribution in the long run for any initialized distribution q. However, if q is close to the target distribution p, it can significantly shorten the number of iterations to achieve the stationary distribution. If we set the leakiness c to be 1, then (6) becomes a simple multivariate Gaussian distribution N ( (I −WW>)−1Wb, (I −WW>)−1 ) , which can be easily sampled without Gibbs sampling. Also, the projection step (7) guarantees it is a valid Gaussian distribution. Then we decrease the leakiness with a small , and use samples from the multivariate Gaussian distribution when c = 1 as the initialization to do Gibbs sampling. Note that the distribution of each region is a truncated Gaussian distribution. When we only decrease the leakiness with a small amount, the resulted distribution is a “similar” truncated Gaussian distribution with more concentrated density. From this observation, we could expect the original multivariate Gaussian distribution serves as a good initialization. The one-dimensional example is shown in Figure 2. We then repeat this procedure until we reach the target leakiness. The algorithm can be seen as annealing the leakiness during the Gibbs sampling procedure. The meta algorithm is shown in Algorithm 2. Next, we show the proposed sampling algorithm can help both the partition function estimation and the training of leaky RBM.
Algorithm 2 Meta Algorithm for Sampling from Leaky RBM. Sample v from N ( (I −WW>)−1Wb, (I −WW>)−1 ) = (1− c)/T and c′ = 1 for t = 1, . . . , T do
Decrease c′ = c′ − and perform Gibbs sampling by using (13) and (4) with leakiness c′ end for
4 PARTITION FUNCTION ESTIMATION
It is known that estimating the partition function of RBM is intractable (Salakhutdinov & Murray, 2008). Existing approaches, including Salakhutdinov & Murray (2008); Grosse et al. (2013); Liu et al. (2015); Carlson et al. (2016) focus on using sampling to approximate the partition function of the conventional Bernoulli RBM instead of the RBM with Gaussian visible units and non-Bernoulli hidden units. In this paper, we focus on extending the classic annealed importance sampling (AIS) algorithm (Salakhutdinov & Murray, 2008) to leaky RBM.
Assuming that we want to estimate the partition function Z of p(v) with p(v) = p∗(v)/Z and p∗(v) ∝ ∑ h exp(−E(v, h)), Salakhutdinov & Murray (2008) start from a initial distribution
p0(v) ∝ ∑ h exp(−E0(v, h)), where computing the partition Z0 of p0(v) is tractable and we can draw samples from p0(v). They then use the “geometric path” to anneal the intermediate distribution as pk(v) ∝ p∗k(v) = ∑ h exp (−βkE0(v, h)− (1− βk)E(v, h)), where they grid βk from 1 to 0. If we let β0 = 1, we can draw samples vk from pk(v) by using samples vk−1 from pk−1(v) for k ≥ 1 via Gibbs sampling. The partition function is then estimated via Z = Z0M ∑M i=1 ω (i), where
ω(i) = p∗1(v (i) 0 )
p∗0(v (i) 0 )
p∗2(v (i) 1 ) p∗1(v (i) 1 ) · · · p∗K−1(v (i) K−2) p∗K−2(v (i) K−2) p∗K(v (i) K−1) p∗K−1(v (i) K−1) , and βK = 0.
Salakhutdinov & Murray (2008) use the initial distribution with independent visible units and without hidden units. We consider application of AIS to the leaky-ReLU case with E0(v, h) = ‖v‖2 2 , which results in a multivariate Gaussian distribution p0(v). Compared with the meta algorithm shown in Algorithm 2 which anneals between leakiness, AIS anneals between energy functions.
4.1 STUDY ON TOY EXAMPLES
As we discussed in Section 3.1, leaky RBM with J hidden units is a union of 2J truncated Gaussian distributions. Here we perform a study on the leaky RBM with a small number hidden units. Since in this example the number of hidden units is small, we can integrate out all possible configurations of h. However, integrating a truncated Gaussian distribution with general affine constraints does not have analytical solutions, and several approximations have been developed (e.g., Pakman & Paninski, 2014). To compare our results with the exact partition function, we consider a special case that has the following form:
p(v) ∝ exp −1 2 v> I −∑ ηj>0 WjW > j − c ∑ ηj≤0 WjW > j  v  . (8)
Compared to (6), it is equivalent to the setting where b = 0. Geometrically, everyWj passes through the origin. We further put the additional constraint Wi ⊥ Wj ,∀i 6= j. Therefore. we divide the whole space into 2J equally-sized regions. A three dimensional example is shown in Figure 3. Then the partition function of this special case has the analytical form
Z = 1
2J ∑ αj∈{1,c},∀j (2π)− I 2 ∣∣∣∣∣∣∣ I − J∑ j=1 αjWjW > j − 12 ∣∣∣∣∣∣∣ .
We randomly initialize W and use SVD to make columns orthogonal. Also, we scale ‖Wj‖ to satisfy I −WW> 0. The leakiness parameter is set to be 0.01. For Salakhutdinov & Murray (2008) (AIS-Energy), we use 105 particles with 105 intermediate distributions. For the proposed method (AIS-Leaky), we use only 104 particles with 103 intermediate distributions. In this small problem we study the cases when the model has 5, 10, 20 and 30 hidden units and 3072 visible units. The true log partition function logZ is shown in Table 1 and the difference between logZ and the estimates given by the two algorithms are shown in Table 2.
From Table 1, we observe that AIS-Leaky has significantly better and more stable estimations than AIS-Energy especially and this gap increases as we increase the number of hidden units. AIS-Leaky achieves this with orders magnitude reduced computation –e.g., here it uses ∼.1% of resources used by conventional AIS. For example, when we increase J from 5 to 30, the bias (difference) of AIS-Leaky only increases from 0.02 to 0.13; however, the bias of AIS-Energy increases from 1.76 to 9.6. We further study the implicit connection between the proposed AIS-Leaky and AIS-Energy in Appendix E, which shows AIS-Leaky is a special case of AIS-Energy under certain conditions.
4.2 COMPARISON BETWEEN LEAKY-RELU RBM AND BERNOULLI-GAUSSIAN RBM
It is known that the reconstruction error is not a proper approximation of the likelihood (Hinton, 2012). One commonly adopted way to compare generative models is to sample from the model, and visualize the images to check the quality. However, Theis et al. (2016) show the better visualization does not imply better likelihood. Also, the single layer model cannot adequately model the complicated natural images (the result for Bernoulli-Gaussian RBM has been shown in Ranzato & Hinton (2010)), which makes the visualization comparison difficult (Appendix F has few visualization results).
Fortunately, our accurate estimate of the partition function for leaky RBM can produce a reliable quantitative estimate of the representation power of leaky RBM. We compare the BernoulliGaussian RBM2, which has Bernoulli hidden units and Gaussian visible units. We trained both models with CD-203 and momentum. For both model, we all used 500 hidden units. We initialized W by sampling from Unif(0, 0.01), a = 0, b = 0 and σ = 1. The momentum parameter was 0.9 and the batch size was set to 100. We tuned the learning rate between 10−1 and 10−6. We studied two benchmark data sets, including CIFAR10 and SVHN. The data was normalized to have zero mean and standard deviation of 1 for each pixel. The results of the log-likelihood are reported in Table 3.
From Table 3, leaky RBM outperforms Bernoulli-Gaussian RBM significantly. The unsatisfactory performance of Bernoulli-Gaussian RBM may be in part due to the optimization procedure. If we tune the decay schedule of the learning-rate for each dataset in an ad-hoc way, we observe the performance of Bernoulli-Gaussian RBM can be improved by ∼ 300 nats for both datasets. Also, increasing CD-steps brings slight improvement. The other possibility is the bad mixing during the CD iterations. The advanced algorithms Tieleman (2008); Tieleman & Hinton (2009) may help. Although Nair & Hinton (2010) demonstrate the power of ReLU in terms of reconstruction error and classification accuracy, it does not imply its superior generative capability. Our study confirms leaky RBM could have much better generative performance compared to Bernoulli-Gaussian RBM.
5 BETTER MIXING BY ANNEALING LEAKINESS
In this section, we show the idea of annealing between leakiness benefit the mixing in Gibbs sampling in other settings. A common procedure for comparison of sampling methods for RBM is through visualization. Here, we are interested in more quantitative metrics and the practical benefits of improved sampling. For this, we consider optimization performance as the evaluation metric.
The gradient of the log-likelihood function L(θ|vdata) of general RBM models is
∂L(θ|vdata) ∂θ = Eh|vdata
[ ∂E(v, h)
∂θ
] − Ev,h [ ∂E(v, h)
∂θ
] . (9)
Since the second expectation in (9) is usually intractable, different approximation algorithms are used (Fischer & Igel, 2012).
2Our GPU implementation with gnumpy and cudamat can reproduce the results of http://www.cs.toronto.edu/ tang/code/GaussianRBM.m
3CD-n means that contrastive divergence was run for n steps
In this section, we compare two gradient approximation procedures. The baselines are the conventional contrastive divergence (CD) (Hinton, 2002) and persistent contrastive divergence (Tieleman, 2008) (PCD). The second method is using Algorithm 2 (Leaky) with the same number of mixing steps as CD. The experiment setup is the same as that of Section 4.
The results are shown in Figure 4. The proposed sampling procedure is slightly better than typical CD steps. The reason is we only anneals the leakiness for 20 steps. To get accurate estimation requires thousands of steps as shown in Section 4 when we estimate the partition function. Therefore, the estimated gradient is still inaccurate. However, it still outperforms the conventional CD algorithm. On the other hand, unlike the binary RBM case shown in Tieleman (2008), PCD does not outperform CD with 20 mixing steps for leaky RBM.
The drawback of Algorithm 2 is that sampling v from N ( (I −WW>)−1Wb, (I −WW>)−1 ) requires computing mean, covariance and the Cholesky decomposition of the covariance matrix in every iteration, which are computationally expensive. We study a mixture algorithm by combining CD and the idea of annealing leakiness. The mixture algorithm replaces the sampling from N ( (I −WW>)−1Wb, (I −WW>)−1 ) with sampling from the empirical data distribution. The resulted mix algorithm is almost the same as CD algorithm while it anneals the leakiness over the iterations as Algorithm 2. The results of the mix algorithm is also shown in Figure 4.
The mix algorithm is slightly worse than the original leaky algorithm, but it also outperforms the conventional CD algorithm without additional computation cost. The comparison in terms of CPU time is shown in Appendix F. Annealing the leakiness helps the mix algorithm explore different modes of the distribution, thereby improves the training. The idea could also be combined with more advanced algorithms (Tieleman, 2008; Tieleman & Hinton, 2009)4.
6 CONCLUSION
In this paper, we study the properties of the exponential family distribution produced by leaky RBM. This study relates the leaky RBM model and truncated Gaussian distribution and reveals an underlying positive definite constraint of training leaky RBM. We further proposed a meta sampling algorithm, which anneals between leakiness during the Gibbs sampling procedure. We first demonstrate the proposed sampling algorithm is significantly more effective and efficient in estimating the partition function than the conventional AIS algorithm. Second, we show that the proposed sampling algorithm has comparatively better mixing properties (compared to CD). A few direction are worth further study; in particular we are investigating on speeding up the naive projection step; either using the barrier function as shown in Hsieh et al. (2011) or by eliminating the need for projection by artificially bounding the domain via additional constraints.
4We studied the PCD extension of the proposed sampling algorithm. However, the performance is not as stable as CD.
A DERIVATION OF LEAKY (RELU) RBM
A.1 CONDITIONAL DISTRIBUTIONS
For leaky RBM, the activation function of hidden units is defined as f(ηj) = max(cηj , ηj), where c ∈ (0, 1) and ηj = ∑I i=1Wijvi + bj . The inverse function of f is f
−1(hj) = min(hj , hj/c). Therefore, the anti-derivatives are
F (ηj) =
{ 1 2η 2 j , if ηj > 0
c 2η 2 j , else,
(10)
and
F ∗(hj) =
{ 1 2h 2 j , if ηj > 0
1 2ch 2 j , else.
(11)
The activation function of Gaussian visible units can be treated as the linear unit f̃(νi) = νi, where νi = ∑J j=1Wijhj . Following the similar steps for deriving F and F
∗, we get the anti-derivatives F̃ (νi) = 1 2ν 2 i and F̃ ∗(vi) = 1 2v 2 i .
From Ravanbakhsh et al. (2016), the conditional distribution is defined as
p(hj |ηj) = exp (−ηjhj + F (ηj) + F ∗(hj)) (12)
By plugging F and F ∗ into (12), we get the conditional distribution for leaky RBM
p(hj |v) = { N (ηj , 1)with g(hj) = − log( √ 2π), if ηj > 0
N (cηj , c)with g(hj) = − log( √ 2cπ), if ηj ≤ 0. (13)
Similarly, we have p(vi|νi) = N (νi, 1) with g(vi) = − log( √ 2π).
A.2 JOINT AND MARGINAL DISTRIBUTIONS
Given the conditional distributions p(v|h) and p(h|v), the joint distribution p(v, h) from the general treatment for MRF model given by Yang et al. (2012) is
p(v, h) ∝ exp v>Wh− I∑ i=1 (F̃ ∗(vi) + g(vi))− J∑ j=1 (F ∗(hj) + g(hj))  , (14) By plugging F ∗, F̃ ∗ and g from Section A.1 into (14), we have
p(v, h) ∝ exp v>Wh− ‖v‖2 2 − ∑ ηj>0 ( h2j 2 + log √ 2π ) − ∑ ηj≤0 ( h2j 2c + log √ 2cπ ) + b>h  , Then the marginal distribution is
p(v) ∝ ∫ h p(v, h)dh
∝ ∫ h exp ( −‖v‖ 2 2 ) ∏ ηj>0 exp ( − h2j 2 + ηjhj − log √ 2π ) ∏ ηj≤0 ( − h2j 2c + hjηj − log √ 2cπ ) dh
∝ exp ( −‖v‖ 2
2 ) ∏ ηj>0 exp ( η2j 2 ) ∏ ηj≤0 ( cη2j 2 )
∝ exp −1 2 v> I −∑ ηj>0 WjW > j − c ∑ ηj≤0 WjW > j  v + ∑ ηj>0 bjW > j v + c ∑ ηj≤0 bjW > j v  .
B PROOF OF THEOREM 1
Proof. Since WW>− ∑ j αjWjWj = ∑ j(1−αj)WjW>j 0, we have WW> ∑ j αjWjWj .
Therefore, I − ∑ j αjWjW > j I −WW> 0.
C PROOF OF THEOREM 2
Proof. Let the SVD decomposition of W and W̃ as W = USV > and W̃ = Ũ S̃Ṽ >. Then we have
‖W − W̃‖2F = ‖USV > − Ũ S̃Ṽ >‖2F ≥ I∑ i=1 (Sii − S̃ii)2, (15)
and the constraint I − W̃W̃> 0 can be rewritten as 0 ≤ S̃ii ≤ 1,∀i. The transformed problem has a Lasso-like formulation and we can solve it by S̃ii = min(Sii, 1) (Parikh & Boyd, 2014). Also, the lower bound ∑I i=1(Sii − S̃ii)2 in (15) becomes a tight bound when we set Ũ = U and Ṽ = V , which completes the proof.
D NECESSITY OF THE PROJECTION STEP
We conduct a short comparison to demonstrate the projection step is necessary for the leaky RBM on generative tasks. We train two leaky RBM as follows. The first model is trained by the same setting in Section 4. We use the convergence of log likelihood as the stopping criteria. The second model is trained by CD-1 with weight decay and without the projection step. We stop the training when the reconstruction error is less then 10−2. After we train these two models, we run Gibbs sampling with 1000 independent chains for several steps and output the average value of the visible units. Note that the visible units are normalized to zero mean. The results on SVHN and CIFAR10 are shown in Figure 5.
From Figure 5, the model trained by weight decay without projection step is suffered by the problem of the diverged values. It confirms the study shown in Section 3.1. It also implies that we cannot
train leaky RBM with larger CD steps when we do not do projection; otherwise, we would have the diverged gradients. Therefore, the projection is necessary for training leaky RBM for the generative purpose. However, we also oberseve that the projection step is not necessary for the classification and reconstruction tasks. he reason may be the independency of different evaluation criteria (Hinton, 2012; Theis et al., 2016) or other implicit reasons to be studied.
E EQUIVALENCE BETWEEN ANNEALING THE ENERGY AND LEAKINESS
We analyze the performance gap between AIS-Leaky and AIS-Energy. One major difference is the initial distribution. The intermediate marginal distribution of AIS-Energy has the following form:
pk(v) ∝ exp −1 2 v> I − (1− βk) ∑ ηj>0 WjW > j − (1− βk)c ∑ ηj≤0 WjW > j  v  . (16)
Here we eliminated the bias terms b for simplicity. Compared with Algorithm 2, (16) not only anneals the leakiness (1 − βk)c ∑ ηj≤0WjW > j when ηj ≤ 0, but also in the case (1 −
βk) ∑ ηj>0 WjW > j when ηj > 0, which brings more bias to the estimation. In other words, AIS-Leaky is a one-sided leakiness annealing while AIS-Energy is a two-sided leakiness annealing method.
To address the higher bias problem of AIS-Energy, we replace the initial distribution with the one used in Algorithm 2. By elementary calculation, the marginal distribution becomes
pk(v) ∝ exp −1 2 v> I −∑ ηj>0 WjW > j − (βk + (1− βk)c) ∑ ηj≤0 WjW > j  v  , (17)
which recovers the proposed Algorithm 2. From this analysis, we understand AIS-Leaky is a special case of conventional AIS-Energy with better initialization inspired by the study in Section 3. Also, by this connection between AIS-Energy and AIS-Leaky, we note that AIS-Leaky can be combined with other extensions of AIS (Grosse et al., 2013; Burda et al., 2015) as well.
F MORE EXPERIMENTAL RESULTS FOR SAMPLING
F.1 SAMPLED IMAGES
We show the sampled images from leaky RBM train on CIFAR10 and SVHN datasets. We randomly initialize 20 chains and run Gibbs sampling for 1000 iterations. The sampled results are shown in Figure 6 The results shows that single layer RBM does not adequately model CIFAR10 and SVHN
when compared to multilayer models. The similar results for single layer Bernoulli-Gaussian RBM from Ranzato & Hinton (2010) (in gray scale) is shown in Figure 7. Therefore, we instead focused on quantitative evaluation of the log-likelihood in Table 3.
F.2 COMPUTATIONAL TIME BETWEEN DIFFERENT SAMPLING STRATEGIES
The comparison in terms of CPU time of different sampling algorithms discussed in Section 5 is shown in Figure 8. Please note that the complexity of CD and Mix are the almost the same. Mix only need a few more constant time steps which can be ignored compared with sampling steps. Leaky is more time-consuming because of computing and decomposing the covariance matrix as we discussed in Section 5. We also report the execution time of each step of algorithms in Table 4.
F.3 STUDY ON RELU-BERNOULLI RBM
We study the idea of annealing leakiness on the RBM model with leaky ReLU hidden units and Bernoulli visible units. We create the toy dataset with 20, 25 and 30 visible units as shown in Figure 9. The small datasets allow exact computation of the partition function. For each dataset, we sample 60,000 images for training and 10,000 images for testing. We use 100 hidden units and PCD to train the model. The log likelihood results are shown in Table 5.
Compared to the Gaussian visible units case we study in Section 3, where p(v) is a multi-variate Gaussian distribution when c = 1, the partition function of p(v) in ReLU-Bernoulli when c = 1 does not have the analytical form. Therefore, we do the following two-stage alternative. We first run the standard AIS algorithm, which anneals the energy, to the distribution with leakiness c = 1. We then change to anneals the leakiness from 1 to the target value. For the typical AIS algorithm (AIS-Energy), we use 104 chains with 2 × 104 intermediate distributions. For the proposed twostaged algorithm (AIS-Leaky), we use 104 chains with 104 intermediate distributions for annealing to c = 1 and the other 104 distributions for annealing the leakiness. The results are shown in Table 6.
In Table 6, the standard AIS algorithm (AIS-Energy) has unsatisfactory performance. We show the performance of AIS for estimating the partition function of models with different leakiness on Toy20. We use the 104 independent chains and 2 × 104 intermediate distributions. The results are shown in Table 7. From Table 7, we observe that the AIS performances worse when the leakiness is closer to 0. Although we observed that increasing chains and intermediate distributions could improve the performance, but the improvements are limited. The study demonstrates when the
(a) I = 20
(b) I = 25
non-linearity of the distribution increases (the leakiness value c decreases), the standard AIS cannot effectively estimate the partition function within feasible computational time. On the other hand, it also confirm the proposed idea, annealing the leakiness, can serve as an effective building block for algorithms without enhancing the algorithm complexity. Note that the unsatisfactory performance of AIS may be addressed by Grosse et al. (2013). From Appendix E, the two-stage algorithm used here can also be improved by applying Grosse et al. (2013).
F.3.1 MNIST AND CALTECH DATASETS
We study MNIST and Caltech 101 Silhouettes datasets with 500 hidden units and train the model with CD-25. The results are shown in Table 8 and Table 9. The leaky RBM is better than conventional Bernoulli RBM and some deep models on MNIST data. Although leaky RBM deos not outperform Su et al. (2017), but it enjoys the advantage of the simpler sampling procedure (Gaussian distribution vs truncated Gaussian distribution) in the binary visible unit case.

Review:
The authors propose a novel energy-function for RBMs, using the leaky relu max(cx, x) activation function for the hidden-units. Analogous to ReLU units in feed-forward networks, these leaky relu RBMs split the input space into a combinatorial number of regions, where each region defines p(v) as a truncated Gaussian. A further contribution of the paper is in proposing a novel sampling scheme for the leaky RBM: one can run a much shorter Markov chain by initializing it from a sample of the leaky RBM with c=1 (which yields a standard multi-variate normal over the visibles) and then slowly annealing c. In low-dimension a similar scheme is shown to outperform AIS for estimating the partition function. Experiments are performed on both CIFAR-10 and SVHN.
This is an interesting paper which I believe would be of interest to the ICLR community. The theoretical contributions are strong: the authors not only introduce a proper energy formulation of ReLU RBMs, but also a novel sampling mechanism and an improvement on AIS for estimating their partition function. 
Unfortunately, the experimental results are somewhat limited. The PCD baseline is notably absent. Including (bernoulli visible, leaky-relu hidden) would have allowed the authors to evaluate likelihoods on standard binary RBM datasets. As it stands, performance on CIFAR-10 and SVHN, while improved with leaky-relu, is a far cry from more recent generative models (VAE-based, or auto-regressive models). While this comparison may be unfair, it will certainly limit the wider appeal of the paper to the community. Furthermore, there is the issue of the costly projection method which is required to guarantee that the energy-function remain bounded (covariance matrix over each region be PSD). Again, while it may be fair to leave that for future work given the other contributions, this will further limit the appeal of the paper.
PROS:
Introduces an energy function having the leaky-relu as an activation function
Introduces a novel sampling procedure based on annealing the leakiness parameter
Similar sampling scheme shown to outperform AIS
CONS:
Results are somewhat out of date
Missing experiments on binary datasets (more comparable to prior RBM work)
Missing PCD baseline
Cost of projection method