Paper:
This paper studies the cooperative training of two probabilistic models of signals such as images. Both models are parametrized by convolutional neural networks (ConvNets). The first network is a descriptor network, which is an exponential family model or an energy-based model, whose feature statistics or energy function are defined by a bottom-up ConvNet, which maps the observed signal to the feature statistics. The second network is a generator network, which is a nonlinear version of factor analysis. It is defined by a top-down ConvNet, which maps the latent factors to the observed signal. The maximum likelihood training algorithms of both the descriptor net and the generator net are in the form of alternating back-propagation, and both algorithms involve Langevin sampling. We observe that the two training algorithms can cooperate with each other by jumpstarting each other’s Langevin sampling, and they can be seamlessly interwoven into a CoopNets algorithm that can train both nets simultaneously.
1 INTRODUCTION
1.1 TWO CONVNETS OF OPPOSITE DIRECTIONS
We begin with a story that the reader of this paper can readily relate to. A student writes up an initial draft of a paper. His advisor then revises it. After that they submit the revised paper for review. The student then learns from his advisor’s revision, while the advisor learns from the outside review. In this story, the advisor guides the student, but the student does most of the work.
This paper is about two probabilistic models of signals such as images, and they play the roles of student and advisor as described above. Both models are parametrized by convolutional neural networks (ConvNets or CNNs) (LeCun et al., 1998; Krizhevsky et al., 2012). The two nets take two opposite directions. One is bottom-up, and the other is top-down, as illustrate by the following diagram:
Bottom-up ConvNet Top-down ConvNet features latent variables ⇑ ⇓
signal signal (a) Descriptor Net (b) Generator Net
(1)
The simultaneous training of such two nets was first studied by the recent work of Kim & Bengio (2016). These two nets belong to two major classes of probabilistic models. (a) The exponential family models or the energy-based models (LeCun et al., 2006) or the Markov random field models (Zhu et al., 1997), where the probability distribution is defined by feature statistics or energy function computed from the signal by a bottom-up process. (b) The latent variable models or the directed graphical models, where the signal is assumed to be a transformation of the latent factors that follow a known prior distribution. The latent factors generate the signal by a top-down process. A classical example is factor analysis.
The two classes of models have been contrasted by Zhu (2003); Teh et al. (2003); Ngiam et al. (2011). Zhu (2003) called the two classes of models the descriptive models and the generative
models respectively. Both classes of models can benefit from the high capacity of the multi-layer ConvNets. (a) In the exponential family models or the energy-based models, the feature statistics or the energy function can be defined by a bottom-up ConvNet that maps the signal to the features and the energy function (Ngiam et al., 2011; Xie et al., 2016). We call the resulting model a descriptive network or a descriptor net following Zhu (2003), because it is built on descriptive feature statistics. (b) In the latent variable models or the directed graphical models, the transformation from the latent factors to the signal can be defined by a top-down ConvNet (Dosovitskiy et al., 2015), which maps the latent factors to the signal. We call the resulting model a generative network or generator net following Goodfellow et al. (2014), who proposed such a model in their work on the generative adversarial networks (GAN).
1.2 TWO TRAINING ALGORITHMS AND THEIR COOPERATION
Fig. 1(a) and (b) display the flowcharts of the maximum likelihood learning algorithms for training the descriptor and generator nets . We call the two algorithms Algorithm D and Algorithm G respectively. Algorithm D (Xie et al., 2016) iterates two steps: Step D1 synthesizes examples by sampling from the current model by Langevin dynamics. Step D2 updates the parameters to shift the density from the synthesized examples towards the observed examples. Algorithm G (Han et al., 2017) also iterates two steps. Step G1 infers latent factors for each observed example by sampling from their posterior distribution by Langevin dynamics. Step G2 updates the parameters by a non-linear regression of the observed examples on their corresponding latent factors. We use Langevin dynamics for Markov chain Monte Carlo (MCMC) sampling because the gradient term of Langevin dynamics can be readily computed via back-propagation. Thus all the steps D1, D2 and G1, G2 are powered by back-propagation, and both Algorithms D and G are alternating back-propagation algorithms.
In this article, we propose to couple Algorithms D and G into a cooperative training algorithm that interweaves the steps of the two algorithms seamlessly. We call the resulting algorithm the CoopNets algorithm, and we show that it can train both nets simultaneously.
Figure 1(c) displays the flowchart of the CoopNets algorithm. The generator is like the student. It generates the initial draft of the synthesized examples. The descriptor is like the advisor. It revises the initial draft by initializing its Langevin dynamics from the initial draft in Step D1, which produces the revised draft of the synthesized examples. The descriptor learns from the outside review in Step D2, which is in the form of the difference between the observed examples and the revised
synthesized examples. The generator learns from how the descriptor revises the initial draft by reconstructing the revised draft in Step G2. For each synthesized example, the generator knows the latent factors that generate the initial draft, so that Step G1 can infer the latent factors by initializing its Langevin dynamics from their known values.
In the CoopNets algorithm, the generator fuels the MCMC of the descriptor by supplying initial synthesized examples, which can be obtained by direct ancestral sampling. The generator then learns from the revised synthesized examples with virtually known latent factors. The cooperation is thus beneficial to both nets.
2 RELATED WORK
Our work is inspired by the generative adversarial networks (GAN) (Goodfellow et al., 2014; Denton et al., 2015; Radford et al., 2015). In GAN, the generator net is paired with a discriminator net. The two nets play adversarial roles. In our work, the generator net and the descriptor net play cooperative roles, and they feed each other the initial, revised and reconstructed synthesized data. The learning of both nets is based on maximum likelihood, and the learning process is quite stable because of the cooperative nature and the consistent directions of the two maximum likelihood training algorithms.
Another method to train the generator network is variational auto-encoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014; Mnih & Gregor, 2014), which learns an inferential or recognition network to approximate the posterior distribution of the latent factors.
The connection between the descriptor net and the discriminator net has been explored by Xie et al. (2016), where the descriptor can be derived from the discriminator.
Our work is most similar to the recent work of Kim & Bengio (2016). In fact, the settings of the two nets are the same. In their work, the generator learns from the descriptor by minimizing the Kullback-Leibler divergence from the generator to the descriptor, which can be decomposed into an energy term and an entropy term. In our work, the two nets interact with each other via synthesized data, and the generator learns from the descriptor by reconstructing the revised draft of synthesized examples. Our method does not need to approximate the intractable entropy term.
Our work is related to the contrastive divergence algorithm (Hinton, 2002) for training the descriptor net. The contrastive divergence initializes the MCMC sampling from the observed examples. The CoopNets algorithm initializes the MCMC sampling from the examples supplied by the generator.
3 TWO NETS AND TWO TRAINING ALGORITHMS
3.1 DESCRIPTOR NET AND TRAINING ALGORITHM
Let Y be the D-dimensional signal, such as an image. The descriptor model is in the form of exponential tilting of a reference distribution (Xie et al., 2016):
PD(Y ;WD) = 1
Z(WD) exp [f(Y ;WD)] q(Y ), (2)
where q(Y ) is the reference distribution such as Gaussian white noise q(Y ) ∝ exp ( −‖Y ‖2/2s2 ) ,
f(Y ;WD) (f stands for features) is the feature statistics or energy function, defined by a ConvNet whose parameters are denoted by WD. This ConvNet is bottom-up because it maps the signal Y to a number. See the diagram in (1). Z(WD) = ∫ exp [f(Y ;WD)] q(Y )dY = Eq{exp[f(Y ;WD)]} is the normalizing constant, where Eq is the expectation with respect to q.
Suppose we observe training examples {Yi, i = 1, ..., n} from an unknown data distribution Pdata(Y ). The maximum likelihood training seeks to maximize the log-likelihood function LD(WD) = 1 n ∑n i=1 logPD(Yi;WD). If the sample size n is large, the maximum likelihood estimator minimizes KL(Pdata|PD), the Kullback-Leibler divergence from the data distribution Pdata to the model distribution PD. The gradient of the LD(WD) is
L′D(WD) = 1
n n∑ i=1 ∂ ∂WD f(Yi;WD)− EWD [ ∂ ∂WD f(Y ;WD) ] , (3)
where EWD denotes the expectation with respect to PD(Y ;WD).
The expectation in equation (3) is analytically intractable and has to be approximated by MCMC, such as Langevin dynamics, which iterates the following step:
Yτ+1 = Yτ − δ2
2 [ Yτ s2 − ∂ ∂Y f(Yτ ;WD) ] + δUτ , (4)
where τ indexes the time steps of the Langevin dynamics, δ is the step size, and Uτ ∼ N(0, ID) is the Gaussian white noise term.
We can run ñ parallel chains of Langevin dynamics according to (4) to obtain the synthesized examples {Ỹi, i = 1, ..., ñ}. The Monte Carlo approximation to L′D(WD) is
L′D(WD) ≈ 1
n n∑ i=1 ∂ ∂WD f(Yi;WD)− 1 ñ ñ∑ i=1 ∂ ∂WD f(Ỹi;WD), (5)
which is used to update WD.
Algorithm D (Xie et al., 2016) iterates the following two steps after initializing WD and {Ỹi, i = 1, ..., ñ}. Step D1: Run lD steps of Langevin from the current {Ỹ } according according to (4). Step D2: update W (t+1)D = W (t) D + γtL ′ D(W (t) D ) with learning rate γt. The convergence of such an algorithm follows Younes (1999).
3.2 GENERATOR NET AND TRAINING ALGORITHM
The generator net (Goodfellow et al., 2014) seeks to explain the signal Y of dimensionD by a vector of latent factors X of dimension d, and usually d D. The model is of the following form:
X ∼ N(0, Id), Y = g(X;WG) + , ∼ N(0, σ2ID). (6) g(X;WG) (g stands for generator) is a top-down ConvNet defined by the parameters WG . The ConvNet g maps the latent factors X to the signal Y . See the diagram in (1).
The joint density of model (6) is PG(X,Y ;WG) = PG(X)PG(Y |X;WG), and
logPG(X,Y ;WG) = − 1
2σ2 ‖Y − g(X;WG)‖2 −
1 2 ‖X‖2 + constant, (7)
where the constant term is independent of X , Y and WG . The marginal density is obtained by integrating out the latent factors X , i.e., PG(Y ;WG) = ∫ PG(X,Y ;WG)dX . The inference of X given Y is based on the posterior density PG(X|Y ;WG) = PG(X,Y ;WG)/PG(Y ;WG) ∝ PG(X,Y ;WG) as a function of X .
For the training data {Yi, i = 1, ..., n}, the generator net can be trained by maximizing the loglikelihood LG(WG) = 1n ∑n i=1 logPG(Yi;WG). For large sample, the learned WG minimizes the Kullback-Leibler divergence KL(Pdata|PG) from the data distribution Pdata to the model distribution PG . The gradient of LG(WG) is obtained according to the following identity
∂
∂WG logPG(Y ;WG) =
1
PG(Y ;WG)
∂
∂WG
∫ PG(Y,X;WG)dX
= 1
PG(Y ;WG)
∫ [ ∂
∂WG logPG(Y,X;WG)
] PG(Y,X;WG)dX
=
∫ [ ∂
∂WG logPG(Y,X;WG)
] PG(Y,X;WG)
PG(Y ;WG) dX
= EPG(X|Y ;WG)
[ ∂
∂WG logPG(X,Y ;WG)
] , (8)
which underlies the EM algorithm. In general, the expectation in (8) is analytically intractable, and has to be approximated by MCMC that samples from the posterior PG(X|Y ;WG), such as Langevin dynamics, which iterates
Xτ+1 = Xτ + δ2
2
∂
∂X logPG(Xτ , Y ;WG) + δUτ , (9)
where Uτ ∼ N(0, Id). With Xi sampled from PG(Xi | Yi,WG) for each observation Yi, the Monte Carlo approximation to L′G(WG) is
L′G(WG) ≈ 1
n n∑ i=1 ∂ ∂WG logPG(Xi, Yi;WG) = 1 n n∑ i=1 1 σ2 (Yi − g(Xi;WG)) ∂ ∂WG g(Xi;WG). (10)
Algorithm G (Han et al., 2017) iterates the following two steps after initializing WG and {Xi, i = 1, ..., n}. Step G1: run lG steps of Langevin from the current {Xi} according to (9). Step G2: update W
(t+1) G =W (t) G +γtL ′ G(W (t) G ) with learning rate γt. The convergence of such an algorithm follows
Younes (1999).
4 COOPNETS ALGORITHM: RECONSTRUCTING THE REVISION
In Algorithms D and G, both steps D1 and G1 are Langevin dynamics, which may be slow to converge. An interesting observation is that the two algorithms can cooperate with each other by jumpstarting each other’s Langevin sampling.
Specifically, in Step D1, we can initialize the synthesized examples by generating examples from the generator net. We first generate X̂i ∼ N(0, Id), and then generate Ŷi = g(X̂i;WG) + i, for i = 1, ..., ñ. If the current generator PG is close to the current descriptor PD, then the generated {Ŷi} should be a good initialization for sampling from the descriptor net, i.e., starting from the {Ŷi, i = 1, ..., ñ}, we run Langevin dynamics in Step D1 for lD steps to get {Ỹi, i = 1, ..., ñ}, which are revised versions of {Ŷi}. These {Ỹi} can be used as the synthesized examples from the descriptor net. We can then update WD according to Step D2 of Algorithm D.
In order to update WG of the generator net, we treat the {Ỹi, i = 1, ..., ñ} produced by the above Step D1 as the training data for the generator. Since these {Ỹi} are obtained by the Langevin dynamics initialized from the {Ŷi, i = 1, ..., ñ} produced by the generator net with known latent factors {X̂i, i = 1, ..., ñ}, we can update WG by learning from {(Ỹi, X̂i), i = 1, ..., ñ}, which is a supervised learning problem, or more specifically, a non-linear regression of Ỹi on X̂i. At W (t) G , the latent factors X̂i generates and thus reconstructs the initial example Ŷi. After updating WG , we want X̂i to reconstruct the revised example Ỹi. That is, we revise WG to absorb the revision from Ŷi to Ỹi, so that the generator shifts its density from {Ŷi} to {Ỹi}. The reconstruction error can tell us whether the generator has caught up with the descriptor by fully absorbing the revision.
The left diagram in (11) illustrates the basic idea.
X̂i
Ŷi Ỹi
W (t) G W (t+1) G
W (t) D
X̂i Xi
Ŷi Ỹi
W (t) G
W (t) G W (t+1) G
W (t) D (11)
In the two diagrams in (11), the double-line arrows indicate generation and reconstruction by the generator net, while the dashed-line arrows indicate Langevin dynamics for revision and inference in the two nets. The diagram on the right in (11) illustrates a more rigorous method, where we initialize the Langevin inference of {Xi, i = 1, ..., ñ} in Step G1 from {X̂i}, and then update WG in Step G2 based on {(Ỹi, Xi), i = 1, ..., ñ}. The diagram on the right shows how the two nets jumpstart each other’s Langevin dynamics.
Algorithm 1 describes the cooperative training that interweaves Algorithm D and Algorithm G. See Figure 1(c) for the flowchart of the CoopNets algorithm. In our experiments, we set lG = 0 and infer Xi = X̂i for simplicity, i.e., we follow the left diagram in (11).
See Appendix for a theoretical understanding of the convergence of the CoopNets algorithm.
Algorithm 1 CoopNets Algorithm Input:
(1) training examples {Yi, i = 1, ..., n} (2) numbers of Langevin steps lD ad lG (3) number of learning iterations T
Output: (1) estimated parameters WD and WG (2) synthesized examples {Ŷi, Ỹi, i = 1, ..., ñ}
1: Let t← 0, initialize WD and WG . 2: repeat 3: Step G0: For i = 1, ..., ñ, generate X̂i ∼ N(0, Id), and generate Ŷi = g(X̂i;W (t)G ) + i. 4: Step D1: For i = 1, ..., ñ, starting from Ŷi, Run lD steps of Langevin dynamics to obtain Ỹi, each step following equation (4). 5: Step G1: Treat the current {Ỹi, i = 1, ..., ñ} as the training data, for each i, infer Xi = X̂i.
Or more rigorously, starting from Xi = X̂i, run lG steps of Langevin dynamics to update Xi, each step following equation (9).
6: Step D2: Update W (t+1)D = W (t) D + γtL ′ D(W (t) D ), where L ′ D(W (t) D ) is computed according to (5). 7: Step G2: Update W (t+1)G = W (t) G + γtLG ′(W (t) G ), where LG
′(WG) is computed according to (10), except that Yi is replaced by Ỹi, and n by ñ.
8: Let t← t+ 1 9: until t = T
5 EXPERIMENTS
We use the MatConvNet of Vedaldi & Lenc (2015) for coding. For the descriptor net, we adopt the structure of Xie et al. (2016), where the bottom-up network consists of multiple layers of convolution by linear filtering, ReLU non-linearity, and down-sampling. We adopt the structure of the generator network of Radford et al. (2015); Dosovitskiy et al. (2015), where the top-down network consists of multiple layers of deconvolution by linear superposition, ReLU non-linearity, and up-sampling, with tanh non-linearity at the bottom-layer (Radford et al., 2015) to make the signals fall within [−1, 1].
5.1 QUANTITATIVE EXPERIMENT ON FACE COMPLETION
We conduct an experiment on learning from complete training images of human faces, and then testing the learned model on completing the occluded testing images. The structure of the generator network is the same as in (Radford et al., 2015; Dosovitskiy et al., 2015). We adopt a 4-layer descriptor net. The first layer has 96 5 × 5 filters with sub-sampling of 2, the second layers has 128 5 × 5 filters with sub-sampling of 2, the third layer has 256 5 × 5 filters with sub-sampling of 2, and the final layer is a fully connected layer with 50 channels as output. We use L=10 steps of Langevin revision dynamics within each learning iteration, and the Langevin step size is set at 0.002. The learning rate is 0.07. The training data are 10, 000 human faces randomly selected from CelebA dataset (Liu et al., 2015). We run 600 cooperative learning iterations. Figure 2 displays 144 synthesized human faces by the descriptor net.
To quantitatively test whether we have learned a good generator net g(X;WG) even though it has never seen the training images directly in the training stage, we apply it to the task of recovering the occluded pixels of testing images. For each occluded testing image Y , we use Step G1 of Algorithm G to infer the latent factors X . The only change is with respect to the term ‖Y − g(X;WG)‖2, where the sum of squares is over all the observed pixels of Y in back-propagation computation. We run 1000 Langevin steps, initializing X from N(0, Id). After inferring X , the completed image g(X;WG) is automatically obtained. We design 3 experiments, where we randomly place a 20×20, 30 × 30, or 40 × 40 mask on each 64 × 64 testing image. These 3 experiments are denoted by M20 M30, and M40 respectively (M for mask). We report the recovery errors and compare our method with 8 different image inpainting methods as well as the DCGAN of Radford et al. (2015).
For DCGAN, we use the parameter setting in Radford et al. (2015) except changing the number of learning iterations to 600. We use the same 10, 000 training images to learn DCGAN. After the model is learned, we keep the generator and use the same method as ours to infer latent factors X , and recover the unobserved pixels. In 8 inpainting methods, Methods 1 and 2 are based on Markov random field prior where the nearest neighbor potential terms are `2 and `1 differences respectively. Methods 3 to 8 are interpolation methods. Please refer to D’Errico (2004) for more details. Table 1 displays the recovery errors of the 3 experiments, where the error is measured by per pixel difference (relative to the range of pixel values) between the original image and the recovered image on the occluded region, averaged over 100 testing images. Fig. 3 displays some recovery results by our method. The first row shows the original images as the ground truth. The second row displays the testing images with occluded pixels. The third row displays the recovered images by the generator net trained by the CoopNets algorithm on the 10,000 training images.
5.2 QUALITATIVE EXPERIMENT ON SYNTHESIS
We conduct an experiment on synthesizing images of categories from Imagenet ILSVRC2012 dataset (Deng et al., 2009) and MIT places205 dataset (Zhou et al., 2014). We adopt a 4-layer descriptor net. The first layer has 64 5 × 5 filters with sub-sampling of 2, the second layers has 128 3 × 3 filters with sub-sampling of 2, the third layer has 256 3 × 3 filters with sub-sampling of 1, and the final layer is a fully connected layer with 100 channels as output. We set the number of Langevin dynamics steps in each learning iteration to 10 and the step size to 0.002. The learning rate is 0.07. For each category, we randomly choose 1,000 images as training data and resize the images to 64× 64. We run 1, 000 cooperative learning iterations to train the model. Figures 4 and 5 display the results for two categories, where for each category, we show 144 original images sampled from the training set, and 144 synthesized images generated by our method. The appendix contains more synthesis results.
As a comparison, we apply the Algorithm G alone and GAN code on the same 1,000 hotel room training images to learn the generator of the same structure as in CoopNets. Figure 6 displays the synthesis results.
We also try to synthesize images at high resolution (224× 224). We adopt a 4-layer descriptor net. The first layer has 128 15× 15 filters with sub-sampling of 3, the second layer has 256 3× 3 filters with sub-sampling of 2, the third layer has 512 3 × 3 filters with sub-sampling of 1, and the final layer is a fully connected layer with 100 channels as output. We enlarge the filters of the final layer of generator net to 14 × 14 to generate 224 × 224 images. The learning rate is 0.05. We run 1000 cooperative learning iterations to train the model. Figures 7 and 8 show the synthesized images of two categories from MIT places205 dataset.
6 CONCLUSION
The most unique feature of our work is that the two networks feed each other the synthesized data in the learning process, including initial, revised, and reconstructed synthesized data.
Another unique feature of our work is that the learning process interweaves the existing maximum likelihood learning algorithms for the two networks.
A third unique feature of our work is that the MCMC for the descriptor keeps rejuvenating the chains by refreshing the samples by independent replacements supplied by the generator, so that a single chain effectively amounts to an infinite number of chains or the evolution of the whole marginal distribution modeled by the generator.
CODE AND DATA
http://www.stat.ucla.edu/˜ywu/CoopNets/main.html
7 APPENDIX: CONVERGENCE
7.1 GENERATOR OF INFINITE CAPACITY
In the CoopNets algorithm, the descriptor learns from the observed examples, while the generator learns from the descriptor through the synthesized examples. Therefore, the descriptor is the driving force in terms of learning, although the generator is the driving force in terms of synthesis. In order to understand the convergence of learning, we can start from Algorithm D for learning the descriptor.
Algorithm D is a stochastic approximation algorithm (Robbins & Monro, 1951), except that the samples are generated by finite step MCMC transitions. According to Younes (1999), Algorithm D converges to the maximum likelihood estimate under suitable regularity conditions on the mixing of the transition kernel of the MCMC and the schedule of the learning rate γt, even if the number of Langevin steps lD is finite or small (e.g., lD = 1), and even if the number of parallel chains ñ is finite or small (e.g., ñ = 1). The reason is that the random fluctuations caused by the finite number of chains, ñ, and the limited mixing caused by the finite steps of MCMC, lD, are mitigated if the
learning rate γt is sufficiently small. At learning iteration t, let W (t) D be the estimated parameter of the descriptor. Let P (t+1)D be the marginal distribution of {Ỹi}. Even though P (t+1) D 6= PD(Y ;W (t) D ) because lD is finite (P (t+1) D = PD(Y ;W (t) D ) if lD → ∞), we still have W (t) D → ŴD in probability according to Younes (1999), where ŴD is the maximum likelihood estimate of WD.
The efficiency of Algorithm D increases if the number of parallel chains ñ is large because it leads to more accurate estimation of the expectation in the gradient L′D(WD) of equation (3), so that we can afford to use larger learning rate γt for faster convergence.
Now let us come back to the CoopNets algorithm. In order to understand how the descriptor net helps the training of the generator net, let us consider the idealized scenario where the number of parallel chains ñ → ∞, and the generator has infinite capacity, and in each iteration it estimates WG by maximum likelihood using the synthesized data from P (t+1)D . In this idealized scenario, the learned generator PG(Y ;W (t+1) G ) will reproduce P (t+1) D by minimizing KL(P (t+1) D (Y )|PG(Y ;WG)), with P (t+1) D serving as its data distribution. Then eventually the learned generator PG(Y, ŴG) will reproduce PD(Y ; ŴD). Thus the cooperative training helps the learning of the generator. Note that the learned generator PG(Y, ŴG) will not reproduce the distribution of the observed data Pdata, unless the descriptor is of infinite capacity too.
Conversely, the generator net also helps the learning of the descriptor net in the CoopNets algorithm. In Algorithm D, it is impractical to make the number of parallel chains ñ too large. On the other hand, it would be difficult for a small number of chains {Ỹi, i = 1, ..., ñ} to explore the state space. In the CoopNets algorithm, because PG(Y ;W (t) G ) reproduces P (t) D , we can generate a completely new batch of independent samples {Ŷi} from PG(Y ;W (t)G ), and revise {Ŷi} to {Ỹi} by Langevin dynamics, instead of running Langevin dynamics from the same old batch of {Ỹi} as in the original Algorithm D. This is like implementing an infinite number of parallel chains, because each iteration evolves a fresh batch of examples, as if each iteration evolves a new set of chains. By updating the generator WG , it is like we are updating the infinite number of parallel chains, because WG memorizes the whole distribution. Even if ñ in the CoopNets algorithm is small, e.g., ñ = 1, viewed from the perspective of Algorithm D, it is as if ñ→∞. Thus the above idealization ñ→∞ is sound.
7.2 GENERATOR OF FINITE CAPACITY
From an information geometry point of view, let D = {PD(Y ;WD),∀WD} be the manifold of the descriptor models, where each distribution PD(Y ;WD) is a point on this manifold. Then the maximum likelihood estimate of WD is a projection of the data distribution Pdata onto the manifold D. Let G = {PG(Y ;WG),∀WG} be the manifold of the generator models, where each distribution PG(Y ;WG) is a point on this manifold. Then the maximum likelihood estimate ofWG is a projection of the data distribution Pdata onto the manifold G. From now on, for notational simplicity and with a slight abuse of notation, we use WD to denote the descriptor distribution PD(Y ;WD), and use WG to denote the generator distribution PG(Y ;WG).
We assume both the observed data size n and the synthesized data size ñ are large enough so that we shall work on distributions or populations instead of finite samples. As explained above, assuming ñ→∞ is sound because the generator net can supply unlimited number of examples.
The Langevin revision dynamics runs a Markov chain from W (t)G towards W (t) D . Let LWD be the Markov transition kernel of lD steps of Langevin revisions towards WD. The distribution of the revised synthesized data is
P (t+1) D = LW (t)D ·W (t)G , (12)
where the notation L ·P denotes the marginal distribution obtained by running the Markov transition L fromP . The distributionP (t+1)D is in the middle between the two netsW (t) G andW (t) D , and it serves as the data distribution to train the generator, i.e., we project this distribution onto the manifold G = {PG(Y ;WG),∀WG} = {WG} (recall we use WG to denote the distribution PG(Y ;WG)) in the
information geometry picture, so that
W (t+1) G = argminG KL(P (t+1) D |WG). (13)
The learning process alternates between Markov transition in (12) and projection in (13), as illustrated by Figure 9.
In the case of lD →∞,
W (t) D → ŴD = argminD KL(Pdata|WD), (14)
W (t) G → ŴG = argminG KL(ŴD|WG). (15)
That is, we first project Pdata onto D, and from there continue to project onto G. Therefore, WD converges to the maximum likelihood estimate with Pdata being the data distribution, while WG converges to the maximum likelihood estimate with ŴD serving as the data distribution.
For finite lD, the algorithm may converge to the following fixed points. The fixed point for the generator satisfies
ŴG = argmin G KL(LŴD · ŴG |WG). (16)
The fixed point for the descriptor satisfies
ŴD = argmin D [ KL(Pdata|WD)−KL(LŴD · ŴG |WD) ] , (17)
which is similar to contrastive divergence (Hinton, 2002), except that ŴG takes the place of Pdata in the second Kullback-Leibler divergence. Because ŴG is supposed to be close to ŴD, the second Kullback-Leibler divergence is supposed to be small, hence our algorithm is closer to maximum likelihood learning than contrastive divergence.
Kim & Bengio (2016) learned the generator by gradient descent on KL(WG |W (t)D ) over G. The objective function is KL(WG |W (t)D ) = EWG [logPG(Y ;WG)]− EWG [logPD(Y ;W (t) D )], where the first term is the negative entropy that is intractable, and the second term is the expected energy that is tractable. Our learning method for the generator is consistent with the learning objective KL(WG |W (t)D ), because
KL(P (t+1) D |W (t) D ) ≤ KL(W (t) G |W (t) D ). (18)
In fact, KL(P (t+1)D |W (t) D ) → 0 monotonically as lD → ∞ due to the second law of thermodynamics. The reduction of the Kullback-Leibler divergence in (18) and the projection in (13) in our learning of the generator are consistent with the learning objective of reducing KL(WG |W (t)D ) in Kim & Bengio (2016). But the Monte Carlo implementation of L in our work avoids the need to approximate the intractable entropy term.
7.3 MORE SYNTHESIS RESULTS
We display more synthesis results at the resolution of 64 × 64.
ACKNOWLEDGEMENT
We thank Hansheng Jiang for her work on this project as a summer visiting student. We thank Tian Han for sharing the code on learning the generator network, and for helpful discussions.
The work is supported by NSF DMS 1310391, DARPA SIMPLEX N66001-15-C-4035, ONR MURI N00014-16-1-2007, and DARPA ARO W911NF-16-1-0579.

Review:
This paper introduces CoopNets, an algorithm which trains a Deep-Energy Model (DEM, the “descriptor”) with the help of an auxiliary directed bayes net, e.g. “the generator”. The descriptor is trained via standard maximum likelihood, with Langevin MCMC for sampling. The generator is trained to generate likely samples under the DEM in a single, feed-forward ancestral sampling step. It can thus be used to shortcut expensive MCMC sampling, hence the reference to “cooperative training”.
The above idea is interesting and novel, but unfortunately is not sufficiently validated by the experimental results. First and foremost, two out of the three experiments do not feature a train /test split, and ignore standard training and evaluation protocols for texture generation (see [R1]). Datasets are also much too small. As such these experiments only seem to confirm the ability of the model to overfit. On the third in-painting tasks, baselines are almost non-existent: no VAEs, RBMs, DEM, etc which makes it difficult to evaluate the benefits of the proposed approach.
In a future revision, I would also encourage the authors to answer the following questions experimentally. What is the impact of the missing rejection step in Langevin MCMC (train with, without ?). What is the impact of the generator on the burn-in process of the Markov chain (show sample auto-correlation). How bad is approximation of training the generator from ({\tilde{Y}, \hat{X}) instead of ({\tilde{Y}, \tilde{X}) ? Run comparative experiments.
The paper would also greatly benefit from a rewrite focusing on clarity, instead of hyperbole (“pioneering work” in reference to closely related, but non-peer reviewed work) and prose (“tale of two nets”). For example, the authors fail to specify the exact form of the energy function: this seems like a glaring omission.
PROS:
+ Interesting and novel idea
CONS:
- Improper experimental protocols
- Missing baselines
- Missing diagnostic experiments
[R1] Heess, N., Williams, C. K. I., and Hinton, G. E. (2009). Learning generative texture models with extended fields of-experts.