Paper:
Training Restricted Boltzmann Machines via the Thouless-Anderson-Palmer Free Energy
Restricted Boltzmann machines are undirected neural networks which have been shown to be effective in many applications, including serving as initializations for training deep multi-layer neural networks. One of the main reasons for their success is the existence of efficient and practical stochastic algorithms, such as contrastive divergence, for unsupervised training. We propose an alternative deterministic iterative procedure based on an improved mean field method from statistical physics known as the Thouless-Anderson-Palmer approach. We demonstrate that our algorithm provides performance equal to, and sometimes superior to, persistent contrastive divergence, while also providing a clear and easy to evaluate objective function. We believe that this strategy can be easily generalized to other models as well as to more accurate higher-order approximations, paving the way for systematic improvements in training Boltzmann machines with hidden units.

Review:
Detailed comments: 40 - You might be interested in [MacKay, David. "Failures of the one-step learning algorithm." 2001.]42 - PCD is a rediscovery of a method that has recurred several times before, and which I believe actually has some nice convergence properties, at least in terms of some limit in the ratio of learning rate and mixing time. [Neal, R. M. (1992). Connectionist learning of belief networks. Artificial Intelligence] is one earlier example of its use that I'm familiar with. A paper by Salakhutdinov suggests that [H. Robbins and S. Monro. A stochastic approximation method. Ann. Math. Stat., 22:400-407, 1951.] is the earliest reference, though I have not read this older paper.48 - Though they tend to break down around criticality, which is exactly where many interesting real world datasets are expected to be. e.g. see mean field theory with TAP doing worse than the random parameter initialization in [Sohl-Dickstein, Jascha, Peter Battaglino, and Michael R. DeWeese. "Minimum probability flow learning." 2011.]72 - nit - common convention is lowercase for vectors, upper case for matrices. maybe w_ij -> W_ij97 - since you never have to sample the visible units, you could us the more flexible sRBMs, rather than RBMs (sRBM = semi-restricted Boltzmann machine, an RBM w/ the addition of visible to visible couplings) (updated -- I see later that when you take your expansion to third order, this would make the sRBM expensive)132 - "where the dependence on beta of the product ..." huh? This didn't make sense to me.194 - re: careful application, what makes this part tricky?202 - "to gradients" -> "the gradients"210 - i.e. implicit differentiation254 - Why do you need multiple points? Doesn't m have a single fixed point?262-267 - Confused by this paragraph.Figure 1 - Label units on the y-axis! right pane - Should note that this pane is evaluating using the same approx. objective as used in training, so not a fair comparison for CD.296 - re: all parameters of training being set identically - this is not fair! For instance, the training gradient in PCD/CD will have a higher variance, and so it will probably perform best w/ a smaller learning rate than EMF. Might consider using a tool like Spearmint [Snoek, Jasper, Hugo Larochelle, and Ryan P. Adams. "Practical Bayesian optimization of machine learning algorithms." Advances in neural information processing systems. 2012.] as a black box (and thus parsimonious) way to set hyperparameters.Figure 2 - These look grayscale not binary. Are they actually samples or are they probabilities of activation? Make sure to label correctly. Samples give a more clear view of performance. - Curious to see P-TAP3!371-377 - What is the computational cost of these algorithms? What is the wall clock time relative to PCD?391 - "over-fitting" suggests that it's getting better on the training data. It's probably not.(one more comment after reading review from Assigned_Reviewer_4 -- score matching would not apply to this specific case, with RBMs with binary visible units. There are a number of other parameter estimation techniques for energy based models that would however -- minimum probability flow, ratio matching, noise contrastive estimation, minimum KL contraction.)