Paper:
Rectified Factor Networks
We propose rectified factor networks (RFNs) to efficiently construct very sparse, non-linear, high-dimensional representations of the input. RFN models identify rare and small events in the input, have a low interference between code units, have a small reconstruction error, and explain the data covariance structure. RFN learning is a generalized alternating minimization algorithm derived from the posterior regularization method which enforces non-negative and normalized posterior means. We proof convergence and correctness of the RFN learning algorithm. On benchmarks, RFNs are compared to other unsupervised methods like autoencoders, RBMs, factor analysis, ICA, and PCA. In contrast to previous sparse coding methods, RFNs yield sparser codes, capture the data's covariance structure more precisely, and have a significantly smaller reconstruction error. We test RFNs as pretraining technique for deep networks on different vision datasets, where RFNs were superior to RBMs and autoencoders. On gene expression data from two pharmaceutical drug discovery studies, RFNs detected small and rare gene modules that revealed highly relevant new biological insights which were so far missed by other unsupervised methods. RFN package for GPU/CPU is available at http://www.bioinf.jku.at/software/rfn.

Review:
The authors present an intriguing take on factor analysis: namely, the use of Ganchev et al's posterior regularization to enforce non negativity constraints on the posterior. They present proofs of convergence and correctness, and present a scalable method for inference and learning in stacked constrained factor analysis models.The method appears to be sound and is an interesting direction overall, a refreshing departure from much of the existing literature, and the first instance of which I am aware that posterior regularization has been highlighted in a deep learning/unsupervised feature learning context. While I did not review it in detail, the degree of thoroughness demonstrated by the supplementary material is truly impressive.My main concern with this paper is one of being somewhat underwhelmed by the empirical evaluation, in light of the norms of the community. It makes the contribution difficult to judge from that vantage point.In particular, one question I'm left with is whether there is a reason that these models, at least in the one layer case (and probably also in the multi-layer case) cannot be evaluated in terms of their test set likelihood. The quantitative evaluation of unsupervised methods on MNIST, for example, is usually phrased in terms of test set likelihood. Synthetic data results are presented for a set of chosen metrics meant to demonstrate the effectiveness of the model at satisfying the design goals of RFNs, but are difficult to interpret. The classification baselines employed also seem to be a bit dated. For example, Komer et al (2014) report a test set error of 11.7% on CONVEX using a combination of a polynomial SVM and PCA preprocessing. While that result is relatively hard to find, unsupervised learning on CIFAR, for example, has progressed quite a bit (albeit in some cases with domain-knowledge-heavy architectures).- "Current unsupervised deep learning approaches like autoencoders or restricted Boltzmann machines (RBMs) do not model specific structures in the data." This is unclear. It seems that the filters of individual units can model "specific structures". The next sentence refers to "generative models" but RBMs and (deep belief networks built from them) _are_ generative models. More precision is required in this section about what is meant. - I'm curious as to the authors' response to the success of representations that eschew sparsity altogether and still manage to perform very well, such as maxout networks (Goodfellow et al, 2013). - More elaboration on the role of normalizing constraints would be helpful in exposition. - I assume the "projected Newton method" is the Newton step projected onto the constraint surface in terms of the closest (in terms of L2 distance) point obeying the constraint. It would be good to explicitly state this.POST-REBUTTAL COMMENTS: - The argument against using likelihood is somewhat compelling though it seems like in a deep model one should be able to recover the lost ground caused by posterior regularization in a shallow model. - I'd suggest moving the CIFAR results front and center along with the drug design stuff, and perhaps relegating the table presented to the supplementary. CIFAR10/100 being a point of contact for many in the deep learning world I think highlighting that in the main text is helpful.I've upgraded my score to a 7.