Paper:
Object based Scene Representations using Fisher Scores of Local Subspace Projections
Several works have shown that deep CNNs can be easily transferred across datasets, e.g. the transfer from object recognition on ImageNet to object detection on Pascal VOC. Less clear, however, is the ability of CNNs to transfer knowledge across tasks. A common example of such transfer is the problem of scene classification, that should leverage localized object detections to recognize holistic visual concepts. While this problems is currently addressed with Fisher vector representations, these are now shown ineffective for the high-dimensional and highly non-linear features extracted by modern CNNs. It is argued that this is mostly due to the reliance on a model, the Gaussian mixture of diagonal covariances, which has a very limited ability to capture the second order statistics of CNN features. This problem is addressed by the adoption of a better model, the mixture of factor analyzers (MFA), which approximates the non-linear data manifold by a collection of local sub-spaces. The Fisher score with respect to the MFA (MFA-FS) is derived and proposed as an image representation for holistic image classifiers. Extensive experiments show that the MFA-FS has state of the art performance for object-to-scene transfer and this transfer actually outperforms the training of a scene CNN from a large scene dataset. The two representations are also shown to be complementary, in the sense that their combination outperforms each of the representations by itself. When combined, they produce a state-of-the-art scene classifier.

Review:
The paper claims that mixture of factor analyzers (MFA) is a better approximation compared to Fisher vector polling for transfer learning of high-order CNN features between object detection and scene classification tasks. This is supported by bunch of experiments that show improvements over other methods with considerable margins.
1. There is no theoretical justification of why MFA outperforms FV on transfer learning form object level to holistic scene descriptor. The main argument of the paper about "... inability of the standard GMM ... to provide good approximation ..." in L73-75 needs proof or reference to appropriate literature rather than only experiment results.  2. Authors claim "... MFA-FS captures highly discriminative covariance statistics ..." in L80-81, but I could not find any support of this except L161-162 which refers to full covariance S compared to Gaussian diagonal covariance. It needs to clarify why full covariance in MFA is the key to transfer learning problem on CNN features. I reckon it as a week argument although it was considered as second contribution of the paper because; any other dictionary learning method with full covariance should generate the same improvement as MFA according to authors' reasoning.  3. To my knowledge, section 2 of the paper seems a review of FV and MFA which does not clarify the novelty of authors' claimed contributions. An experienced reader is already aware of these formulations; hence it is expected to see the focus of formulation towards main claims which I could not see them there.  4. The paper seems like an experimental work which just plugs an idea to bunch of experiments to beat state-of-the-arts. I personally think that it may be of interest of computer vision community or venues like CVPR or WACV, but NIPS is a top-tier machine learning event which engineering tricks could not be that appreciated without sound theory and understanding in the background.  5. My other concern is about deployment of PCA with MFA-FS scores. Authors claim that MFA is employed due to its better ability to approximate nonlinear manifold of CNN. I expect that applying PCA to this high-order descriptor should degrade the accuracy much higher than what's reported in table 4, because PCA gets rid of many local sub-spaces with second and higher derivatives. Actually, it must diminish the advantage of MFA on factorizing to high-order sub-spaces, but the results remain almost the same i.e. 79.9 vs 79.3 on MIT. If the key contribution of the paper is a better description of high-dimensional and nonlinear CNN feature space by MFA, I’m wondering that it still remains competitive after a simple PCA.