Paper:
Encoding High Dimensional Local Features by Sparse Coding Based Fisher Vectors
Deriving from the gradient vector of a generative model of local features, Fisher vector coding (FVC) has been identified as an effective coding method for image classification. Most, if not all, FVC implementations employ the Gaussian mixture model (GMM) to characterize the generation process of local features. This choice has shown to be sufficient for traditional low dimensional local features, e.g., SIFT; and typically, good performance can be achieved with only a few hundred Gaussian distributions. However, the same number of Gaussians is insufficient to model the feature space spanned by higher dimensional local features, which have become popular recently. In order to improve the modeling capacity for high dimensional features, it turns out to be inefficient and computationally impractical to simply increase the number of Gaussians. In this paper, we propose a model in which each local feature is drawn from a Gaussian distribution whose mean vector is sampled from a subspace. With certain approximation, this model can be converted to a sparse coding procedure and the learning/inference problems can be readily solved by standard sparse coding methods. By calculating the gradient vector of the proposed model, we derive a new fisher vector encoding strategy, termed Sparse Coding based Fisher Vector Coding (SCFVC). Moreover, we adopt the recently developed Deep Convolutional Neural Network (CNN) descriptor as a high dimensional local feature and implement image classification with the proposed SCFVC. Our experimental evaluations demonstrate that our method not only significantly outperforms the traditional GMM based Fisher vector encoding but also achieves the state-ofthe-art performance in generic object recognition, indoor scene, and fine-grained image classification problems.

Review:
This paper proposes a method for fisher vector coding (FVC) of high dimensional local features. Most existing approaches that use FVC assume that local features are drawn from a Gaussian Mixture Model (GMM) where the number of mixture components is specified a priori. This paper argues that while this has worked well for FVC of low-dimensional local features like SIFT, it does not scale to high dimensional features because the number of Gaussians required to model the distribution of local features becomes prohibitive. Instead of using a GMM with a finite number of mixture components, this paper proposes to approximate an infinite Gaussian mixture using a set of basis vectors. Under the assumption of iid feature dimensions, the basis vectors and feature encoding can be learned using standard sparse coding. The paper provides derivation of the corresponding fisher vector, which can be computed analytically.The paper is well-written. There are minor typos and grammatical errors throughout the paper, but they do not significantly impact comprehensibility. The proposed approach is technically sound and appears to yield good results empirically. One limitation of the approach that the paper glosses over is the assumption of a diagonal covariance matrix with constant sigma in deriving the log-likelihood in eq. 4. This is an important theoretical limitation, as GMMs can model correlation between feature dimensions, while the sparse coding based approach proposed in this paper does not. The authors should mention this limitation, and explicitly state that using PCA to first transform the local features is a practical method to get around this problem.Experiments demonstrate that the proposed approach outperforms the use of GMM-based FVC and achieves state-of-the-art performance on several image classification, scene classification, and fine-grained image classification datasets. The paper investigates the tradeoff between increasing the number of mixture components versus using higher dimensional local features for GMM-based FVC. However, it would be even more informative if the paper could also provide a similar investigation on the tradeoff between the number of basis vectors and the dimensionality of local features for the proposed sparse coding FVC.Overall, the paper contains good algorithmic novelty and demonstrates convincing and extensive empirical results. Hence, I would recommend that it be accepted for publication if the authors could provide an analysis of the tradeoff between the number of basis vectors and the dimensionality of local features for the proposed sparse coding FVC.Minor Questions:- For Pascal VOC 2007 results, there is a huge performance difference between FVC-based methods and CNN-SVM for bottle (in CNN-SVM's favor) and bus (in FVC's favor). Have you looked into the cause of these differences?Typos:- line 151: By "assuming" ...- line 240: ... coding value u_k "multiplied by" the reconstruction residue ...