Paper:
Learning rich and compact representations is an open topic in many fields such as word embedding, visual question-answering, object recognition or image retrieval. Although deep neural networks (convolutional or not) have made a major breakthrough during the last few years by providing hierarchical, semantic and abstract representations for all of these tasks, these representations are not necessary as rich as needed nor as compact as expected. Models using higher order statistics, such as bilinear pooling, provide richer representations at the cost of higher dimensional features. Factorization schemes have been proposed but without being able to reach the original compactness of first order models, or at a heavy loss in performances. This paper addresses these two points by extending factorization schemes to codebook strategies, allowing compact representations with the same dimensionality as first order representations, but with second order performances. Moreover, we extend this framework with a joint codebook and factorization scheme, granting a reduction both in terms of parameters and computation cost. This formulation leads to state-of-the-art results and compact second-order models with few additional parameters and intermediate representations with a dimension similar to that of first-order statistics.

Review:
Summary:
This paper proposes a novel bilinear representation based on a codebook model.
The proposed method is build upon the form of Hadamard Product for efficient representation of a bilinear model.
Through introducing the framework of codebook, it is naturally extended into a multiple-rank representation while the efficient pooling scheme is also derived from the (sparse) codeword assignment.
In addition, the authors also present an efficient formulation in which the codebook-based projections are factorized via a shared projection to further reduce the parameter size.
The experimental results on image retrieval tasks show that the proposed method produces better classification accuracy with a limited amount of parameters.
Comments:
Pros:
+ It is interesting that one-rank bilinear pooling is naturally extended to multiple-rank one via introducing codebooks.
+ Good performance in the image retrieval tasks.
Cons:
- This paper lacks important comparison for fairly evaluating the effectiveness of the proposed formulation.
- It also lacks detailed description and discussion for the methods.
Due to the above-mentioned weak points, the reviewer cannot fully understand whether the performance improvement really comes from the proposed formulation or not. Thus, this manuscript is currently judged as border. The detailed comments are shown in the followings.
- Comparison
Eventually, the proposed method is closely related to the multiple-rank representation of a bilinear model;
z_i = x^T W_i x (Eq.5) ~ x^T u_i v_i^T x (one-rank, Eq.6) ~ x^T U_i V_i^T x (multiple-rank), ... Eq.(A)
which is a straightforward extension from the one-rank model. From this viewpoint, the proposed form in Eq.10 is regarded as an extension of (A) by introducing non-linearity as
z_i = x^T U_i {h(x)h(x)^T} V_i^T x.  ... Eq.(10)
Thus, the main technical contribution is found in the weighting by {h(x)h(x)^T}, but its impact on the performance is not evaluated in the experiments. Furthermore, it is also possible to simply introduce such a non-linearity into the model (A) according to [Kim et al.,2017];
z_i = \sigma(x^T U_i) \sigma(V_i^T x) = 1^T {\sigma(U_i^T x) .* \sigma(V_i^T x)}, ... Eq.(B)
where ".*" indicates Hadamard Product, and we can more directly apply the method of [Kim et al., 2017] to the multiple-rank model by
z_i = p^T {\sigma(U_i^T x) .* \sigma(V_i^T x)}, ... Eq.(C)
where p is a R-dimensional vector. On the other hand, it is also necessary to compare the proposed method with [Kim et al.,2017] which is formulated by
z = P^T {\sigma(U^T x) .* \sigma(V^T x)}, ... Eq.(D)
where U and V are matrices of d x K and P is K x D. The parameter K (shared rank) should be determined so that the total parameter size of (2dK + KD) is compatible to that of the proposed method, 2NdD.
In summary, for demonstrating the effectiveness of the proposed method in Eq.(10), it is inevitable to compare it with the models (A, B, D) and hopefully (C).
- Presentation
In Section 4.2, the performance results of the factorization model in Eq.(13) are merely shown without deep discussion nor analysis on them. In particular, it is unclear why the JCF of N=32 and R=32 outperforms the CHPBP of N=32. Those two methods are different only in the form of U and V:
(CHPBP) U_i -> U'_i A (JCF),
where U_i and U'_i have the same dimensionality of d x 32, and thus we can say that JCF overly parameterizes the projection by redundantly introducing A of 32 x 32. Thus, the projection capacity of JCF is completely the same as that of CHPBP. Therefore, it needs detailed discussion for the performance improvement shown in Figure 1.
Minor comments are:
* There are lots of notations, and thus it would be better to show a summary of the notations.
* In Section 4.1, there is no clear description about the methods of BP and HPBP. Actually, the method of HPBP is different from the one presented in [Kim et al., 2017].