From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper45/Authors
comment: Thanks for the reference suggestions. We will add them in the revision. 

-- Regarding computing pi(h) with pooling: (1) if h is 3D image tensor of shape (H, W, C), a global average pooling is used, which reduce the tensor to (1, 1, C); (2) if h is a 1D vector already (e.g. d dim), we can divide the vector into multiple parts and average each part to get d’ dim vector (d’ << d).

-- Regarding MIX-ALL-PJ: Yes, you are right, MIX-ALL-PJ is similar to MIX-RND-PJ but with learnable projection matrix P instead of a random fixed one.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper45/Authors
comment: Thanks for your questions. Indeed the current paper mainly focus the FLOPs, as the actual inference time depends on implementation, and also include runtime by other factors (such as final softmax layer in RNN language modeling). Nevertheless, we conducted some preliminary experiments with non-optimized implementation on RNN (without final softmax), and measure the actual inference time with CPU as shown in the table below. We observed that the proposed method still provides similar speedup as regular low-rank (while getting significantly better accuracy/perplexity).

-----------------------------------------------------------------
  Method      |      low-rank ratio; time in ms
-----------------------------------------------------------------
                      |    1   |  1/2  |  1/4  | 1/8 | 1/16 
-----------------------------------------------------------------
Full  Rank     | 10.8 | N/A  | N/A | N/A | N/A
regular LR    | N/A  | 13.1 | 6.6  | 3.3   | 2.0
adaptive LR  | N/A | 16.5  | 8.6  | 4.5   | 2.8
-----------------------------------------------------------------

We would like to improve our implementation of the adaptive low-rank to further speed up the actual inference time, and conduct more run-time comparisons in the future.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper45/Authors
comment: Thanks for your questions. Please find our answers listed below.

1. Yes, the weight matrix P is trained end-to-end according to the task, except in strategy MIX-RND-PJ for generating mixing weights pi(h), where P is a fixed random matrix.

2. We tested different K, and found that using more mixtures generally leads to better results, although the performance starts to plateau when the number of mixtures is large enough. However, to obtain a larger compression rate and speedup, the rank-d we use in the low-rank factorization can be already small, thus the extras of using different number of mixtures may not differ too much. For this reason, we can just set K as rank-d.

3. The perplexity (https://en.wikipedia.org/wiki/Perplexity) is a standard performance metric for language modeling [12], which is basically a exponentiated negative log-likelihood, and the lower the better. And thanks for pointing out the typo, it should be "LSTM's baseline perplexity" instead of "LSTM's baseline accuracy".

3+. We use MIX-POOL-PJ for the MobileNet. The performance improvement of the proposed adaptive low-rank, compared to the regular/standard low-rank, is likely due to the nonlinearity introduced by the mixing weights, which potentially increases the expressiveness of the model.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper45/Authors
comment: Thanks for your questions. We tested different K, and found that using more mixtures generally leads to better results, although the performance starts to plateau when the number of mixtures is large enough. However, to obtain a larger compression rate and speedup, the rank-d we use in the low-rank factorization can be already small, thus the extras of using different number of mixtures may not differ too much. For this reason, we can just set K as rank-d.

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: This work proposes to mix multiple input-based low-rank factorizations to model a large weight matrix and demonstrate its effectiveness on LSTM for language modeling and CNN for image classification. 

Advantage:
The paper organization and presentation is clear, and the experiment results are impressive. 

Weakness:
The authors merely present the FLOPs comparison. It is better to present the execution time in real GPU or CPU environments.

How to set K? How does it affect the results?

Rating: 3/5
Confidence: 2/3

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper45/AnonReviewer4
comment: Rating: 3/5

This paper proposes to use an adaptive, input-dependent mixture of multiple low-rank factorizations instead of one fixed low-rank factorization to approximate the weight matrix. By introducing new input-dependent parameters -- "mixing weights", the proposed method is able to increase the expressiveness and performance of the compressed model. The authors further provide three strategies to form the projection function which maps each input to its corresponding mixing weights, including direct linear projection, pooling before projection, and random projection. Experiments on LSTM models and MobileNet show that the proposed method achieves the goal of reducing the computation and parameter size while preserving performance. 
This method is novel in the sense that it targets improving expressiveness of low-rank factorization. The paper is clear and easy to follow. Maybe because of lack of space, the authors don’t provide detail information on the choice of multiple low-rank factorizations. Overall I think this paper is worthy of acceptance. 

Questions: 
1. For the direct linear projection, the projection function can be seen as a small neural network. How to get the extra weight matrix P? Does it need retraining? 
2. The authors talk little about the design of multiple low-rank components U(k) and V(k) and the number of components k. Could you elaborate it more? 
3. What's the meaning of "perplexity" in Figure 2? The caption of Figure 2 uses "LSTM's baseline accuracy". I think there should be "perplexity". In addition, which projection strategy is used to compress MobileNet? The accuracy of adaptive low-rank MobileNet is better than that of low-rank Mobilenet. Could you please give some explanations?

Confidence: 2/3

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper45/AnonReviewer3
title: Review Comments
review: This paper proposes a CNN compression/acceleration method based-on a learned adaptive mixture of  low-rank decompositions to improve the expressiveness of decomposed network at little expense of FLOPs.  Experiments show encouraging results over regular low-rank decomposition.

Pros:
- Clear statement
- Good experiment results

Cons:
- Comparison is only explored in terms of FLOPs reduction. How about the actual inference time  reduction (on both CPU and GPU) for Figure 2 and Table 1?  Since the introduction of more matrices may lead to I/O bound for some layers, despite the reduction of FLOPs, the actual speedup can be insignificant.
rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper45/AnonReviewer1
comment: Rating: 3 of 5

Summary:
This paper presents a mixture of low-rank factorization to model a large weight matrix in deep neural networks. The main novelty of the proposed framework is that the mixture weights of adaptive to the input, using several computational efficient ways to induce the non-linear mixture weights.

Pros:
- Clear presentation of the paper
- good result, computationally efficient and higher expressive power

Cons:
- lack of related works in a “mixture of expert” papers such as [1,2], to name just a few. 

Questions:
- In line 53~57, it is still not clear to me how pooling to the input h results in the mixture weights pi(h). Could you elaborate more?
-  In line 76, what is the linear projection in MIX-ALL-PJ? Is this implying that the P is learned instead of using some random projection matrix?


Confidence:  3 of 3

[1] Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, ICLR 2017
[2] Breaking the softmax bottleneck: A High-rank RNN Language Model, ICLR 2018