Paper:
Modern deep neural networks have a large amount of weights, which make them difficult to deploy on computation constrained devices such as mobile phones. One common approach to reduce the model size and computational cost is to use lowrank factorization to approximate a weight matrix. However, performing standard low-rank factorization with a small rank can hurt the model expressiveness and significantly decrease the performance. In this work, we propose to use a mixture of multiple low-rank factorizations to model a large weight matrix, and the mixture coefficients are computed dynamically depending on its input. We demonstrate the effectiveness of the proposed approach on both language modeling and image classification tasks. Experiments show that our method not only improves the computation efficiency but also maintains (sometimes outperforms) its accuracy compared with the full-rank counterparts.

Review:
The paper proposes an input-dependent low rank approximations of weight matrices in neural nets with a goal of accelerating inference. Instead of decomposing a matrix (a layer) to a low rank space like previous work did, the paper proposes to use a linear combination/mixture of multiple low rank spaces. The linear combination is dynamic and data dependent, making it some sense of non-linear combination. The paper is interesting, however,
I doubt its significance at three aspects:
(1) computation efficiency: the primary motivation of this paper is to accelerate inference stage; however, it might not be wise to break computation in a single low-rank space to segments in multiple low-rank spaces. In the original low-rank approximation, only two matrix-vector multiplications are needed, but this paper increases it to 2*K plus some additional overheads. Although the theoretical FLOPs can be cut, but modern hardware runs much faster when a whole bulk of data are computed all together. Because of this, the primary motivation in this paper wasn't successfully supported by wall-clock time;
(2) low-rank approximation: low-rank approximation only makes sense when matrices are known and redundant, otherwise, no approximation target exists (i.e., what matrix is it approximating?). Because of this, low-rank neural nets [1][2] start from trained models, approximate it and fine-tune it, while this method trains from scratch without an approximation target. Although, we can fit the method to approximate trained matrices, then decomposing a matrix to a mixture of low-rank spaces is equivalent to decomposing to one single low-rank space (the only difference is the combination is data dependent). Therefore, I view this paper more in a research line of designing compact neural nets, which brings me to a concern in (3).
(3) efficient architecture design: essentially, the paper proposes a class of compact neural nets, at each layer of which there are K "low-rank" branches with a gating mechanism to select those branches. However, branching and gating is not new [3][4]. I like the results in Table 2, but, to show the efficiency, it should have been compared with more SOTA compact models like CondenseNet. 
 
Clarity:
How FLOPs reduction are exactly calculated? I am not convinced by FLOPs reduction in the LSTM experiments, since in LSTM (especially in language modeling), FLOPs in the output layer are large because of a large vocabulary size (650x10000 in the experiments). However, output layer is not explicitly mentioned in the paper.
Improvement:
(1) Accuracy improvement in Table 1 is not statistically significant, but used more parameters. For example, an improvement of 93.01% over 92.92% is within an effect of training noise;
(2) It is a little hacking to conclude that a random matrix  P_random has a small storage size because we can storage a seed for recovery. When we deploy models across platforms, we cannot guarantee they use the same random generator and has a consistent implementation;
(3) sparse gating of low-rank branches may make this method more computation efficient.
[1] E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Information Processing Systems (NIPS). 2014.
[2] M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural networks with low rank expansions. In Proceedings of the British Machine Vision Conference (BMVC), 2014.
[3] Szegedy, Christian, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. "Going deeper with convolutions." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9. 2015.
[4] Huang, Gao, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q. Weinberger. "Multi-scale dense networks for resource efficient image classification." arXiv preprint arXiv:1703.09844 (2017).