Paper:
Predicting Parameters in Deep Learning
We demonstrate that there is significant redundancy in the parameterization of several deep learning models. Given only a few weight values for each feature it is possible to accurately predict the remaining values. Moreover, we show that not only can the parameter values be predicted, but many of them need not be learned at all. We train several different architectures by learning only a small number of weights and predicting the rest. In the best case we are able to predict more than 95% of the weights of a network without any drop in accuracy.

Review:
Motivated by recent attempts to learn very large 
      networks this work proposes an approach for reducing the number of free 
      parameters in neural-network type architectures. The method is based on 
      the intuition that there is typically strong redundancy in the learned 
      parameters (for instance, the first layer filters of of NNs applied to 
      images are smooth): The authors suggest to learn only a subset of the 
      parameter values and to then predicted the remaining ones through some 
      form of interpolation. The proposed approach is evaluated for several 
      architectures (MLP, convolutional NN, reconstruction-ICA) and different 
      vision datasets (MNIST, CIFAR, STL-10). The results suggest that in 
      general it is sufficient to learn fewer than 50% of the parameters without 
      any loss in performance (significantly fewer parameters seem sufficient 
      for MNIST). The paper is clear and easy to follow. The method is 
      relatively simple: The authors assume a low-rank decomposition of the 
      weight matrix and then further fix one of the two matrices using prior 
      knowledge about the data (e.g., in the vision case, exploiting the fact 
      that nearby pixels - and weights - tend to be correlated). This can be 
      interpreted as predicting the "unobserved" parameters from the subset of 
      learned filter weights via kernel ridge regression, where the kernel 
      captures prior knowledge about the topology / "smoothness" of the weights. 
      For the situation when such prior knowledge is not available the authors 
      describe a way to learn a suitable kernel from data. The idea of 
      reducing the number of parameters in NN-like architectures through 
      connectivity constraints in itself is of course not novel, and the authors 
      provide a pretty good discussion of related work in section 5. Their 
      method is very closely related to the idea of factorizing weight matrices 
      as is, for instance, commonly done for 3-way RBMs (e.g. ref [22] in the 
      paper), but also occasionally for standard RBMs (e.g. [R1], missing in the 
      paper). The present papers differs from these in that the authors propose 
      to exploit prior knowledge to constrain one of the matrices. As also 
      discussed by the authors, the approach can further be interpreted as a 
      particular type of pooling -- a strategy commonly employed in 
      convolutional neural networks. Another view of the proposed approach is 
      that the filters are represented as a linear combination of basis 
      functions (in the paper, the particular form of the basis functions is 
      determined by the choice of kernel). Such representations have been 
      explored in various forms and to various ends in the computer vision and 
      signal processing literature (see e.g. [R2,R3,R4,R5]). [R4,R5], for 
      instance, represent filters in terms of a linear combination of basis 
      functions that reduce the computational complexity of the filtering 
      process). In the experimental section the authors make an effort 
      to demonstrate that the proposed method is practically useful and widely 
      applicable, considering multiple datasets and several different 
      architectures. I think, however, that this section could have been 
      stronger in several ways: Firstly, it would have been nice if the 
      paper had not focused exclusively on vision applications, and had put 
      generally more emphasis on scenarios where there is a less obvious topolgy 
      in the data that can be exploited when predicting weights (which will pose 
      more of a challenge to the method). The data-dependent kernel is not very 
      well evaluated. Secondly, especially for the vision case, I am 
      wondering why the authors are limiting themselves to the particular set of 
      basis functions derived from the kernel regression view. It would seem 
      natural to explore other linear basis representations of the filters (a 
      simple choice would be a PCA basis, for instance). These might be more 
      efficient in terms of reducing the number of free parameters, and might 
      have other desirable properties (e.g. [R4,R5]). Finally, I would 
      have hoped for a truly compelling experimental use case demonstrating the 
      impact of the approach in practice. Since the work is motivated as a way 
      of reducing computational and memory complexity, I think it would have 
      been useful to include a more detailed discussion and empirical 
      demonstration how the reduction in number of learned parameters translates 
      into such savings and consequently allows the training of larger networks 
      that achieve better performance than otherwise possible. At the moment, 
      the evaluation appears to be focused on moderately large networks, and the 
      authors show that a moderate reduction in parameters can be achieved that 
      way, without loosing performance (for MNIST the potential reduction in 
      parameters seems to be very large, but for the more interesting CIFAR / 
      STL-10 datasets it seems that at least 40% of the parameters are required 
      to avoid a loss in accuracy). Why is computational complexity and speed of 
      convergence not considered at all in the evaluation? I think that Fig. 
      6-right (which plots performance against # of free parameters for a 
      standard network and a reduced parameterization) goes in the right 
      direction, but it seems that there is really only a clear advantage of the 
      reduced parameterization for CIFAR (much less so for STL), and I'm 
      wondering whether the performance difference on CIFAR would disappear even 
      for currently realistic network sizes. All in all, I think the 
      paper takes an interesting perspective although related approaches have 
      appeared in the literature in various guises (see discussion above). I 
      think that the ideas can have practical impact, and to my knowledge, they 
      are currently not widely used in the NN/deep learning community. A 
      stronger case could probably be made, however, by further exploring 
      alternative implementations of the idea and by having a more compelling 
      demonstration of the effectiveness and versatility of the approach. 
      Further remarks: ** It would have been really helpful to 
      include state of the art results from the literature for the datasets / 
      and specific evaluation regimes considered. This would put the reported 
      performance into perspective and make it easier to judge whether savings 
      in learned parameters can be achieved for competitive architectures. 
      ** I find the evaluation of the data-dependent kernel somewhat 
      limited: it is only explored for a single dataset / architecture 
      combination (the MNIST, MLP experiments in section 4.1). As mentioned 
      above, it would have been nice to inlcude some other type of data that 
      requires the use of the empirical kernel. For the experiments in 
      section 4.1 it would have further been useful to also consider the case 
      SE-rand. At the moment it's not clear that the learned kernel really 
      outperforms the random approaches. The difference might simply be due to 
      the fact that for the empirical kernel you're using the SE kernel in the 
      first layer. Another interesting control for the effectiveness of the 
      empirical kernel would be to have a setting where you're using the 
      empirical kernel in both layers. ** Do you have any insights as to 
      why the naive low-rank scheme, where both matrices are learned, is working 
      so poorly? Have you made any attempt to remove the redundancy inherent in 
      the parameterization? Would it be possible (and potentially advantageous) 
      to start of with a fixed U (e.g. the empirical kernel), but allow U to 
      change later at some point, possibly in an optimization scheme where U and 
      V are updated in an alternating manner? ** In your conv-net 
      experiments I suppose you are not using any pooling layers as described in 
      [18]? This would also reduce the dimensionality of the representation in 
      the higher layers. Furthermore, based on Fig. 5 I am wondering whether you 
      can really argue that predicting 75% of the parameters has negligible 
      impact on performance: when you're predicting 50% of the parameters the 
      loss seems to be about 5-6 percentage points which does not seem so 
      negligible to me (I guess it would help to have some errorbars here). 
      ** How did you optimize the hyperparameters (learning rate etc.) 
      for your comparisons? Shouldn't the kernel width vary with the size of the 
      subset of learned parameters alpha? ** In section 4.1, for the MLP 
      experiments, how do the columns differ? Do they use the same empirical 
      kernel and differ only in the set of sampled pixels? Is the great 
      advantage of using multiple columns here due to the fact that otherwise 
      you don't sample the space well? ** Fig. 6 right: I find it 
      curious that the reduced parameterization is advantageous for for CIFAR 
      but there seems to be very little difference for STL - given the higher 
      resolution of STL wouldn't one expect the opposite? ** For 
      conv-nets where, in the higher-layers, there are many input channels 
      relative to the spatial extent of each filter, would it be possible to use 
      a data-dependent kernel to subsample across channels (if I understand 
      correctly you are currently performing the subsampling only spatially, 
      i.e. the elements of alphas are tied across channels)? ** Is there 
      a way to extend your scheme such as to start off with a small fraction of 
      learned parameters, but to then increase the fraction of learned 
      parameters slowly until a desired performance is reached / no further 
      improvement is observed? References: [R1] Dahl 
      et al., 2012: Training Restricted Boltzmann Machines on Word Observations 
      [R2] Roth & Black, IJCV, 2009: Fields of Experts (Section 5.3) 
      [R3] Freeman & Adelson, 1991: The design and use of steerable 
      filters [R4] Rubinstein et al., 2010: Double Sparsity: Learning 
      Sparse Dictionaries for Sparse Signal Approximation [R5] 
      Rigamonti et al., 2013: Learning Separable Filters