Paper:
Recurrent Neural Networks (RNNs) are used in state-of-the-art models in domains such as speech recognition, machine translation, and language modelling. Sparsity is a technique to reduce compute and memory requirements of deep learning models. Sparse RNNs are easier to deploy on devices and high-end server processors. Even though sparse operations need less compute and memory relative to their dense counterparts, the speed-up observed by using sparse operations is less than expected on different hardware platforms. In order to address this issue, we investigate two different approaches to induce block sparsity in RNNs: pruning blocks of weights in a layer and using group lasso regularization with pruning to create blocks of weights with zeros. Using these techniques, we can create block-sparse RNNs with sparsity ranging from 80% to 90% with a small loss in accuracy. This technique allows us to reduce the model size by roughly 10×. Additionally, we can prune a larger dense network to recover this loss in accuracy while maintaining high block sparsity and reducing the overall parameter count. Our technique works with a variety of block sizes up to 32×32. Block-sparse RNNs eliminate overheads related to data storage and irregular memory accesses while increasing hardware efficiency compared to unstructured sparsity.

Review:
Thanks to the authors for their response.
Though the paper presents an interesting approach, but it relies heavily on heuristics (such as those mentioned in the initial review) without a thorough investigation of scenarios in which this might not work. Also, it might be helpful to investigate if there ways to better group the variables for group  lasso regularization. The paper therefore needs further improvements towards following a more principled approach.
=====================================
This paper presents methods for inducing sparsity in terms of blocks of weights in neural networks which aims to combine benefits of sparsity and faster access based on computing architectures. This is achieved by pruning blocks of weights and group lasso regularization. It is demonstrated empirically that model size can be reduced by upto 10 times with some loss in prediction accuracy.
Though the paper presents some interesting evaluations on the impact of block based sparsity in RNNs, some of the shortcomings of the paper seem to be :
- The approach taken consists of several heuristics rather than following a more principled approach such as taking the maximum of the weights in a block to represent that block and stop pruning till 40% training has been achieved. Also, the algorithm for computing the pruning threshold is based on a new set of hyper-parameters. It is not clear under what conditions the above settings will (not) work.
 - For the group lasso method, since there are many ways to group the variable, it is not clear how the variables are grouped. Is there a reasoning behind a particular grouping of the variables. Individually, group lasso does not seem to work, and gives much worse results. The reasons for worse performance could be investigated. It is possible that important weights are in different groups, and group sparsity is forcing some of them to be zero, and hence leading to worse results. It would be insightful to explain the kind of solver used for group lasso regularization, and if that works for large-scale problems.
 - The results for various kinds of sparsity are unclear in the sense that it is not clear how to set the block size a-priori for having minimum reduction in accuracy and still significant sparsity without having to repeat the process for various choices.
Overall, the paper does not seem to present novel ideas, and is mainly focused on evaluating the impact of block-based sparsity instead of weight pruning by Han etal. As mentioned in Section 2, regularization has been used earlier to achieve sparsity in deep networks. In this view the significance over existing work is relatively narrow, and no explicit comparison with existing methods is provided. It is possible that an existing method leads to pruning method such as by Han etal. leads to 8x decrease in model size while retaining the accuracy, while the proposed method leads to 10x decrease while also decreasing the accuracy by 10%. Scenarios like these need to be evaluated to understand the impact of the method proposed in this paper.