Paper:
In this paper, we address the problem of reducing the memory footprint of convolutional network architectures. We introduce a vector quantization method that aims at preserving the quality of the reconstruction of the network outputs rather than its weights. The principle of our approach is that it minimizes the loss reconstruction error for in-domain inputs. Our method only requires a set of unlabelled data at quantization time and allows for efficient inference on CPU by using bytealigned codebooks to store the compressed weights. We validate our approach by quantizing a high performing ResNet-50 model to a memory size of 5 MB (20× compression factor) while preserving a top-1 accuracy of 76.1% on ImageNet object classification and by compressing a Mask R-CNN with a 26× factor.

Review:
The suggested method proposes a technique to compress neural networks bases on PQ quantization. The algorithm quantizes matrices of linear operations, and, by generalization, also works on convolutional networks. Rather than trying to compress weights (i.e. to minimize distance between original and quantized weights), the algorithm considers a distribution of unlabeled inputs and looks for such quantization which would affect output activations as little as possible over that distribution of data. The algorithm works by splitting each column of W_ij into m equal subvectors, learning a codebook for those subvectors, and encoding each of those subvectors as one of the words from the codebook.
The method provides impressive compression ratios (in the order of x20-30) but at the cost of a lower performance. Whether this is a valuable trade-off is highly application dependent.
Overall I find the paper interesting and enjoyable. However, as I am not an expert in the research area, I can not assess how state of the art the suggested method is.
There are a few other questions that I think would be nice to answer. I will try to describe them below:
Suppose we have a matric W_{ij} with dimensions NxM where changing i for a given j defines a column. By definition, linear operation is defined 
y_i = sum_j W_ij x_j . Now say each column of matrix W is quantized into m subvectors. We can express W_ij in the following way:
W_ij = (V^1_ij + V^2_ij + ... V^m_ij)x_j where V^m_ij is zero everywhere except for the rows covering a given quantized vector.
For example, if W had dimensions of 8x16 and m=4, 
V^2_{3,j}=0, for all j, V^2_{4,j}=non_zero, V^2_{7,j}=non_zero, V^2_{8,j}=0, V^2_{i=4:8,j}=one_of_the_quantized_vectors.
y_i = sum_j W_ij x_j = sum_k sum_j (V^k_ij) x_j =def= sum_k z^k_i where z^k are partial products: z^k_i=0 for i<k*N/m and i>(k+1)N/m
Thus, the suggested solution effectively splits the output vector y_i into m sections, defines sparse matrices V^k_{ij} 1<=k<=m, and performs column-wise vector quantization for these matrices separately.
Generally, it is not ovious or given that the current method would be able to compress general matrices well, as it implicitly assumes that weight W_{ij} has a high "correlation" with weights W_{i+kN/m,j} (which I call "vertical" correlation), W_{i,k+some_number} (which I call "horizontal" correlation) and W_{i+kN/m,k+some_number} (which I call "other" correlation). It is not given that those kind of redundancies would exist in arbitrary weight matrices.
Naturally, the method will work well when weight matrices have a lot of structure and then quantized vectors can be reused. Matrices can have either "horizontal" or "vertical" redundancy (or "other" or neither). It would be very interesting to see which kind of redundancy their method managed to caprture.
In the 'horizontal' case, it should work well when inputs have a lot of redundancy (say x_j' and x_j'' are highly correlated making it possible to reuse code-words horizontally within any given V^k: V^k_ij'=V^k_ij''). However, if thise was the case, it would make more sense to simply remove redundancy by prunning input vector x_j by removing either x_j' or x_j'' from it. This can be dome by removing one of the outputs from the previous layer. This can be a symptom of a redundant input.
Another option is exploiting "vertical" redundancy: this happens when output y_i' is correlated with output y_{i'+N/m}. This allows the same code-word to be reused vertically. This can be a symptom of a redundant output. It could also be the case that compressibility could be further subtantially improved by trying different matrix row permutations. Also, if one notices that y_i' ir correlated with y_i'', it might make sense to permute matrix rows in such a way that both rows would end up a multiple N/m apart. It would be interesting to see how this would affect compressibility.
The third case is when code words are reused in arbitrary cases. 
Generally, I think that answering the following questions would be interesting and could guide further research:
1. It would be very interesting to know what kind of code-word reusa patterns the algorithm was able to capture, as this may guide further research.
2. How invariance copressibility is under random permutations of matrix rows (thus also output vectors)?