Paper:
Traditional compression methods including network pruning, quantization, low rank factorization and knowledge distillation all assume that network architectures and parameters are one-to-one mapped. In this work, we propose a new perspective on network compression, i.e., network parameters can be disentangled from the architectures. From this viewpoint, we present the Neural Epitome Search (NES), a new neural network compression approach that learns to find compact yet expressive epitomes for weight parameters of a specified network architecture end-to-end. The complete network to compress can be generated from the learned epitome via a novel transformation method that adaptively transforms the epitomes to match weight shapes of the given architecture. Compared with existing compression methods, NES allows the weight tensors to be independent of the architecture design and hence can achieve a good trade-off between model compression rate and performance given a specific model size constraint. Experiments demonstrate that, on ImageNet, when taking MobileNetV2 as backbone, our approach improves the full-model baseline by 1.47% in top-1 accuracy with 25% MAdd reduction, and with the same compression ratio, improves AutoML for Model Compression (AMC) by 2.5% in top-1 accuracy. Moreover, taking EfficientNet-B0 as baseline, our NES yields an improvement of 1.2% but has 10% less MAdd. In particular, our method achieves a new state-of-the-art results of 77.5% under mobile settings (<350M MAdd). Code will be made publicly available.

Review:
In this work, the authors describe a technique for compressing neural networks by learning a so-called Epitome (E), and a transformation function (\theta) such that the weights for each layer can be constructed using \theta(E). The epitome and the transformation function can be learnt jointly while optimizing the network for the task specific loss. 
The main idea of this paper is really interesting, and the experimental results which compare against other recent techniques also validate the proposed technique. However, while I think the main idea is relatively clear, I personally found the description of the proposed techniques -- particularly 3.2 and 3.4 -- to be somewhat hard to follow, particularly given that some details only appear in the Appendix. I would suggest that the authors try to revise this section by trying to move Figures 4 and 5 from the appendix into the main text. Some additional suggestions also appear below.
Overall, I would while I like the ideas in this paper, based on the current presentation I am inclined to rate the paper as a “weak reject”, though I would raise my rating if the paper was revised to improve the presentation.
Main comments:
1. The authors mention that “During inference, (the) routing map enables the model to reuse computations when the expanded weight tensors are formed based on the same set of elements in the epitomes and therefore effectively reduces the computation cost.” It would be nice to include some results which indicate what the savings are with the proposed routing map.
2. Section 3.2: There were a few aspects of section 3.2 that I think could be improved for clarity.
A.) Personally, I found Figure 2 somewhat tricky to follow. I would suggest removing the 3x2 “Epitome” and “Generated Kernel” figures on the extreme right of the image, since I’m assuming the only goal of these is to indicate that “orange” and “blue” correspond to “Epitome” and “Kernel” respectively. I would also suggest mentioning the correspondence of the colors as the first sentence in the caption. Finally, if possible, I would suggest adding a small description alongside the (a), (b), (c) subcaptions: e.g., (a) straightforward but non-differentiable mapping, … , (c) Generated Weigh Kernel.
B.) I believe that the authors use a separate E for each layer, and that Epitomes are not shared across layers. I may have missed this in the text, but it would be useful to clarify this explicitly again in the section.
C.) The “parameterized transformation layer” is mentioned before Equation 3. I think it would be useful to mention that this is implemented using neural networks in your work for clarity. E.g.: “To handle the above two obstacles, ... three
parts: (1) a parameterized transformation learner η (implemented using a neural network in this work) used to learn a set of starting indices for patches in epitome (i.e. all elements in the same epitome patch share identical starting indices); … and an interpolation based generator (Eqn. 3).”
or
“To handle the above two obstacles, ... three
parts: (1) a parameterized transformation learner η (See Section 3.3) used to learn a set of starting indices for patches in epitome (i.e. all elements in the same epitome patch share identical starting indices); … and an interpolation based generator (Eqn. 3).”
D.) The exact structure of the “parameterized transformation layer” wasn’t exactly clear to me. In 3.3, the authors mention that it consists of “of two convolutional layer, followed by a sigmoid function … takes the feature map of the convolutional layer as input”. Please clarify exactly what is fed in as the input to this network e.g., (input feature map: F, and the indices i,j).
3. I personally also found Section 3.4 which discusses the Computation reduction was also somewhat hard to follow. Some clarification questions: Is the memory/computation cost of the storing/creating the routing map included in the compression calculations? I think it is important to factor these costs when computing the savings achieved by the model. Also, I was unclear on what R_{cin} and R_{cout} are, and why they appear in Equation 5. Could the authors please clarify. 
Minor Comments:
1. Abstract: “Traditional compression methods … all assume that network architectures
and parameters should be hardwired.” What does it mean for them to be “hardwired” in this context?
2. Abstract: “Experiments demonstrate that, … with 25% MAdd reduction and AutoML for Model Compression (AMC) by 2.5% with nearly the same compression ratio.” --> “Experiments demonstrate that, … with 25% MAdd reduction, and a 2.5% Madd reduction for AutoML for Model Compression (AMC) with nearly the same compression ratio.”
------- Update after Author Response --------
I would like to thank the authors for their responses and for the updates which strengthen the paper in my view. I have updated my score accordingly.