Paper:
Convolutional Neural Networks (CNN) have achieved state-of-the-art performance in different computer vision tasks, but at a price of being computationally and power intensive. At the same time, only a few attempts were made toward a deeper understanding of CNNs. In this work, we propose to use semantic pruning technique toward not only CNN optimization but also as a way toward getting some insight information on convolutional filters correlation and interference. We start with a pre-trained network and prune it until it behaves as a single class classifier for a selected class. Unlike the more traditional approaches which apply retraining to the pruned CNN, the proposed semantic pruning does not use retraining. Conducted experiments showed that a) for each class there is a pruning ration which allows removing filters with either an increase or no loss of classification accuracy, b) pruning can improve the interference between filters used for classification of different classes c) effect between classification accuracy and correlation between pruned filters groups specific for different classes.

Review:
Summary
---
(motivation)
This paper proposes a new approach to pruning activations from neural networks,
but uses it to understand neural nets better rather than trying to make them
more efficient. It prunes filters per-class 1) to measure how sensitive image
classes are to pruning and 2) to measure how similar classes are by comparing
the filters they prune.
(approach)
Using only images from class c, CNN pre-activations are aggregated across spatial
dimensions and examples. This gives an average feature vector for that class.
T percent of neurons are pruned. Features with lower pre-activations
(possibly negative with large magnitude) are pruned first with successively
higher activations pruned later.
No re-training is performed.
(experiments)
Experiments use AlexNet and 50 of the 1000 ImageNet challenge classes to show:
1. Pruning filters results in decreased accuracy with smaller decreases in accuracy as the first and last few filters are pruned.
2. More filters from middle layers are pruned than are those from early and late layers.
3. After some filters have been pruned, sometimes pruning can increase accuracy.
4. I could not understand section 3.2.
Strengths
---
The proposed pruning method is simple and efficient.
I like the broad goal of understanding CNNs. We could use more papers that just focus on analysis like this one.
I think the idea of class conditional pruning is novel.
Weaknesses
---
# Major Weaknesses
I don't see why these results are significant. I do not find these results very surprising (see next comment) and I do not see why the community will find them useful.
* In particular, consider the conclusion. Sentences 2, 3, 5, and 6 seem to all be observations about what happened in the experiments. The conclusion should re-iterate the results, but it should also say why they are important. How does the work relate to the goals of the community at large? Which goals? Will this enable important new capabilities? What general concepts did we learn from this that we didn't know before?
Some of the positions in the paper could be more carefully considered. There are alternate explanations for many of these phenomena.
* Pruning smallest activations doesn't make sense to me. Typically redundant or un"important" activations are pruned because doing so has negligible impact on accuracy. The smallest activations are pruned here. Should the smallest activations be redundant or unimportant in some way? Does this rely on ReLU activations following the pre-activations? I think these values are not necessarily redundant and could be quite important (e.g., as measured by some saliency explanation like Integrated Gradients), but I could be wrong. It would be useful to provide a baseline which removes highest instead of lowest activations.
* Much of the surprise about figure 5 seems to be because it is not monotonic but it was expected to be monotonic. I agree that these curves should generally go down as theta increases, but I don't see why that relationship should be strict. I would be surprised if activations were not in some way dependent on one another. Furthermore, why does that dependence have to be interference? Couldn't it also be that some activations are complementary (and thus ineffective when only one is present)?
* Does Network Wise Pruning (NWP) favor more layers than others? It may be that some filters have higher Accumulated Responses per filter simply because there are more feature maps in the previous layer (thus more things summed up) and not because of what information they capture or their relationships with other filters. Does this happen? This could be an alternative to the following conclusion: "This means that the encoding provided by the first and last layer seems to be the most crucial and the densest."
# Other General Weaknesses:
* AlexNet is a rather old architecture to use for this analysis, so I can't be confident these methods or behaviors will generalize to other architectures. Does it hold for more modern architectures?
# Missing details / Points of confusion:
* The paper says the correlation from Figure 6 should be expected to decrease as theta increases. Why should this be expected?
* What exactly does Figure 6 measure? I think it's the ratio of the size of the intersection rho_sigma to the size of the union of the same two sets. However, the text calls the metric "correlation." This should be made clearer.
# Minor presentation weaknesses
Some parts of the notation/explanation don't make sense to me:
* "Let Lambda=... be the number of object classes in the dataset." But Lambda is defined as a set, not a numeral.
* The number p needs more context/subscripts as it depends on class c and filter i.
* "θ = [0, . . . , 1]" This defines theta as a finite set, but I think it's meant to be a number in the interval range (0, 1).
* Why is lambda_c needed instead of just using the index c to specify a class? The variable lambda doesn't seem to be any different than a class index.
* "let Fσ and Fσ ̄ be the set of unpruned and pruned filters for a given γ..." I don't see how this works. According to eq. 3 gamma depends on a particular filter i and class c, so the only filter F could contain is the ith one. (Later it became clear that F was only mean to class conditional, not filter conditional.)
Figure 5: These plots shoul all have the same y axis range. This would make them comparable and allow readers to much more easily compare trends across classes. Similar steps should be taken so the same range is used any time theta is plotted on the x axis.
Figure 3: I find it hard to get an overall ordering of the approaches in this figure because there is so much variance from class to class. It effectively conveys the variance, but I'd also like to know what the means across classes are for each method so I can compare the proposed approaches more effectively.
"The results indicate that classification classes are asymmetrically represented by filters resulting in the fact that some object classes have their classification accuracy increased when pruned for."
* I'm not sure what it means to be asymmetrically represented by filters.
Suggestions
---
This analysis would have been more interesting with an existing pruning approach because we would already know that such an approach is good a removing unnecessary filters.
Final Evaluation
---
Quality: Experiments could have been cleaner, but they basically demonstrate the patterns the paper intended to show.
Clarity: I could understand most experiments at a high level, but I found it hard to understand the motivation and the rest of the experiments.
Significance: As explained above, I do not see why the paper is significant.
Originality: The experiments and the proposed class conditional pruning approach are somewhat novel.
The paper is somewhat novel, but do not find it very clear and I do not see why it is important, so I cannot reccomend it for acceptance.