From: ICLR.cc/2023/Conference/Paper4987/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper4987/Reviewer_Sfxd
title: Following up
comment: I appreciate the reviewers double-checking their work based on my feedback. I am not seeing how the input (0,0,0,0,0) would lead to an output of 10, although that is tangent to the discussion about the paper at this point. The algorithm that you proposed seemed "too good to be true", which is the reason why I focused my entire review on finding a counter-example to illustrate that. I am glad that you came back to its design to understand that the assumptions would need to be a little stronger to make it work.

I have not verified if your algorithm under updated assumptions is correct, but assuming that to be true we would still have some roadblocks here:

1) The case in which your algorithm works is very specific and not that useful in practice. You could perhaps evaluate the extent to which your algorithm, or some heuristic based on it, would nevertheless produce better outcomes than it is currently possible with other methods. That lead us to the next point.

2) Your experimental section is very small and not very informative, as pointed out by other reviewers as well. Even under the assumption that you had found an algorithm that produces optimal solutions for the broader class of networks as you have previously thought, I would have expected a much more thorough benchmark than you presented here. For example, compared to other methods that you mentioned in the literature review, is your method faster? How much better are its results?

I believe that the authors would find it helpful to dive a little bit in the discrete optimization literature (applications using integer programming solvers and heuristics, for example) to see how you could sell your method when it is optimal and build on it when it is not to show how useful it can be.

From: ICLR.cc/2023/Conference/Paper4987/Authors
title: Response to Reviewer Sfxd
comment: We thank reviewer Sfxd for his careful analysis. Their review has been incredibly helpful. 

The counterexample given by the reviewer does not quite work: it would seem that (0,0,0,0,0) is not below the threshold for the given architecture (it yields an output of 10), and nor is (1,1,1,0,0) as it yields an output of 1.2; however, the ideas behind the proposed counter-example were interesting and we were able to exploit them to construct a correct counterexample ourselves.  

Using the constructed counter-example we have identified the incorrect step in our proof: in Equation 3 of Lemma 1, it is not possible to factorize by int_0^1 g_m(p(t))dt, as the weight in front of it differs for each m. Therefore, when p(t) changes, the order can be also be changed, which was the key argument of the proof of Theorem 2. This mistake was made because there is indeed a conserved quantity, but one needs to assume that all the weights are positive for it to hold true. The proofs then work similarly to what we intended.  

Consequently, the greedy algorithm is correct and the proof works under the additional assumption that the network implements a monotonic function (non-negative matrix weights + monotonic activations). 

Furthermore, by considering the point made by Reviewer Sfxd, we were able to strengthen the intractability result to show that the minimal feature removal problem for ReLU networks with arbitrary (positive and negative) weights is already NP-hard. We have thus written a new version of the paper to account for these changes. 

Although monotonic networks do not generally achieve good performance for image analysis, there are still relevant to a number of interesting tasks where the underpinning function is monotonic.  Several applications have been described in the literature on monotonic networks, and more recently applications in new settings such as knowledge graph completion have also been described. In order to validate our approach in practice,  however, we will need to re-run our experiments on a setting other than image analysis. 

We understand that this was a technical bug in our proof and that the changes made to the paper are significant. We would appreciate it if the reviewers could evaluate the revised version (see the rebuttal revision) with the correct theory but no experiments, but we would also be more than happy to withdraw our paper.

From: ICLR.cc/2023/Conference/Paper4987/Reviewer_Sfxd
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper discusses the problem of obtaining a minimum feature removal set of a neural network. The idea is that you start with an an input belonging to some class, which is characterized as having an output above a certain threshold, and an input not belonging to that same class, which is characterized as having an output below that same threshold. The goal is to find a minimum set of inputs which, if changed from the first input to the second, would lead the input to be classified as not belonging to the said class.

The authors present a proof, which seems correct, about this problem being NP-hard in general.

The authors also present a greedy algorithm which they claim to efficiently find the minimum removal set when the activation functions are monotonic. If correct, that algorithm would indicate that this problem is actually easy to solve in the settings that really matter in practice. However, I am not convinced that this algorithm is correct.
strength_and_weaknesses: Consider a neural network with 5 inputs, 3 ReLUs in the first layer, and 1 ReLU in the second layer producing the output.

First layer with the following ReLUs:
- A with weights 0.5, 0.4, 0.3, 0.2, 0.1 and bias 0
- B with weights -1, -1, 0, 0, 0 and bias 1
- C with weights 0, 0, 0, -1, -1 and bias 0

Second layer with the following ReLUs:
- D with weights 1, 10, -10 and bias 0

Threshold:
- Consider an output of D of at least 0.5

Inputs:
- Above threshold: (1,1,1,1,1) 
- Below threshold: (0,0,0,0,0)

Note that (1,1,1,1,1) evaluates to 1.5 because that is the output of A while B and C are not activated. Since making one input alone equal to zero does not active B or C, the variations below are all due to the weights in A and are as follows:
- (0,1,1,1,1) drops by 0.5
- (1,0,1,1,1) drops by 0.4
- (1,1,0,1,1) drops by 0.3
- (1,1,1,0,1) drops by 0.2
- (1,1,1,1,0) drops by 0.1

Following that reasoning, with the algorithm proposed by the authors we should set the first variable to 0, then the second variable to 0, and so on until we are below the threshold, but here is what we would get:
- The output for (0,0,1,1,1) is 10.6
- The output for (0,0,0,1,1) is 10.3
- The output for (0,0,0,0,1) is 10.1
- The output for (0,0,0,0,0) is 0

Hence, we conclude that all inputs should be changed in order to get an output below the specific threshold. However, note that (1,1,1,0,0) would evaluate to 0. Therefore, this would not be the minimum (or minimal) set.

The trick in the counterexample is that having the first two inputs set to zero triggers a neuron that adds a positive and very large contribution to the output. When the last two inputs are set to zero, a third neuron cancels the effect of the second neuron. 
clarity,_quality,_novelty_and_reproducibility: The fact that each activation function is monotonic does not imply that the neural network would model a monotonic function as well, which seems to be the reasoning of the authors based on the wording used along the paper. 

I would also caution the authors against using minimal as a synonym to minimum. If you want the smallest set that satisfies a certain property, that set is minimum. If you want a set satisfying a certain property such that not subset of it satisfies the same property, that set is minimal. 
summary_of_the_review: The algorithm presented does not produce the intended result. This paper should be rejected.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 1: strong reject

From: ICLR.cc/2023/Conference/Paper4987/Reviewer_1YgT
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The authors propose and analyze a strategy for finding a minimal feature removal set.  Here, feature removal corresponds to setting a feature to a prespecified baseline.  The aim then is to find the smallest set of features such that setting them to the baseline would cause a NN whose output is thresholded to change its prediction.  The authors show that, while this problem is in general NP-hard by reduction a to subset sum, a simple greedy strategy works for NN that satisfy a natural monotonicity property.  The theoretical results are validated via a simple experiment on MNIST data.
strength_and_weaknesses: Strengths:  The theoretical contribution appears to be novel to me, and it was somewhat surprising to me that such a simple greedy method is provably correct under mild assumptions.  The paper is well-motivated.

Weaknesses:  The empirical evaluation is a bit simplistic.  The authors claim (and I believe them) that the approach is widely applicable, but no real effort is made to perform more interesting real-world experiments.  There are no baselines or competitors considered in the empirical evaluation.  Somewhat less critical, in my opinion, there is some question as to how one would choose an appropriate baselines in domains that have a less obvious on/off switch than MNIST.
clarity,_quality,_novelty_and_reproducibility: Clarity:  I found the exposition clear in general.  One minor note:  the notation in (1) with tau(...) looks confusing.  I suggest adding a cdot between tau and the parentheses.

Novelty:  The is outside my area of expertise, but the contribution is novel as far as I can tell.

Quality:

(1) See above for a discussion of the experimental weaknesses.  

(2) Can you give an example of this for regression tasks (the experimental section mentions that it can be applied more generally)?  How are you determining a change in prediction in the regression task?

Reproducibility:  Everything seems straightforward to reproduce from the description in the text.
summary_of_the_review: A well-written and more-or-less self-contained paper that makes simple, but nice theoretical contributions.  Given the relative ease of implementation, a more thorough set of experiments should be conducted.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4987/Reviewer_a2S5
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper presents the minimal feature removal problem: compute the smallest set of input features s.t. the model prediction changes when these features are disregarded by setting them to a uninformative baseline value. The paper first shows that this problem is NP-hard, but has a greedy algorithm for monotonic FCNs. Section 6 of the papers shows their approach can be, in principle, adapted to CNNs and GNNs. Section 7 applies their algorithm to ReLU FCNs trained on MNIST.
strength_and_weaknesses: Strengths

- Well-written. The proposed feature removal problem is easy to understand
- minimal assumptions on FCNs, so proposed algorithm is realistic. Works well on FCNs trained on MNIST.
- Approach can be extended to other model architectures 

Weaknesses
- Connection to work in interpretability (i.e. integrated gradients) seems superficial. Not sure what the takeaway is after reading Section 3.
- Unclear why this problem is important given prior work on interpretability
- Unclear what is different when compared to \ell_0 adversarial attacks, other than baseline vs. arbitrary value (a minor difference)
- Empirical evaluation is quite limited and can be extended in multiple ways: realistic datasets, different architectures, comparison to \ell_0 attacks and off-the-shelf saliency map methods casted as algorithms for feature removal. More empirical evaluation should shed light on (a) this works well in practice and (b) if the proposed algorithm works better than previous approaches. 

Overall, the paper needs more work to (a) clearly motivate the importance of the problem and (b) thorough empirical evaluation to show that the proposed algorithm works well in practice.
clarity,_quality,_novelty_and_reproducibility: See Strength And Weaknesses
summary_of_the_review: See Strength And Weaknesses
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough