Paper: Improving Branching in Neural Network Verification with Bound Implication Graph (ICLR-2024)

Reviewer comment (Experiments):
The paper only considers certifying adversarial robustness on perception networks.

Author response:
Our method largely relied on state-of-the-art BaB-based verification frameworks such as [2,3,4,5,6], and is limited by the capability of existing verifiers, which focuses on branch-and-bound (BaB) for ReLU networks only. We want to point out that this is a limitation of the majority of verification papers. For other non-linear activations and non-perception networks (such as Transformers), one very recent workshop paper [1] (still unpublished) considered BaB. Our method can be extended to the generalized BaB in [1], by changing the right-hand-side constraint in Eq. (5) and (6) to a non-zero splitting point. We will work towards this direction in our future work.

Reviewer comment (Experiments):
What is the actual perturbation bounds and perturbation dimensions on the VeriHard benchmark sets?

Author response:
Thanks for pointing this out. We include the perturbation bounds and dimensions on the VeriHard benchmark sets in an [anonymous Google sheet](https://docs.google.com/spreadsheets/d/1hHL9_03N7zEAAyLRwfasTBwYXHEIRG-pBetcYtS7hqE/edit?usp=sharing) so you can take a look. We will publish our dataset once our paper is accepted.

Reviewer comment (Novelty):
The paper is highly incremental to [1, 2].

Author response:
We thank the reviewer for giving detailed technical explanation for your mentioned work [1,2]. We have significantly revised our paper based on your suggestions, and correctly **cited and discussed [1,2] throughout the paper**; major changes were marked in blue.  We now discuss [1,2] early in our paper in the introduction, do not claim the implication graph as part of our contribution in Section 3, and remove imprecise sentences in Section 5. We also added a **detailed experimental comparison to [1,2]** to show our strong improvements.

The main novelty of our paper is not about constructing the graph, but a new formulation that is more powerful and less restrictive to find bound implications. **Compared to [1,2], we have both theoretical advantage and strong experimental evidence with 7x - 170x more bound implications found** (Table 1 in our revised paper). We respectfully disagree this is a “small delta” or incremental contribution. We hope the reviewer can reevaluate our paper based on new discussions and experiments about [1,2], where we have made it clear that [1,2] are important prior work in this field.

Reviewer comment (Novelty):
The paper often presents several ideas as novel when they were first discussed in [1, 2].

Author response:
We thank the reviewer for giving detailed technical explanation for your mentioned work [1,2]. We have significantly revised our paper based on your suggestions, and correctly **cited and discussed [1,2] throughout the paper**; major changes were marked in blue.  We now discuss [1,2] early in our paper in the introduction, do not claim the implication graph as part of our contribution in Section 3, and remove imprecise sentences in Section 5. We also added a **detailed experimental comparison to [1,2]** to show our strong improvements.

The main novelty of our paper is not about constructing the graph, but a new formulation that is more powerful and less restrictive to find bound implications. **Compared to [1,2], we have both theoretical advantage and strong experimental evidence with 7x - 170x more bound implications found** (Table 1 in our revised paper). We respectfully disagree this is a “small delta” or incremental contribution. We hope the reviewer can reevaluate our paper based on new discussions and experiments about [1,2], where we have made it clear that [1,2] are important prior work in this field.

Reviewer comment (Evaluation):
While the experimental results from [1] clearly show that implication constraints help with verification times, the paper says "they were handled as constraints to a black-box MIP solver without validating their usefulness".

Author response:
We have also addressed each individual weakness you pointed out:

> a. While the experimental results from [1] clearly show that implication constraints help with verification times, the paper says "they were handled as constraints to a black-box MIP solver without validating their usefulness".

We removed this confusing sentence. Our new characterization of [1,2] is provided throughout the paper, particularly in Sections 1 and 5.

Reviewer comment (Novelty):
The paper does not clarify with the introduction of the implication constraints that they were first introduced in [1].

Author response:
We have also addressed each individual weakness you pointed out:

> d. The paper does not clarify with the introduction of the implication constraints that they were first introduced in [1].

We rewrote the introduction and cited [1,2] at the beginning.

Reviewer comment (Experiments):
The implication constraint identification procedures are not compared with the ones from [1]. It is not clear that they perform better. For instance, [1, 2] show similar gains. Also, the procedures from [1] should be faster because they deal with concrete bounds instead of bound equations.

Author response:
> The implication constraint identification procedures are not compared with the ones from [1]. It is not clear that they perform better.

We added comparisons in both experiments (Section 4) and in theoretical perspectives (Section 5).

**Experimental comparison to [1,2]**: We added experiments in Section 4 to compare to [1, 2] and showed that we could find significantly more neuron implications. We evaluate our method on a few networks used in [1,2], and we use the official code of Venus2. To allow a fair comparison, here we do not use any particular BaB solver and consider the implications found in the initial verification problem. The results are presented in Table 1 in our revised paper and also highlighted below. We outperform the baselines by finding 7x - 180x more implications. The difference is especially significant in convolutional networks - the zero results from Venus2 are not errors (see our contribution 4 above).

|   | MNIST-MLP | CIFAR-MLP | CIFAR-CNN-A-Adv | CIFAR-CNN-A-Mix |
|----------------------|:---------:|:---------:|:---------------:|:---------------:|
| Venus2 (inter-layer) |   56.15   |    4.66   |       6.42      |      19.17      |
| Ours (inter-layer)   |   203.5   |   397.5   |      340.0      |      486.5      |
| Venus2 (intra-layer) |    9.80   |   35.33   |        0        |        0        |
| Ours (intra-layer)   |   263.4   |   502.1   |      763.4      |      803.2      |

**Theoretical comparison to [1,2]**: Compared to [1,2], our formulation is significantly stronger, for the following reasons:

1. For inter-layer dependency, [1,2] can only consider the case where a neuron in one layer is set to be inactive (0), and propagation the bound to the layers following it. In Lemma 3 of [2], cases 3 and 4 are empty, indicating that they cannot find any implications when one neuron is set to be active. Our methods can be used to find all four kinds of implications.

2. Furthermore, our method can find implicated neurons *before* the implicant neuron. For example, by setting a neuron in layer 3 to be active or inactive, we may imply that a neuron in layer 1 or 2 (before layer 3) is active or inactive. This is impossible in [1, 2].

3. For intra-layer dependency, our methods use bound equations that are propagated to the input layer, which utilizes tighter bounds on the input layer rather than the relatively looser intermediate layer bounds in [1,2]. Our formulation is also optimization-based, allowing us to find the strongest implications under this formulation. As a result, we found significantly more intra-layer neuron dependencies (Table 1).

4. Furthermore, the method in [1,2] does not work well for convolutional networks, because the equivalent weight matrices are sparse, so the pair of neurons under investigation may have no correlation at all  (Table 1 where Venus2 reports 0). Our method uses the bound equations at the input x, which is often dense (as large as the receptive field of the entire network, rather than one layer) and can capture neuron correlations better.

Reviewer comment (Experiments):
An important baseline, GCP (Zhang et al. 2022), the most recent version of $\alpha\beta$-CROWN, is missing from all comparisons. However the MILP constraints leveraged by this approach might be highly correlated to the bound implications described in this work. Thus, a direct comparison or even better combination with GCP would be essential to judge the marginal contribution of this work over state-of-the-art methods.

Author response:
Thanks for pointing this out. We carefully reviewed the GCP-CROWN paper you mentioned and we also integrated BIG into GCP-CROWN. However, we want to emphasize that GCP-CROWN heavily relies on a MIP solver to find cutting planes, which cannot be applied to large models (e.g. CIFAR100, TinyImageNet models) due to its high computational cost. Thus, in many settings of our paper, GCP-CROWN cannot bring improvements, yet our method still can.

We evaluate the total number of branches (less is better) on several benchmarks below. GCP-CROWN+BIGs could further reduce the number of branches compared with GCP-CROWN only. Note that GCP-CROWN cannot scale to CIFAR100 and TinyImageNet, yet our bound implications can still bring improvements. We added these results to Appendix G.


|            | $\beta$-CROWN | GCP-CROWN     | BIG     | GCP-CROWN+BIG     |
|-------------|---------|---------|---------|-------------|
| MNIST-A-Adv | 53120.9 | 22250.4 | 45130.4 | **21230.4** |
| CIFAR-A-Adv | 50614.2 | 24103.7 | 46132.4 | **23792.5** |
| CIFAR-A-Mix | 55764.8 | 28607.1 | 50014.2 | **28204.6** |
| TinyImageNet-Medium | 64253.9 | - | 60082.5 | - |
| CIFAR100-small | 74113.5 | - | 70006.6 | - |

Reviewer comment (Evaluation):
Significance of improvements on established benchmarks (less than $1\%$) is unclear.

Author response:
> Significance of improvements on established benchmarks

Thanks for mentioning the issue. We believe that minor improvements on established benchmarks are largely due to the benchmark design. These benchmarks were proposed when neural network verifiers were relatively weak. Nowadays a majority of SDP-FO[1] and ERAN[2] benchmarks have been largely solved, while a small portion of unsolved instances with unknown (potentially very high) difficulty. The improvement in the verification framework cannot be fully reflected by considering these instances with unknown difficulty. Our new VeriHard benchmark contains difficult examples that existing verifiers can guarantee to solve with a long timeout (so the difficulty is known, similar to the classic oval20 benchmark used in many papers). The progress of neural network verifiers can be reflected better in this new benchmark.

[1] Dathathri, Sumanth, et al. "Enabling certification of verification-agnostic networks via memory-efficient semidefinite programming." Advances in Neural Information Processing Systems 33 (2020): 5318-5331.

[2] Singh, Gagandeep, et al. "Fast and effective robustness certification." Advances in neural information processing systems 31 (2018).

Reviewer comment (Novelty):
Did you investigate the interaction of these bound implications with the general cutting planes from GCP (also based on beta-CROWN)? It seems like similar implications might be captured by the MILP solver used there.

Author response:
> Did you investigate the interaction of these bound implications with the general cutting planes from GCP (also based on beta-CROWN)?

That’s a very interesting question. We exported the cutting planes from GCP-CROWN and compared them to our implications on the same set of instances. We report the average number of implications found by us, the average number of cutting planes in GCP-CROWN, and the overlaps. Here overlap means that GCP-CROWN finds a constraint mentioning the same pair of two neurons as we found as implicated and implicant neurons.

| Average Implications | GCP-CROWN |   BIG  | Overlaps |
|----------------------|:---------:|:------:|:------:|
| MNIST-A-Adv          |   574.2   |  761.7 |  102.2 |
| CIFAR-A-Adv          |   683.7   | 1103.3 |  89.9  |
| CIFAR-A-Mix          |   796.9   | 1289.7 |  104.2 |

As we can see, the overlaps only take a minority of the overall implications we found with our method. This is expected because GCP-CROWN uses a MIP solver to generate cutting planes, which is unaware of the underlying neural network architecture and uses generic methods (such as Gomory cuts) to find implications among variables. We added these results to Appendix G.

Reviewer comment (Theory):
In Theorem 3.2, it seems like the two enumerated points add an additional (exhaustive) case distinction if Equation (7) holds, where the first one can be checked independently of the implicated neuron. I would consider moving them to a separate Corollary. What exactly does it mean for its “linear bounds to not have an intersection with C”? Is the first constraint (functionally) equivalent to $a x_0 + c + \epsilon ||a|| < 0 $, i.e. the lower bound will remain negative for any $x \in \mathcal{C}$? I believe the presentation of Theorem 3.2 could be improved significantly. In particular, it is not initially clear, that either of the two cases follows from Equation (7) rather than being additional constraints.

Author response:
> Presentation of Theorem 3.2 can be improved. An illustration of Theorem 3.2 could help communicate the underlying intuition.

We have split Theorem 3.2 into a Theroem and a Corollary, as suggested. Here “linear bounds to not have an intersection with C” means that the implicant lower constraint Eq. (6) is too far away from $x_0$, and it is outside of the box C centered at $x_0$. We have revised the presentation of the theorem accordingly. In addition, **we’ve added a figure** (Fig. 2) to showcase the cases in the Corollary, as you suggested.

Reviewer comment (Presentation):
The scaling of the y-axis in Figure 2 is unclear making it hard to interpret. I would consider a linear scale instead.

Author response:
> The scaling of the y-axis in Figure 2 is unclear making it hard to interpret. I would consider a linear scale instead.

Thanks for pointing this out. We revised our figures to have a linear y-axis scale in our revision.

Reviewer comment (Writing):
I believe it should be "(inputs $\leq$ 0)" for the "inactive" case in Section 1 paragraph 3.

Author response:
> I believe it should be "(inputs <= 0)" for the "inactive" case in Section 1 paragraph 3. Other typos.

Thanks for your nice catch! We have fixed these typos in our revision.

Reviewer comment (Presentation):
An illustration of Theorem 3.2 could help communicate the underlying intuition.

Author response:
> Presentation of Theorem 3.2 can be improved. An illustration of Theorem 3.2 could help communicate the underlying intuition.

We have split Theorem 3.2 into a Theroem and a Corollary, as suggested. Here “linear bounds to not have an intersection with C” means that the implicant lower constraint Eq. (6) is too far away from $x_0$, and it is outside of the box C centered at $x_0$. We have revised the presentation of the theorem accordingly. In addition, **we’ve added a figure** (Fig. 2) to showcase the cases in the Corollary, as you suggested.

Reviewer comment (Writing):
Mn-BaB instead of MN-BaB in Section 1 paragraph 2.

Author response:
> I believe it should be "(inputs <= 0)" for the "inactive" case in Section 1 paragraph 3. Other typos.

Thanks for your nice catch! We have fixed these typos in our revision.

Reviewer comment (Writing):
Extra “the” in the third line of Section 2.

Author response:
> I believe it should be "(inputs <= 0)" for the "inactive" case in Section 1 paragraph 3. Other typos.

Thanks for your nice catch! We have fixed these typos in our revision.

Reviewer comment (Writing):
The second sentence after Equation (3) is broken and missing a full-stop.

Author response:
> I believe it should be "(inputs <= 0)" for the "inactive" case in Section 1 paragraph 3. Other typos.

Thanks for your nice catch! We have fixed these typos in our revision.

Reviewer comment (Writing):
“Activate and inactive” third last line in Section 2.

Author response:
> I believe it should be "(inputs <= 0)" for the "inactive" case in Section 1 paragraph 3. Other typos.

Thanks for your nice catch! We have fixed these typos in our revision.

Reviewer comment (Evaluation):
More discussions on the bottleneck of the proposed method would further strengthen the paper.

Author response:
Thanks for your suggestion. We’ve added a discussion to our paper (Section H) on the bottleneck as suggested. Our proposed method is highly relied on BaB-based verification framework so it has some common issues: 1) Our derived implications follow the large body of papers for verifying ReLU networks [1,2,3,4] and need further extensions to other non-piecewise linear activation functions. 2) We cannot support the verification of very large neuron networks, such as Transformer with billions of parameters. We will keep investigating the possibility of applying our implication concept into the verification of more general models in the future.

Reviewer comment (Experiments):
Could you comment on how would the BIG can be used to discover hard adversarial examples or train the model to be robust against those examples?

Author response:
This is a very interesting direction. In our work, we did not use our implication graph (BIG) to discover adversarial examples during branch-and-bound, but there exist some papers focusing on finding adversarial examples through branch-and-bound procedure [5]. Our method can be applied on top of these BaB-based methods to reduce the research space when finding adversarial examples, and this could be an interesting future work.