Paper: Interpreting Robustness Proofs of Deep Neural Networks (ICLR-2024)

Reviewer comment (Theory):
The proposed algorithm seems only valid for the linear property, how can we deal with the nonlinear properties?

Author response:
Existing works [1, 2] that handle non-linear properties typically compute a linear approximation of these properties and utilize existing DNN verifiers designed for linear specifications. For instance, in [1], when verifying robustness against geometric perturbations like rotation, an overapproximated convex input region is computed, specified by a set of linear inequalities encompassing all rotated images. Subsequently, existing DNN verifiers are applied. A similar approach is employed in [2] for non-linear output specifications. 
Both these cases with linear approximations of non-linear input and output specification can be handled with ProFIt. In general, for DNN verification, nonlinear specifications are rare, and most of the common norm-based local robustness properties can be specified as a conjunction of linear inequalities, and the corresponding proofs can be analyzed with ProFIt.

[1] "Certifying Geometric Robustness of Neural Networks", M. Balunovic, NeurIPS, 2019. \
[2] "Verification of Non-Linear Specifications for Neural Networks", C. Qin, ICLR, 2019.

Reviewer comment (Theory):
I find the motivation of this work to be a little ambiguous. Specifically, the author argues that the challenge this work aims to address here is "investigating the entire set $\mathcal{F}$ is always a valid but expensive option considering the size of $\mathcal{F}$". However, Algorithm 1 needs to iteratively query the verifier on subsets of $\mathcal{F}$, with the subset in the first couple of iterations potentially quite large. Particularly, at the first iteration, the cardinality of $F_{S_0} \cup F_{S_1}$ seems to be half of $\mathcal{F}$. Therefore, it is not immediately clear to me how the proposed algorithm reduces the overall computational complexity, and some additional explanation would be preferred.

Author response:
First, let us clarify the statement: "Investigating the entire set $\mathcal{F}$ is always a valid but expensive option, considering the size of $\mathcal{F}$." For DNNs, the proof feature set $\mathcal{F}$ may comprise thousands of proof features, making it impractical to manually investigate each one (e.g., by examining the gradient maps) individually. For this reason, one of our objectives is to automatically extract a proof feature set that is small (the first of three expectations outlined in Section 4.1) and, consequently, easy to manually investigate.
In Table 1 of the paper, we demonstrate that ProFIt successfully extracts a small yet sufficient and important proof feature set, containing fewer than 10 proof features. This significantly reduces the effort required for the manual investigation of all constituent proof features. We will provide further clarification in the revised version of the paper.

Reviewer comment (Experiments):
While the visualizations in Figure 3 are nice, it would be helpful and more meaningful if they could be generated from commonly used model architectures, as ConvNets and vision transformers may attend to different parts of the input image compared with the feed-forward networks used in this paper. Are there any challenges for applying the proposed methods to these more sophisticated networks, since these models usually also have a linear layer as the final layer? I believe that this paper will benefit from a more diverse empirical evaluation.

Author response:
All the networks used in the experiments in Table 1 of the paper, are convolutional. We have provided detailed information about the architecture of the DNNs, handled by ProFIt, in the response to Q1 in the common response. For even larger networks like vision transformers, the state-of-the-art deterministic DNN verifiers do not scale. As the scalability of ProFIt is ultimately constrained by the limitations of existing DNN verifiers, it currently does not scale to vision transformers. However, it's crucial to note that ProFIt is a general tool compatible with any deterministic DNN verification algorithm. Therefore, ProFIt will benefit from any future advances to enable the deterministic DNN verifiers to scale to vision transformers. We have discussed this limitation in Section 4.5 of the paper.

Reviewer comment (Writing):
Some typos: "uppper" on page 5; "both the baselines" on page 8.

Author response:
Thanks for pointing out we will fix this in the revised version of the paper.

Reviewer comment (Evaluation):
Is the certificate issued by ProFIt related to the robust accuracy in any way? My impression after reading the paper is that since $\it{F}_{S_0}$ returned by Algorithm 1 is always sufficient, the percentage of data that can be certified will be the same as the verifier $\mathcal{V}$. Is this correct?

Author response:
ProFIt analyzes the proof only if the underlying verifier $\mathcal{V}$ can prove the property. So, the percentage of cases ProFIt generates proof dissection is the same as the robust accuracy of the DNN with verifier $\mathcal{V}$.

Reviewer comment (Experiments):
I see that the experiments and calculations are performed using a CPU. Is the proposed algorithm also compatible with GPU acceleration?

Author response:
Yes, ProFIt is compatible with GPU acceleration. We provide the runtime analysis of ProFIt with GPUs in the response to Q1 in the common response. We will add these runtimes in the revised version of the paper.

Reviewer comment (Novelty):
The significance of generating human-understandable proof features and the use case of the extracted proof features need to be explained more clearly, as a DNN verifier is designed to provide a mathematically sound robustness certificate that can be trusted by design.

Author response:
The primary objective of ProFIt is to explain why the verifier successfully proves a property on the given DNN. This motivation aligns with traditional DNN interpretation techniques, where, given a DNN correctly computes the output, the goal is to comprehend the decision-making process of the DNN. In contrast to existing DNN verifiers that often act as black boxes, traditional program verifiers do offer insights into why certain properties are proven for the given program.
For instance, in programs featuring loops, loop invariants characterize the expected behavior of the program after each iteration, making the proof human understandable (discussed in Section 1 of the paper). We want to check whether the proofs generated by DNN verifiers provide similar insights or not. As shown in Section 5.3, even if a DNN is robust for some $L_{\infty}$ region, the input features, the proof relies on can be different. Moreover, dissection results obtained by ProFIt can serve as a qualitative metric to compare different DNNs achieving similar levels of robustness. For example, proofs generated on standard DNNs highlight spurious input features that are part of the background rather than those associated with the actual object. In contrast, the proof generated on COLT networks relies on input pixels that are part of the actual object.  These applications emphasize the importance of delving into proof dissection techniques.

Reviewer comment (Novelty):
It is unclear why there is a need for a DNN verifier to be explainable in the first place, given its purpose of providing a trustworthy robustness certificate.

Author response:
The primary objective of ProFIt is to explain why the verifier successfully proves a property on the given DNN. This motivation aligns with traditional DNN interpretation techniques, where, given a DNN correctly computes the output, the goal is to comprehend the decision-making process of the DNN. In contrast to existing DNN verifiers that often act as black boxes, traditional program verifiers do offer insights into why certain properties are proven for the given program.
For instance, in programs featuring loops, loop invariants characterize the expected behavior of the program after each iteration, making the proof human understandable (discussed in Section 1 of the paper). We want to check whether the proofs generated by DNN verifiers provide similar insights or not. As shown in Section 5.3, even if a DNN is robust for some $L_{\infty}$ region, the input features, the proof relies on can be different. Moreover, dissection results obtained by ProFIt can serve as a qualitative metric to compare different DNNs achieving similar levels of robustness. For example, proofs generated on standard DNNs highlight spurious input features that are part of the background rather than those associated with the actual object. In contrast, the proof generated on COLT networks relies on input pixels that are part of the actual object.  These applications emphasize the importance of delving into proof dissection techniques.

Reviewer comment (Writing):
Although the paper explains relevant concepts of proof features and develops a well-motivated algorithm to prioritize them, the underlying motivation of the considered problem should be better explained.

Author response:
The primary objective of ProFIt is to explain why the verifier successfully proves a property on the given DNN. This motivation aligns with traditional DNN interpretation techniques, where, given a DNN correctly computes the output, the goal is to comprehend the decision-making process of the DNN. In contrast to existing DNN verifiers that often act as black boxes, traditional program verifiers do offer insights into why certain properties are proven for the given program.
For instance, in programs featuring loops, loop invariants characterize the expected behavior of the program after each iteration, making the proof human understandable (discussed in Section 1 of the paper). We want to check whether the proofs generated by DNN verifiers provide similar insights or not. As shown in Section 5.3, even if a DNN is robust for some $L_{\infty}$ region, the input features, the proof relies on can be different. Moreover, dissection results obtained by ProFIt can serve as a qualitative metric to compare different DNNs achieving similar levels of robustness. For example, proofs generated on standard DNNs highlight spurious input features that are part of the background rather than those associated with the actual object. In contrast, the proof generated on COLT networks relies on input pixels that are part of the actual object.  These applications emphasize the importance of delving into proof dissection techniques.

Reviewer comment (Evaluation):
In the gradient map visualizations shown in Figure 3, the statement "CROWN-IBP filters out most of the spurious features, but it also misses out on some meaningful features" is unclear. The meaning of "spurious" and "meaningful" features in the context of gradient maps needs clarification. Additionally, does the provably robust training method CROWN-IBP have a much lower standard accuracy because it overly filters the meaningful features?

Author response:
As defined in Section 5.3 in the paper, for gradient maps generated on MNIST and CIFAR10 images, spurious input features are the pixels coming from the image background, whereas meaningful input features are the pixels that are a part of the actual object being identified by the DNN. 
Robustly trained networks generally report lower standard accuracy than commonly trained networks [1]. Additionally, certifiably robust DNNs (CROWN-IBP) generally report smaller standard accuracy when compared to empirically robust DNNs (PGD-trained) [2]. It appears that the loss in standard accuracy is due to the fact that the certifiably robust networks overly filters the meaningful input features.

[1] "Robustness May Be at Odds with Accuracy", D. Tsipras. ICLR, 2019.

[2] "Certified Training: Small Boxes are All You Need ", M. Mueller, ICLR, 2023.