Paper: Understanding Certified Training with Interval Bound Propagation (ICLR-2024)

Reviewer comment (Theory):
The theoretical results have strong assumptions such as linear neural networks, and neural network weights under Gaussian distribution. This is generally not a big concern if the authors can demonstrate that these theoretical insights can lead to great practical improvements.

Author response:
**Q1. Can you discuss the implications of the (strong) assumptions (DLNs and Gaussian weight distributions) made during the theoretical analysis?**  

First, we want to highlight that the definitions of propagation invariance and bound tightness (Lemma 3.3 and Definition 3.6) do not assume a Gaussian weight distribution. Only our results on tightness at initialization make this assumption. There, however, it agrees well with commonly used initialization schemes, which draw weights from a Gaussian distribution.
Second, while we do assume (local) linearity for many of our theoretical results, we want to highlight that ReLU networks are locally linear and thus behave linearly for infinitesimally small perturbation magnitudes. Even at the typically considered perturbation magnitudes, as few as 1% of ReLUs are unstable, leading to empirically small approximation errors (see Figure 2 and the new Figure 13). As we empirically validate all our results on ReLU networks and even generalize some of them theoretically to this setting (see Corollary A.2), we believe the insights obtained under the assumption of local linearity to be valuable and likely to generalize.

Reviewer comment (Experiments):
The theoretical results developed do not lead to a better model that can outperform existing approaches, and some evaluations are quite weak (e.g., on a single MNIST model only). Since only a few models and networks are shown, it is unsure how general the results are.

Author response:
**Q2. Can you discuss the generality of the empirical support of your theoretical findings given that some experiments are conducted for only one dataset and network architecture?**  

As most of our experiments involve sweeps over either architectures or perturbation magnitudes and certification with a precise neural network verifier, they can be quite compute-intensive. We have thus focused on what we believe to be the most informative and interesting settings, using the popular CNN7 (which yields state-of-the-art results in most settings) when trying to demonstrate performance improvements and a smaller CNN3 when computing exact bounds using MILP or aiming to certify PGD trained networks, which would be intractable for larger architectures. 
In addition, we confirm key results across multiple training methods and datasets (see below), e.g., while the results in Figure 6 already constitute a new state-of-the-art, we confirm it on CIFAR-10 in Table 1, where we also observe strict improvements on multiple state-of-the-art training methods. We have added new results (Figures 13 and 14) to the appendix confirming the results previously established in only one setting (Figures 2 and 9).
Below, we provide an overview of results and settings we considered. 

If the reviewer believes any results to still be insufficiently established, we are more than happy to conduct additional experiments to address this.

*Approximation error between local (Definition 3.6) and true tightness:* CNN3 on MNIST for 5 perturbation magnitudes (Figure 2). Now additionally for CIFAR-10 (Figure 13)

*Tightness at initialization (Lemma 3.7 and Corollary 3.8, rigorous theoretical results):* 11 architectures based on CNN7 on CIFAR-10, although this is mostly dataset and completely perturbation magnitude independent, as we consider the network at initializtion (Figure 3).

*Low dimensional embedding and reconstruction (Theorem 3.10, rigorous theoretical results):* 19 toy datasets constructed from projections of multivariate standard Gaussians (Figure 4).

*Effect of depth on trained networks:* 6 architectures derived from CNN7 for CIFAR-10 at $\epsilon=2/255$ (Figure 5).

*Effect of width on trained networks:* 6 architectures derived from CNN7 for CIFAR-10 at $\epsilon=2/255$ (Figure 5) and 3 training methods, two width factors, and 2 datasets (all using state-of-the-art networks as baseline) (Table 1 and Figure 6), now extended to 3 datasets and 5 perturbation magnitudes.

*Effect of perturbation magnitude on trained networks (Theorem 3.9):* 8 perturbation magnitudes and 3 training methods for CNN3 and CIFAR-10 (Figure 7). 2 Perturbation magnitudes and 5 training methods for a state-of-the-art architecture from prior work (Table 2).

*Regularization strength at same perturbation magnitude:* 2 training methods with 4 regularization strengths each for CNN7 on CIFAR-10 (Figures 8 and 12).

*Perturbation magnitude vs propagation region size:* SABR training with 3 different lambda and 6 perturbation region sizes each for a CNN3 on CIFAR-10 (Figure 9), now also for MNIST (Figure 14).

**References:**  
[1] Müller et al. "Certified training: Small boxes are all you need." ICLR 2023    
[2] Shi et al. "Fast certified robust training with short warmup." NeurIPS 2021    
[3] Mao et al. "Connecting Certified and Adversarial Training." NeurIPS 2023    
[4] Singh et al. "An abstract domain for certifying neural networks." POPL 2019

Reviewer comment (Evaluation):
Although some empirical results are shown to support the theory for ReLU networks, it is hard to argue these observations are indeed the consequence of the theory. For example, “certified training increases tightness” and “larger networks lead to better accuracy” are very generic conclusions, and it is hard to convince the readers that they result from the theory developed in this work.

Author response:
**Q5. Can you discuss to what extent your empirical observations directly confirm your theoretical results?**  

While some of our key results can be easily confirmed experimentally, others are indeed more difficult to disentangle. We want to highlight that we confirm our results on tightness at initialization both for DLNs and ReLU networks in Figure 3. There, we show that tightness falls polynomially with increasing width (Lemma 3.7) and exponentially with increasing depth (Corollary 3.8). We further show that ReLU networks of fixed depth yield a constant factor increase in tightness (Corollary A.2). Similarly, we confirm the linear growth of the reconstruction error under IBP and square root growth under optimal propagation in the dimensionality reduction setting (Theorem 3.10 and Figure 4). While we can not rigorously show that the gradient alignment described in Theorem 3.9 is directly responsible for increasing tightness with IBP training, our empirical results (Figure 7 and Table 2) show that this increase in tightness is consistently observed for IBP-based methods, but to a much smaller degree (COLT) or not at all (PGD) for non-IBP based methods, even if they achieve good certified accuracy (COLT). Additionally, we observe that for SABR, tightness is dominated by the propagation region size and not the perturbation magnitude, agreeing well with Theorem 3.9 (Figures 9 and 14). We believe that taken together, these empirical results provide strong support for the applicability of our theoretical results on DLNs (rigorously proven in Appendix B) to ReLU networks (as acknowledged by Reviewer HbFM).

Reviewer comment (Experiments):
Figure 6 shows that increasing model width is beneficial, however, it is on a simple MNIST network. Can you demonstrate this result on larger networks and datasets? In particular, if we use a state-of-the-art method and model, and enlarge the model size by 4x, how much gain can we obtain over the state-of-the-art? Is the gain consistent over multiple models (e.g., CNN, resnet), training methods (IBP, CROWN-IBP, SABR, MTL-IBP), and datasets (CIFAR10, CIFAR100, TinyImageNet)?

Author response:
**Q3. Can you demonstrate that the improved model performance with increased width observed for MNIST in Figure 6 also holds for other data sets, (state-of-the-art) architectures, and training methods?**  

*Settings:*  
First, we want to note, that MNIST (at $\epsilon \in \\{0.1, 0.3\\}$) and CIFAR-10 (at $\epsilon \in \\{2/255, 8/255\\}$) are the four most frequently studied settings in certified adversarial robustness [1,2,3]. Between Figure 6 and Table 1, we considered two of these settings and show improvements in the state-of-the-art for both. We added experiments for the remaining two settings where we observed strict improvements in 3 out of 4 cases (see updated Table 1). Additionally, we are conducting experiments TinyImageNet and will add these results once completed.

*Architectures:*  
We consider a range of convolutional architectures derived from CNN7, which is used by all recent methods to achieve state-of-the-art results [1,2,3]. ResNets are not typically studied in the field, as they yield worse performance. We have confirmed these results for fully connected networks, which used to be the standard in the field [4], in Table 3, but believe the results around CNN7 to be of greater significance.

*Methods:*  
We study IBP as a baseline and two state-of-the-art methods, SABR [1] and IBP-R [5], which are both based on IBP. As we see improvements in certified accuracy across almost all methods and settings, we believe our findings to generalize across IBP-based methods.
We did not consider CROWN-IBP [7] as it is outdated, yielding worse results than IBP [2], and MTL-IBP [8] as its code has not been released.

Reviewer comment (Theory):
Based on Theorem 3.4, can we reparameterize the network such that the bounds are always tight, and the training process just needs to search from a subspace of weights that lead to tight bounds, rather than using gradient descent to enforce tight bounds? For L2 norm certified defense, the state-of-the-art methods use this approach (such as orthogonal convolution layers and their variants).

Author response:
**Q6. Can we reparameterize networks based on Theorem 3.4 such that the bounds are always tight?**  
Great question! The easiest way to ensure propagation invariance based on Lemma 3.3 / Theorem 3.4 is to use non-negative weights. However, while this is sufficient it is not necessary and an even stricter constraint than implied by Theorem 3.4. Empirically, it leads to poor performance already in the non-robust setting [8]. We have further experimented with propagation invariant initializations based on Lemma 3.3 but have found the propagation invariant nature to be unlearned in the early stages of training and enforcing it without overregularizing to be non-trivial. We believe such a reparameterization to be interesting an
direction and promising item for future work.

Reviewer comment (Theory):
Although there are some examples in the introduction and formulation, the theory details lack some intuitive insights or explanations, e.g. Theorem 3.9 needs more insights to make it intuitive as it is one of the core theorems in this work.

Author response:
**Q1. Can you add more intuitions for the theoretical results you prove, e.g., Theorem 3.9?**  
While many of the proofs are quite technical, the results are intuitive, we have thus focused on providing intuitions on the results rather than the proof techniques.   
For example, the proof of Theorem 3.9 uses a first-order Taylor approximation (in perturbation magnitude) of the robust risk to show that the gradient change $R(\epsilon + \Delta \epsilon) - R(\epsilon)$ with increasing $\epsilon$ is aligned with the tightness gradient. We then sum over many perturbation magnitude increments $\Delta \epsilon$, and use the linearity of the inner product to obtain that $R(\epsilon) - R(0)$ is also aligned with the tightness gradient. As we let $\Delta$ go against 0, we obtain our result.   
The best intuition we can give is that every increase of the perturbation magnitude makes the gradient more aligned with tightness.

Reviewer comment (Experiments):
The details of the experiments are not given in the main text; specifically, the datasets and models used in Fig. 3 are not clear. It is better to re-organize experiments part by adding a setup subsection for these necessary details.

Author response:
**Q2. Can you provide more details on the experimental setup? For example, the setting of Fig. 3 remains unclear.**  
Of course! We have added the key details for every experiment (in particular for Figures 2, 3, and 9) to the main text. However, as we consider a large number of training methods and multiple datasets, each with different (default for these methods) hyperparameters, we defer additional details to Appendix C (almost one page) due to space constraints.

Reviewer comment (Evaluation):
Why is the certified accuracy decreasing when $\epsilon$ increases in the middle figure in Fig. 7? More explanations and justifications are needed. Besides, it seems that there is no explanation of the trade-off between accuracy and robustness as claimed in the abstract and introduction, except for tightness defined in the work. How about the certified robustness (e.g. certified accuracy or certified radii) for the trade-off?

Author response:
**Q3. Why is the certified accuracy decreasing as $\epsilon$ increases in the middle part of Figure 7?**  
In Figure 7, we vary the $\epsilon$ used for both training and testing. Thus, the certified accuracy in the middle of Figure 7 is w.r.t. larger perturbation magnitudes as $\epsilon$ increases, which makes it a strictly harder task. Thus two effects come together. First, standard accuracy decreases (Figure 7 left) as networks are more heavily regularized for these larger perturbation magnitudes (see increases in tightness in Figure 7 right). Second, for the same network, every sample that is robust at $\epsilon$ is also robust at $\epsilon’ < \epsilon$ but not vice versa, thus certified accuracy is always monotonically decreasing with perturbation magnitude (for a fixed network).

**Q4: Can you provide a more detailed explanation of your results on the robustness accuracy trade-off and in particular how your results on tightness relate to certified accuracy?**  
Yes. We argue along the following lines: To achieve high certified accuracies, certified training is required*. Empirically, IBP-based training methods have proven the most successful, yielding state-of-the-art results in every setting [1,2,3]. However, training with IBP increases tightness (Theorem 3.9), which induces strong regularization (Theorem 3.4), which in turn leads to reduced accuracy. We thus provide an explanation for the robustness accuracy trade-off for IBP-based training methods, which are practically the most relevant group of certified training methods.

*We note that while the combination of a relatively small network (CNN3) and a complete verifier (MN-BaB) yields comparatively high certified accuracies for the PGD-trained network in Figure 7, especially at small perturbation magnitudes, certified training methods are required for state-of-the-art performance (which require larger networks).

Reviewer comment (Writing):
Some analysis in Section 4.1 is not clear, please see the questions below.

Author response:
**Q1. Why does a lower tightness lead to higher accuracy in Figure 5? And why is increasing depth worse than increasing the width if less tightness is desirable?**  
At initialization, both higher width and depth decrease tightness significantly (see Figure 3). Then, IBP-based training increases tightness (Theorem 3.9), inducing (strong) regularization in the process (Theorem 3.4). The larger this increase in tightness, the stronger the induced regularization. As (IBP-)trained networks exhibit roughly similar tightness across depths and widths (compared to those at initialization), smaller tightness at initialization requires stronger regularization to reach this level (see the updated Figure 5). If this increase in regularization dominates the increase in capacity, we achieve worse goodness of fit (training set robust accuracy) and thus performance. This is what we observe in Figure 5, tightness increases much more over training for large depth than for large width, indicating stronger regularization, explaining the worse certified accuracy. So less tightness is only a positive sign (of less regularization) after IBP-training.

Reviewer comment (Novelty):
Some missing related works: [1] has a relevant conclusion on the diminishing improvement with increasing width in IBP training. [1] On the Convergence of Certified Robust Training with Interval Bound Propagation.

Author response:
**Q2. Can you relate your results to those of Wang et al. [1] which conclude that there are diminishing improvements with increasing width in IBP training?**  
First, we want to highlight that we already cite Wang et al. [1] in our related work section. Their work considers the convergence of IBP training in the over-parameterized setting for small perturbation magnitudes with high probability and finds that sufficient width is required to achieve zero training loss with high probability. In this setting, the required width increases inversely proportional to the square of the non-convergence probability (for very small perturbation magnitudes). However, while this can be seen as diminishing “improvements” of convergence probability with increasing width, we observe diminishing improvements in test set certified accuracy, a fundamentally different metric. Further, the work of Wang et al. [1] is limited to 10x smaller perturbations than we consider (see e.g. their Figure 1).

Reviewer comment (Evaluation):
In figure 5, why does a decreasing tightness lead to higher accuracy? If a looser bound is better due to weaker regularization, why is increasing the depth worse than increasing the width?

Author response:
**Q1. Why does a lower tightness lead to higher accuracy in Figure 5? And why is increasing depth worse than increasing the width if less tightness is desirable?**  
At initialization, both higher width and depth decrease tightness significantly (see Figure 3). Then, IBP-based training increases tightness (Theorem 3.9), inducing (strong) regularization in the process (Theorem 3.4). The larger this increase in tightness, the stronger the induced regularization. As (IBP-)trained networks exhibit roughly similar tightness across depths and widths (compared to those at initialization), smaller tightness at initialization requires stronger regularization to reach this level (see the updated Figure 5). If this increase in regularization dominates the increase in capacity, we achieve worse goodness of fit (training set robust accuracy) and thus performance. This is what we observe in Figure 5, tightness increases much more over training for large depth than for large width, indicating stronger regularization, explaining the worse certified accuracy. So less tightness is only a positive sign (of less regularization) after IBP-training.