Original: Improved techniques for deterministic l 2 robustness
Revised: Improved techniques for deterministic l 2 robustness

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Training convolutional neural networks (CNNs) with a strict 1 -Lipschitz constraintunder the l 2 norm is useful for adversarial robustness, interpretable gradients andstable training.
Revised: Training convolutional neural networks (CNNs) with a strict 1 -Lipschitz constraint under the l 2 norm is useful for adversarial robustness, interpretable gradients and stable training.

Original: 1 -Lipschitz CNNs are usually designed by enforcing each layer tohave an orthogonal Jacobian matrix (for all inputs) to prevent the gradients fromvanishing during backpropagation.
Revised: 1 -Lipschitz CNNs are usually designed by enforcing each layer to have an orthogonal Jacobian matrix (for all inputs) to prevent the gradients from vanishing during backpropagation.

Original: However, their performance often significantlylags behind that of heuristic methods to enforce Lipschitz constraints where theresulting CNN is not provably 1 -Lipschitz.
Revised: However, their performance often significantly lags behind that of heuristic methods to enforce Lipschitz constraints where the resulting CNN is not provably 1 -Lipschitz.

Original: In this work, we reduce this gap byintroducing (a) a procedure to certify robustness of 1 -Lipschitz CNNs by replacingthe last linear layer with a 1 -hidden layer MLP that significantly improves theirperformance for both standard and provably robust accuracy, (b) a method tosignificantly reduce the training time per epoch for Skew Orthogonal Convolution(SOC) layers ( > 30% reduction for deeper networks) and (c) a class of poolinglayers using the mathematical property that the l 2 distance of an input to a manifoldis 1 -Lipschitz.
Revised: In this work, we reduce this gap by introducing (a) a procedure to certify robustness of 1 -Lipschitz CNNs by replacing the last linear layer with a 1 -hidden layer MLP that significantly improves their performance for both standard and provably robust accuracy, (b) a method to significantly reduce the training time per epoch for Skew Orthogonal Convolution (SOC) layers ( > 30% reduction for deeper networks) and (c) a class of pooling layers using the mathematical property that the l 2 distance of an input to a manifold is 1 -Lipschitz.

Original: Using these methods, we significantly advance the state-of-the-artfor standard and provable robust accuracies on CIFAR-10 (gains of +1 .
Revised: Using these methods, we significantly advance the state-of-the-art for standard and provable robust accuracies on CIFAR-10 (gains of +1 .

Original: 79% and+3 .
Revised: 79% and +3 .

Original: 82% ) and similarly on CIFAR-100 (+ 3 .
Revised: 82% ) and similarly on CIFAR-100 (+ 3 .

Original: 78% and + 4 .
Revised: 78% and + 4 .

Original: 75% ) across all networks.
Revised: 75% ) across all networks.

Original:  Introduction
Revised:  Introduction

Original: The Lipschitz constant 1 of a neural network f : R d → R c , denoted by Lip( f ) , controls the changein the output divided by change in the input (both changes measured in the l 2 norm).
Revised: The Lipschitz constant 1 of a neural network f : R d → R c , denoted by Lip( f ) , controls the change in the output divided by change in the input (both changes measured in the l 2 norm).

Original: Previouswork provides evidence that a small Lipschitz constant is useful for interpretable saliency maps[Tsipras et al., 2018], generalization bounds [Long and Sedghi, 2020], Wasserstein distance estimation[Villani, 2008], adversarial robustness [Szegedy et al., 2014] and preventing gradient explosion duringbackpropagation [Xiao et al., 2018].
Revised: Previous work provides evidence that a small Lipschitz constant is useful for interpretable saliency maps [Tsipras et al., 2018], generalization bounds [Long and Sedghi, 2020], Wasserstein distance estimation [Villani, 2008], adversarial robustness [Szegedy et al., 2014] and preventing gradient explosion during backpropagation [Xiao et al., 2018].

Original: Several prior works [Miyato et al., 2018, Gulrajani et al., 2017]use heuristic methods to enforce Lipschitz constraints to successfully address problems such asstabilizing GAN training.
Revised: Several prior works [Miyato et al., 2018, Gulrajani et al., 2017] use heuristic methods to enforce Lipschitz constraints to successfully address problems such as stabilizing GAN training.

Original: However, these methods do not enforce a guaranteed lipschitz bound and itremains challenging to achieve strong results with provable guarantees.
Revised: However, these methods do not enforce a guaranteed lipschitz bound and it remains challenging to achieve strong results with provable guarantees.

Original: Using the composition property i.e. Lip( g ◦ h ) ≤ Lip( g ) Lip( h ) , we can construct a 1 -Lipschitzneural network by constraining each layer to be 1 -Lipschitz.
Revised: Using the composition property i.e. Lip( g ◦ h ) ≤ Lip( g ) Lip( h ) , we can construct a 1 -Lipschitz neural network by constraining each layer to be 1 -Lipschitz.

Original: However, a key difficulty with thisapproach is that because a 1 -Lipschitz layer can only reduce the gradient norm during backpropagation,for deeper networks, this results in small gradient values for layers closer to the input making trainingslow and difficult.
Revised: However, a key difficulty with this approach is that because a 1 -Lipschitz layer can only reduce the gradient norm during backpropagation, for deeper networks, this results in small gradient values for layers closer to the input making training slow and difficult.

Original: To address this problem, Anil et al.
Revised: To address this problem, Anil et al.

Original: [2018] introduce Gradient Norm Preserving(GNP) architectures where each layer preserves the gradient norm during backpropagation.
Revised: [2018] introduce Gradient Norm Preserving (GNP) architectures where each layer preserves the gradient norm during backpropagation.

Original: For1 -Lipschitz Convolutional Neural Networks (CNNs), this involves using orthogonal convolutions(convolution layers with an orthogonal Jacobian matrix)
Revised: For 1 -Lipschitz Convolutional Neural Networks (CNNs), this involves using orthogonal convolutions (convolution layers with an orthogonal Jacobian matrix)

Original: [Li et al., 2019b, Trockman and Kolter,2021, Singla and Feizi, 2021, Yu et al., 2022, Kiani et al., 2022] and using a class of GNP activationfunctions called HouseHolder activations [Singla et al., 2022]. 
Revised: [Li et al., 2019b, Trockman and Kolter, 2021, Singla and Feizi, 2021, Yu et al., 2022, Kiani et al., 2022] and using a class of GNP activation functions called HouseHolder activations [Singla et al., 2022]. [Table]

Revised: Among orthogonal convolutions

Original: [Table]
Revised: [Table]

Original: Among orthogonal convolutions, SOC [Singla and Feizi, 2021] achieves state-of-the-art resultson CIFAR-10, 100 [Singla et al., 2022].
Revised: SOC [Singla and Feizi [Table] achieves state-of-the-art results on CIFAR-10, 100 [Singla et al., 2022].

Original: SOC uses the following two mathematical properties toconstruct an orthogonal convolution: (a) the exponential of a skew symmetric matrix is orthogonaland (b) the matrix exponential can be computed using the exponential series.
Revised: SOC uses the following two mathematical properties to construct an orthogonal convolution: (a) the exponential of a skew symmetric matrix is orthogonal and (b) the matrix exponential can be computed using the exponential series.

Original: A drawback of usingthe exponential series is that it requires us to apply the convolution operation multiple times toachieve a reasonable approximation of an orthogonal matrix.
Revised: A drawback of using the exponential series is that it requires us to apply the convolution operation multiple times to achieve a reasonable approximation of an orthogonal matrix.

Original: Consequently, during backpropagation,computing the gradient with respect to the weights for a single SOC layer requires us to computethe weight gradient for each of these convolutions resulting in significant training overhead.
Revised: Consequently, during backpropagation, computing the gradient with respect to the weights for a single SOC layer requires us to compute the weight gradient for each of these convolutions resulting in significant training overhead.

Original: In thiswork, we show that because the matrix is skew symmetric, using some approximations, the gradientswith respect to weights can be computed in significantly reduced time with no loss in performance.
Revised: In this work, we show that because the matrix is skew symmetric, using some approximations, the gradients with respect to weights can be computed in significantly reduced time with no loss in performance.

Original: This enables us to reduce the training time per epoch using SOC by > 30% for networks with largenumber of SOC layers (Results in Table 1).
Revised: This enables us to reduce the training time per epoch using SOC by > 30% for networks with large number of SOC layers (Results in Table 1).

Original: Another limitation of 1 -Lipschitz CNNs is that their performance is often significantly below compared to that of standard CNNs.
Revised: Another limitation of 1 -Lipschitz CNNs is that their performance is often significantly below compared to that of standard CNNs.

Original: Recently, Singla et al.
Revised: Recently, Singla et al.

Original: [2022] introduced a procedure for certifyingrobustness by relaxing the orthogonality requirements of the last linear layer (i.e. the linear layermapping penultimate neurons to class logits) achieving state of the art results on CIFAR-10, 100[Krizhevsky, 2009].
Revised: [2022] introduced a procedure for certifying robustness by relaxing the orthogonality requirements of the last linear layer (i.e. the linear layer mapping penultimate neurons to class logits) achieving state of the art results on CIFAR-10,[Krizhevsky, 2009].

Original: Since MLPs are more expressive than linear layers, one would expect improvedstandard accuracy by replacing the last linear layer with them.
Revised: Since MLPs are more expressive than linear layers, one would expect improved standard accuracy by replacing the last linear layer with them.

Original: However, the resulting networks arenot 1 -Lipschitz and achieving high robustness guarantees ( provable robust accuracy ) is difficult.
Revised: However, the resulting networks are not 1 -Lipschitz and achieving high robustness guarantees ( provable robust accuracy ) is difficult.

Original: To certify robustness for these networks, note that since the mapping from input to penultimate layeris 1 -Lipschitz, the robustness certificate for the penultimate output also provides a certificate for theinput.
Revised: To certify robustness for these networks, note that since the mapping from input to penultimate layer is 1 -Lipschitz, the robustness certificate for the penultimate output also provides a certificate for the input.

Original: Thus, we first replace the linear layer mapping penultimate output to logits with a 1 -hiddenlayer MLP (Multi layer Perceptron) because such MLPs are easier to certify compared to deep MLPs.
Revised: Thus, we first replace the linear layer mapping penultimate output to logits with a 1 -hidden layer MLP (Multi layer Perceptron) because such MLPs are easier to certify compared to deep MLPs.

Original: To certify robustness for the MLP, we use the Curvature-based Robustness Certificate or CRC [Singlaand Feizi, 2020].
Revised: To certify robustness for the MLP, we use the Curvature-based Robustness Certificate or CRC [Singla and Feizi, 2020].

Original: To train MLP to have high robustness guarantees, we use a variant of adversarialtraining that only applies adversarial perturbations to the MLP (not the whole network).
Revised: To train MLP to have high robustness guarantees, we use a variant of adversarial training that only applies adversarial perturbations to the MLP (not the whole network).

Original: We call ourcertification procedure CRC - Lip .
Revised: We call our certification procedure CRC - Lip .

Original: This results in improved results for both the standard and also theprovable robust accuracy across all network architectures on CIFAR-10 ( ≥ 1 .
Revised: This results in improved results for both the standard and also the provable robust accuracy across all network architectures on CIFAR-10 ( ≥ 1 .

Original: 63% , ≥ 3 .
Revised: 63% , ≥ 3 .

Original: and
Revised: and

Original: CIFAR-100 ( ≥ 2 .
Revised: CIFAR-100 ( ≥ 2 .

Original: 49% , ≥ 2 .
Revised: 49% , ≥ 2 .

Original: 27% respectively).
Revised: 27% respectively).

Original: Results are in Tables 3 and 4.
Revised: Results are in Tables 3 and 4.

Original: While several works have attempted to construct novel and more expressive orthogonal convolutionlayers and GNP activation functions, current state-of-the-art 1 -Lipschitz CNNs still use poolinglayers by taking the max of different elements.
Revised: While several works have attempted to construct novel and more expressive orthogonal convolution layers and GNP activation functions, current state-of-the-art 1 -Lipschitz CNNs still use pooling layers by taking the max of different elements.

Original: In this work, we introduce a class of 1 -Lipschitzpooling layers called projection pooling using the following mathematical property:
Revised: In this work, we introduce a class of 1 -Lipschitz pooling layers called projection pooling using the following mathematical property:

Original: Given a manifold
Revised: Given a manifold

Original: M ⊂ R n and input x ∈ R n , the function d M : R n → R defined as the l 2 distance of x to Mis 1 -Lipschitz.
Revised: M ⊂ R n and input x ∈ R n , the function d M : R n → R defined as the l 2 distance of x to M is 1 -Lipschitz.

Original: Thus, to construct a pooling layer, we can first define a learnable manifold withparameters Θ (Example in Section 6).
Revised: Thus, to construct a pooling layer, we can first define a learnable manifold with parameters Θ (Example in Section 6).

Original: During training, for input x ∈ R n , the pooling layer simplyoutputs d M ( x ) ∈ R as the output, resulting in decrease in input dimension by a factor of n .
Revised: During training, for input x ∈ R n , the pooling layer simply outputs d M ( x ) ∈ R as the output, resulting in decrease in input dimension by a factor of n .

Original: Moreover,since the output d M ( x ) is also a function of Θ , Θ can be learned during training.
Revised: Moreover, since the output d M ( x ) is also a function of Θ , Θ can be learned during training.

Original: However, solvingfor the distance d M ( x ) can be difficult especially when M is a high-dimensional manifold.
Revised: However, solving for the distance d M ( x ) can be difficult especially when M is a high-dimensional manifold.

Original: To address this limitation, (a) we use 2D projection pooling layers that reduce the dimension by factorof 2 and (b) we construct these layers using piecewise linear curves for which the distance can becomputed efficiently by computing the minimum distance to all line segments and the connecting points (Example in Appendix Figure 2).
Revised: To address this limitation, (a) we use 2D projection pooling layers that reduce the dimension by factor of 2 and (b) we construct these layers using piecewise linear curves for which the distance can be computed efficiently by computing the minimum distance to all line segments and the connecting points (Example in Appendix Figure 2).

Original: If the curve is closed and without self-intersections, we canalso define a signed projection pooling for which the signs of the output for points inside and outsidethe region enclosed by the curve are different ( x and y in Figure 2).
Revised: If the curve is closed and without self-intersections, we can also define a signed projection pooling for which the signs of the output for points inside and outside the region enclosed by the curve are different ( x and y in Figure 2).

Original: This allows the subsequentlayers to distinguish between the inputs inside and outside the region.
Revised: This allows the subsequent layers to distinguish between the inputs inside and outside the region.

Original: In this work, we show somepreliminary results using a simple 2D projection pooling layer (Section 6).
Revised: In this work, we show some preliminary results using a simple 2D projection pooling layer (Section 6).

Original: We leave the problem ofconstructing high performance 2D projection pooling layers open for future research.
Revised: We leave the problem of constructing high performance 2D projection pooling layers open for future research.

Original: In summary, in this paper, we make the following contributions:
Revised: In summary, in this paper, we make the following contributions:

Original: • We introduce a method for faster computation of the weight gradient for SOC layers.
Revised: • We introduce a method for faster computation of the weight gradient for SOC layers.

Original: For deepernetworks, we observe reduction in trainingtime per epoch by
Revised: For deeper networks, we observe reduction in training

Revised: time per epoch by

Original: >
Revised: >

Original: (Table 1)
Revised: (Table 1)

Original: • We introduce a certification procedure called CRC - Lip that replaces the last linear layer with a 1hidden layer MLP and results in significantly improved standard and provable robust accuracy.
Revised: • We introduce a certification procedure called CRC - Lip that replaces the last linear layer with a 1hidden layer MLP and results in significantly improved standard and provable robust accuracy.

Original: Fordeeper networks ( ≥ 35 layers), we observe improvements of ≥ 5 .
Revised: For deeper networks ( ≥ 35 layers), we observe improvements of ≥ 5 .

Original: 84% , ≥ 8 . 00% in standard and≥ 8 .
Revised: 84% , ≥ 8 . 00% in standard and ≥ 8 .

Original: 76% , ≥ 8 .
Revised: 76% , ≥ 8 .

Original: 65% in provable robust accuracy ( l 2 radius 36 / 255 ) on CIFAR-10,100 respectively.•
Revised: 65% in provable robust accuracy ( l 2 radius 36 / 255 ) on CIFAR-10,100 respectively.

Original:  We introduce a large class of 1 -Lipschitz pooling layers called projection pooling using themathematical property that the l 2 distance d M ( x ) of an input x to the manifold M is 1 -Lipschitz.•
Revised: • We introduce a large class of 1 -Lipschitz pooling layers called projection pooling using the mathematical property that the l 2 distance d M ( x ) of an input x to the manifold M is 1 -Lipschitz.

Original:  On CIFAR-10, across all architectures, we achieve the best standard and provable robust accuracy(at 36 / 255 ) of 79 . 
Revised: • On CIFAR-10, across all architectures, we achieve the best standard and provable robust accuracy (at 36 / 255 ) of 79 . 57 , 67 .

Original: 13% respectively (gain of +1 .
Revised: 13% respectively (gain of +1 .

Original: 79% , +3 .
Revised: 79% , +3 .

Original: 82% from prior works).
Revised: 82% from prior works).

Original: Similarly,on CIFAR- 100 , we achieve 51 .
Revised: Similarly, on CIFAR- 100 , we achieve 51 .

Original: 27% ( +3 .
Revised: 27% ( +3 .

Original: 78% , +4 .
Revised: 78% , +4 .

Original: 75% from prior works).
Revised: 75% from prior works).

Original: These results establish new state-of-the-art results in the standard and provable robust accuracy on both datasets.
Revised: These results establish new state-of-the-art results in the standard and provable robust accuracy on both datasets.

Original:  Related work
Revised:  Related work

Original: Provable defenses against adversarial examples : For a provably robust classifier, we can guaranteethat its predictions remain constant within some region around the input.
Revised: Provable defenses against adversarial examples : For a provably robust classifier, we can guarantee that its predictions remain constant within some region around the input.

Original: Most of the existing methodsfor provable robustness either bound the Lipschitz constant or use convex relaxations [Singh et al. ,2017, 2018a, 2019a,b,c, Weng et al., 2018, Salman et al., 2019b, Zhang et al., 2019, 2018, Wonget al., 2018, Wong and Kolter, 2018, Singh et al., 2018b, Raghunathan et al., 2018, Dvijotham* et al. ,2018, Croce et al., 2019, Gowal et al., 2019, Dvijotham* et al., 2020, Lu and Kumar, 2020, Singlaand Feizi, 2020, Bunel et al., 2020, Leino et al., 2021, Leino and Fredrikson, 2021, Zhang et al. ,2021, 2022, Wang et al., 2021, Huang et al., 2021, Müller et al., 2021, Singh et al., 2021, Palma et al.
Revised: Most of the existing methods for provable robustness either bound the Lipschitz constant or use convex relaxations [Singh et al., 2017, 2018a, 2019a,b,c, Weng et al., 2018, Salman et al., 2019b, Zhang et al., 2019, 2018, Wong et al., 2018, Wong and Kolter, 2018, Singh et al., 2018b, Raghunathan et al., 2018, Dvijotham* et al., 2018, Croce et al., 2019, Gowal et al., 2019, Dvijotham* et al., 2020, Lu and Kumar, 2020, Singla and Feizi, 2020, Bunel et al., 2020, Leino et al., 2021, Leino and Fredrikson, 2021, Zhang et al., 2021, 2022, Wang et al., 2021, Huang et al., 2021, Müller et al., 2021, Singh et al., 2021, Palma et al., 2021].

Original: However, these methods are often not scalable to large neural networks while achieving highperformance.
Revised: However, these methods are often not scalable to large neural networks while achieving high performance.

Original: In contrast, randomized smoothing [Liu et al., 2018, Cao and Gong, 2017, Lécuyeret al., 2018, Li et al., 2019a, Cohen et al., 2019, Salman et al., 2019a, Levine et al., 2019, Kumaret al., 2020a,b, Salman et al., 2020, 2021] scales to large neural networks but is a probabilisticallycertified defense : certifying robustness with high probability requires generating a large number ofnoisy samples leading to high inference-time.
Revised: In contrast, randomized smoothing [Liu et al., 2018, Cao and Gong, 2017, Lécuyer et al., 2018, Li et al., 2019a, Cohen et al., 2019, Salman et al., 2019a, Levine et al., 2019, Kumar et al., 2020a,b, Salman et al., 2020, 2021] scales to large neural networks but is a probabilistically certified defense : certifying robustness with high probability requires generating a large number of noisy samples leading to high inference-time.

Original: The defense we propose in this work is deterministicand not comparable to randomized smoothing.
Revised: The defense we propose in this work is deterministic and not comparable to randomized smoothing.

Original: Provably Lipschitz neural networks : The class of Gradient Norm Preserving (GNP) and 1-Lipschitzfully connected neural networks was first introduced by Anil et al.
Revised: Provably Lipschitz neural networks : The class of Gradient Norm Preserving (GNP) and 1-Lipschitz fully connected neural networks was first introduced by Anil et al.

Original: To design each layer tobe GNP, they orthogonalize weight matrices and use a class of piecewise linear GNP activationscalled GroupSort .
Revised: To design each layer to be GNP, they orthogonalize weight matrices and use a class of piecewise linear GNP activations called GroupSort .

Original: Later, Singla et al.
Revised: Later, Singla et al.

Original: [2022] proved that for any piecewise linear GNP function to becontinuous, Jacobian matrices  a neighborhood must change via householder transformations, implying that GroupSort is a special case of more general HouseHolder activations.
Revised: [2022] proved that for any piecewise linear GNP function to be continuous, different Jacobian matrices in a neighborhood must change via householder transformations, implying that GroupSort is a special case of more general HouseHolder activations.

Original: Severalprevious works enforce Lipschitz constraints on convolution layers using spectral normalization,clipping or regularization [Cissé et al., 2017, Tsuzuku et al., 2018, Qian and Wegman, 2019, Gouket al., 2020, Sedghi et al., 2019].
Revised: Several previous works enforce Lipschitz constraints on convolution layers using spectral normalization, clipping or regularization [Cissé et al., 2017, Tsuzuku et al., 2018, Qian and Wegman, 2019, Gouk et al., 2020, Sedghi et al., 2019].

Original: However, these methods either enforce loose lipschitz bounds or donot scale to large networks.
Revised: However, these methods either enforce loose lipschitz bounds or do not scale to large networks.

Original: To ensure that the Lipschitz constraint on convolutional layers is tight,recent works construct convolution layers with an orthogonal Jacobian [Li et al., 2019b, Trockmanand Kolter, 2021, Singla and Feizi, 2021, Yu et al., 2022, Su et al., 2022, Kiani et al., 2022].
Revised: To ensure that the Lipschitz constraint on convolutional layers is tight, recent works construct convolution layers with an orthogonal Jacobian [Li et al., 2019b, Trockman and Kolter, 2021, Singla and Feizi, 2021, Yu et al., 2022, Su et al., 2022, Kiani et al., 2022].

Original: Theseapproaches avoid the aforementioned issues and allow training of large, provably 1 -Lipschitz CNNsachieving state-of-the-art results for deterministic l 2 robustness.
Revised: These approaches avoid the aforementioned issues and allow training of large, provably 1 -Lipschitz CNNs achieving state-of-the-art results for deterministic l 2 robustness.

Original:  Problem setup and Notation
Revised:  Problem setup and Notation

Original: For a vector v , v j denotes its j th element.
Revised: For a vector v , v j denotes its j th element.

Original: For a matrix A , A j, : and A : ,k denote the j th row and k th column respectively.
Revised: For a matrix A , A j, : and A : ,k denote the j th row and k th column respectively.

Original: Both A j, : and A : ,k are assumed to be column vectors (thus A j, : is the transpose of j th row of A ).
Revised: Both A j, : and A : ,k are assumed to be column vectors (thus A j, : is the transpose of j th row of A ).

Original: A j,k denotes the element in j th row and k th column of A .
Revised: A j,k denotes the element in j th row and k th column of A .

Original: A : j, : k denotes the matrix containing the first j rows and k columns of A .
Revised: A : j, : k denotes the matrix containing the first j rows and k columns of A .

Original: We define A : j = A : j, : and A j : =
Revised: We define A : j = A : j, : and A j : =

Original: A j : , : .
Revised: A j : , : .

Original: Similar notation applies to higher order tensors.
Revised: Similar notation applies to higher order tensors.

Original: I denotes the identity matrix, R to denote the field of real numbers.
Revised: I denotes the identity matrix, R to denote the field of real numbers.

Original: We construct a 1 -Lipschitz neural network, f : R d → R c ( d is the input dimension, c is the number of classes) by composing 1 -Lipschitz convolution layers and GNP activation functions.
Revised: We construct a 1 -Lipschitz neural network, f : R d → R c ( d is the input dimension, c is the number of classes) by composing 1 -Lipschitz convolution layers and GNP activation functions.

Original: We often use the abbreviation f i − f j : R d → R to denote the function so that:
Revised: We often use the abbreviation f i − f j : R d → R to denote the function so that:

Original: [Equation]
Revised: [Equation]

Original: For a matrix A ∈ R q × r and a tensor B ∈ R p × q × r , −→ A denotes the vector constructed by stacking therows of A and −→ B by stacking the vectors −−−→ B j, : , : , j ∈ [ p − 1] so that:
Revised: For a matrix A ∈ R q × r and a tensor B ∈ R p × q × r , −→ A denotes the vector constructed by stacking the rows of A and −→ B by stacking the vectors −−−→ B j, : , : , j ∈ [ p − 1] so that:

Original: [Equation]
Revised: [Equation]

Original: For a 2 D convolution filter, L ∈ R p × q × r × s and input X ∈ R q × n × n , we use L ⋆ X ∈ R p × n × n todenote the convolution of filter L with X .
Revised: For a 2 D convolution filter, L ∈ R p × q × r × s and input X ∈ R q × n × n , we use L ⋆ X ∈ R p × n × n to denote the convolution of filter L with X .

Original: We use the the notation L ⋆ i X ≜ L ⋆ i − 1 ( L ⋆ X ) and
Revised: We use the the notation L ⋆ i X ≜ L ⋆ i − 1 ( L ⋆ X ) and

Original: L ⋆ 0 X
Revised: L ⋆ 0 X

Original: = X .
Revised: = X .

Original: Unless specified, we assume zero padding and stride 1 in each direction.
Revised: Unless specified, we assume zero padding and stride 1 in each direction.

Original:  Faster gradient computation for Skew Orthogonal Convolutions
Revised:  Faster gradient computation for Skew Orthogonal Convolutions

Original: Among the existing orthogonal convolution layers in the literature, Skew Orthogonal Convolutions(or SOC) by Singla and Feizi [2021] achieves state-of-the-art results on CIFAR-10,100 [Singlaet al., 2022].
Revised: Among the existing orthogonal convolution layers in the literature, Skew Orthogonal Convolutions (or SOC) by Singla and Feizi [2021] achieves state-of-the-art results on CIFAR-10,100 [Singla et al., 2022].

Original: SOC first constructs a convolution filter L whose Jacobian is skew-symmetric.
Revised: SOC first constructs a convolution filter L whose Jacobian is skew-symmetric.

Original: This isfollowed by a convolution exponential [Hoogeboom et al., 2020] operation.
Revised: This is followed by a convolution exponential [Hoogeboom et al., 2020] operation.

Original: Since the exponential ofa skew-symmetric matrix is orthogonal, the Jacobian of the resulting layer is an orthogonal matrix.
Revised: Since the exponential of a skew-symmetric matrix is orthogonal, the Jacobian of the resulting layer is an orthogonal matrix.

Original: However, a drawback of this procedure is that convolution exponential requires multiple convolutionoperations per SOC layer to achieve a reasonable approximation of the orthogonal Jacobian.
Revised: However, a drawback of this procedure is that convolution exponential requires multiple convolution operations per SOC layer to achieve a reasonable approximation of the orthogonal Jacobian.

Original: Consequently, if we use k convolution operations in the SOC layer during forward pass, we need to computethe gradient with respect to the weights L (called convolution weight gradient ) per convolutionoperation ( k times) which can lead to slower training time especially when the number of SOC layersis large.
Revised: Consequently, if we use k convolution operations in the SOC layer during forward pass, we need to compute the gradient with respect to the weights L (called convolution weight gradient ) per convolution operation ( k times) which can lead to slower training time especially when the number of SOC layers is large.

Original: To address this limitation, we show that even if we use k convolutions in the forward pass ofan SOC layer, the weight gradient for the layer can be computed using a single convolution weightgradient during backpropagation leading to significant reduction in training time.
Revised: To address this limitation, we show that even if we use k convolutions in the forward pass of an SOC layer, the weight gradient for the layer can be computed using a single convolution weight gradient during backpropagation leading to significant reduction in training time.

Original: For simplicity, let us first consider the case of an orthogonal fully connected layer (i.e. not convolution)with the same input and output size ( n ).
Revised: For simplicity, let us first consider the case of an orthogonal fully connected layer (i.e. not convolution) with the same input and output size ( n ).

Original: Later, we will see that our analysis leads to improvements fororthogonal convolutional layers.
Revised: Later, we will see that our analysis leads to improvements for orthogonal convolutional layers.

Original: Further, assume that the weights i.e. A ∈ R n × n are skew-symmetrici.e. A = − A T and given the input x ∈ R n , the output z ∈ R n is computed as follows:
Revised: Further, assume that the weights i.e. A ∈ R n × n are skew-symmetric i.e. A = − A T and given the input x ∈ R n , the output z ∈ R n is computed as follows:

Original: [Equation]
Revised: [Equation]

Original: x + b
Revised: where A

Original: [Equation]
Revised: [Equation]

Original: where A = − A T (1)
Revised: A [Equation]

Original: We approximate the exponential series: exp( A ) =
Revised: We approximate the exponential series: exp( A ) =

Original: (cid:80) ∞
Revised: (cid:80) ∞

Original: i =

Revised: i =

Original: A i /i !
Revised: A i /i !

Original: using a finite number of terms ( k ).
Revised: using a finite number of terms ( k ).

Original: Forward pass ( z ) :
Revised: Forward pass ( z ) :

Original: To compute z during the forward pass, we use the following iterations:
Revised: To compute z during the forward pass, we use the following iterations:

Original: [Equation]
Revised: [Equation]

Original: It can be shown that z = u (0) (Appendix C).
Revised: It can be shown that z = u (0) (Appendix C).

Original: During backpropagation, given the gradient of loss ℓw.r.t.
Revised: During backpropagation, given the gradient of loss ℓ w.r.t.

Original: layer output z i.e. ∇ z ℓ , we want to compute ∇ x ℓ (input gradient) and ∇ A ℓ (weight gradient).
Revised: layer output z i.e. ∇ z ℓ , we want to compute ∇ x ℓ (input gradient) and ∇ A ℓ (weight gradient).

Original: Input gradient ( ∇ x ℓ ) : To compute ∇ x ℓ , observe that z is a linear function of x .
Revised: Input gradient ( ∇ x ℓ ) : To compute ∇ x ℓ , observe that z is a linear function of x .

Original: Thus, using thechain rule, skew-symmetricity ( A T = − A ) and the property (cid:0) A i (cid:1) T = (cid:0) A T (cid:1) i , we have:
Revised: Thus, using the chain rule, skew-symmetricity ( A T = − A ) and the property (cid:0) A i (cid:1) T = (cid:0) A T (cid:1) i , we have:

Original: [Equation]
Revised: [Equation]

Original: We can again approximate the exponential series using the same number of terms as in the forwardpass i.e. k .
Revised: We can again approximate the exponential series using the same number of terms as in the forward pass i.e. k .

Original: To compute the finite term approximation, we use the following iterations:
Revised: To compute the finite term approximation, we use the following iterations:

Original: [Equation]
Revised: [Equation]

Original: Similar to forward pass, it can be shown that ∇ x ℓ = v (0) (Appendix C).
Revised: Similar to forward pass, it can be shown that ∇ x ℓ = v (0) (Appendix C).

Original: Weight gradient ( ∇ A ℓ ) :
Revised: Weight gradient ( ∇ A ℓ ) :

Original: We first derive the exact expression for ∇ A ℓ in the Theorem below:
Revised: We first derive the exact expression for ∇ A ℓ in the Theorem below:

Original: Theorem 1 The gradient of the loss function ℓ w.r.t A i.e. ∇ A ℓ is given by:
Revised: Theorem 1 The gradient of the loss function ℓ w.r.t A i.e. ∇ A ℓ is given by:

Original: [Equation]
Revised: [Equation]

Original: where v ( i ) is defined as in equation (3) .
Revised: where v ( i ) is defined as in equation (3) .

Original: Note that the first outer product i.e. (cid:0) A i − 1 x (cid:1) (cid:0) v ( i ) (cid:1) T and the second i.e. (cid:0) v ( i ) (cid:1) (cid:0)
Revised: Note that the first outer product i.e. (cid:0) A i − 1 x (cid:1) (cid:0) v ( i ) (cid:1) T and the second i.e. (cid:0) v ( i ) (cid:1) (cid:0)

Original: A
Revised: A

Original: i − 1 x (cid:1) T aretranspose of each other implying that each term in the summation is skew-symmetric.
Revised: i − 1 x (cid:1) T are transpose of each other implying that each term in the summation is skew-symmetric.

Original: Although theseouter products can be computed in a straightforward way for orthogonal fully connected layers, this isnot the case for orthogonal convolution layers (SOC in this case).
Revised: Although these outer products can be computed in a straightforward way for orthogonal fully connected layers, this is not the case for orthogonal convolution layers (SOC in this case).

Original: This is because, for a convolutionfilter L ∈ R p × q × r × s , the term analogous to A i − 1 x is a patch of size q × r × s and that analogous tov ( i ) is another patch of size p × 1 × 1 resulting in an outer product of the desired size p × q × r × s perpatch .
Revised: This is because, for a convolution filter L ∈ R p × q × r × s , the term analogous to A i − 1 x is a patch of size q × r × s and that analogous to v ( i ) is another patch of size p × 1 × 1 resulting in an outer product of the desired size p × q × r × s per patch .

Original: Thus, for SOC layers, each term inside the summation is computed by summing over the outerproducts for all such patches.
Revised: Thus, for SOC layers, each term inside the summation is computed by summing over the outer products for all such patches.

Original: For large input sizes, the number of such patches can often be large,making this computation expensive.
Revised: For large input sizes, the number of such patches can often be large, making this computation expensive.

Original: To address this limitation, we use the following approximation:
Revised: To address this limitation, we use the following approximation:

Original: [Equation]
Revised: [Equation]

Original: In Appendix B, we show that the above approximation is principled because after subtracting theexact and approximation gradients (equations (4) and (5)) and simplifying, each term in the resultingseries is divided by a large value in its denominator ( ≈ 0 ).
Revised: In Appendix B, we show that the above approximation is principled because after subtracting the exact and approximation gradients (equations (4) and (5)) and simplifying, each term in the resulting series is divided by a large value in its denominator ( ≈ 0 ).

Original: This approximation is useful because inequation (5), the outer product needs to be computed once whereas in equation (4), the outer productsneed to be computed k times.
Revised: This approximation is useful because in equation (5), the outer product needs to be computed once whereas in equation (4), the outer products need to be computed k times.

Original: Also, both u (1) and v (1) are computed while computing u (0) = zduring the forward pass and v (0) =
Revised: Also, both u (1) and v (1) are computed while computing u (0) = z during the forward pass and v (0) =

Original: ∇ x z during the backward pass using the recurrences in equations(2) and (3).
Revised: ∇ x z during the backward pass using the recurrences in equations (2) and (3).

Original: Thus, we can simply store u (1) , v (1) during the forward,backward pass respectively sothat ∇ A ℓ can be computed directly using equation (5).
Revised: Thus, we can simply store u (1) , v (1) during the forward,backward pass respectively so that ∇ A ℓ can be computed directly using equation (5).

Original: In our experiments, we observe that this leadsto significant reduction in training time with almost no drop in performance (Tables 1, 3, 4).
Revised: In our experiments, we observe that this leads to significant reduction in training time with almost no drop in performance (Tables 1, 3, 4).

Original:  Curvature-based Robustness Certificate
Revised:  Curvature-based Robustness Certificate

Original: A key property of 1 -Lipschitz CNNs is that the output of each layer is 1 -Lipschitz with respect tothe input.
Revised: A key property of 1 -Lipschitz CNNs is that the output of each layer is 1 -Lipschitz with respect to the input.

Original: Given an input x ∈ R d , consider the penultimate output g ( x ) ∈ R m and logits f ( x ) ∈ R cfor some 1 -Lipschitz CNN.
Revised: Given an input x ∈ R d , consider the penultimate output g ( x ) ∈ R m and logits f ( x ) ∈ R c for some 1 -Lipschitz CNN.

Original: Existing robustness certificates [Li et al., 2019b, Singla et al., 2022] relyon the linearity of the function from g ( x ) → f ( x ) .
Revised: Existing robustness certificates [Li et al., 2019b, Singla et al., 2022] rely on the linearity of the function from g ( x ) → f ( x ) .

Original: However, since MLPs have higher expressivepower than linear functions [Cybenko, 1989], one way to improve performance could be to replacethis mapping with MLPs.
Revised: However, since MLPs have higher expressive power than linear functions [Cybenko, 1989], one way to improve performance could be to replace this mapping with MLPs.

Original: However, certifying robustness for deep MLPs is difficult.
Revised: However, certifying robustness for deep MLPs is difficult.

Original: To address these challenges, we first replace the mapping from g ( x ) → f ( x ) with a 1 -hiddenlayer MLP because they are easier to certify compared to deeper networks.
Revised: To address these challenges, we first replace the mapping from g ( x ) → f ( x ) with a 1 -hidden layer MLP because they are easier to certify compared to deeper networks.

Original: Because computingexact certificates for ReLU networks is known to be NP-complete [Sinha et al., 2018], we use thedifferentiable Softplus activation [Dugas et al., 2000] to certify robustness.
Revised: Because computing exact certificates for ReLU networks is known to be NP-complete [Sinha et al., 2018], we use the differentiable Softplus activation [Dugas et al., 2000] to certify robustness.

Original: To certify robustness,we use the Curvature-based Robustness Certificate or CRC [Singla and Feizi, 2020] because itprovides exact certificates for a significant fraction of inputs for shallow MLPs.
Revised: To certify robustness, we use the Curvature-based Robustness Certificate or CRC [Singla and Feizi, 2020] because it provides exact certificates for a significant fraction of inputs for shallow MLPs.

Original: We provide a briefreview of CRC in Appendix Section D. Let g : R d → R m be a 1 -Lipschitz continuous function andh : R m → R c be a 1 -hidden layer MLP such that f = h ◦ g .
Revised: We provide a brief review of CRC in Appendix Section D. Let g : R d → R m be a 1 -Lipschitz continuous function and h : R m → R c be a 1 -hidden layer MLP such that f = h ◦ g .

Original: Further, let l be the predicted classfor input x i.e. f l ( x ) ≥ max i ̸
Revised: Further, let l be the predicted class for input x i.e. f l ( x ) ≥ max i ̸

Original: = l f i ( x ) .
Revised: = l f i ( x ) .

Original: Since g is 1 -Lipschitz, it can be shown that if h is provablyrobust in an l 2 radius R around input g ( x ) , then f is also provably robust in the l 2 radius R around x .
Revised: Since g is 1 -Lipschitz, it can be shown that if h is provably robust in an l 2 radius R around input g ( x ) , then f is also provably robust in the l 2 radius R around x .

Original: The resulting procedure is called CRC - Lip and is given in the following proposition:
Revised: The resulting procedure is called CRC - Lip and is given in the following proposition:

Original: [Figure]
Revised: [Figure]

Original: Proposition 1 (CRC - Lip) For input x such that f l ( x ) ≥ max i ̸ =
Revised: Proposition 1 (CRC - Lip) For input x such that f l ( x ) ≥ max i ̸ =

Original: l f i ( x ) , let R be the robustnesscertificate for h using input g ( x ) , then R is also the robustness certificate for the function f :
Revised: l f i ( x ) , let R be the robustness certificate for h using input g ( x ) , then R is also the robustness certificate for the function f :

Original: [Equation]
Revised: [Equation]

Original: Proof is in Appendix A.2.
Revised: Proof is in Appendix A.2.

Original: In our experiments, we find that although replacing the linear layer with an
Revised: In our experiments, we find that although replacing the linear layer with an

Original: MLP achieves high standard accuracy, directly using the above certificate often leads to very smallrobustness certificates and thus low certified robust accuracy.
Revised: MLP achieves high standard accuracy, directly using the above certificate often leads to very small robustness certificates and thus low certified robust accuracy.

Original: To address this problem, we introduce(a) an adversarial training procedure that only applies to the input of the MLP g ( x ) (i.e. not the inputof the neural network x ) and (b) curvature regularization to reduce the curvature of the MLP.
Revised: To address this problem, we introduce (a) an adversarial training procedure that only applies to the input of the MLP g ( x ) (i.e. not the input of the neural network x ) and (b) curvature regularization to reduce the curvature of the MLP.

Original: It wasshown in Singla and Feizi [2021] that combining adversarial training with curvature regularizationleads to significantly improved provable robust accuracy with small reduction in standard accuracy.
Revised: It was shown in Singla and Feizi [2021] that combining adversarial training with curvature regularization leads to significantly improved provable robust accuracy with small reduction in standard accuracy.

Original: This results in the following loss function for training:
Revised: This results in the following loss function for training:

Original: [Equation]
Revised: [Equation]

Original: In the above equation, Ω denote the parameters of the MLP (i.e. h ), Ψ are the parameters of the1 -Lipschitz function g , K h is the bound on the curvature of the MLP, ρ denotes the l 2 perturbationradius and γ denotes the curvature regularization coefficient.
Revised: In the above equation, Ω denote the parameters of the MLP (i.e. h ), Ψ are the parameters of the 1 -Lipschitz function g , K h is the bound on the curvature of the MLP, ρ denotes the l 2 perturbation radius and γ denotes the curvature regularization coefficient.

Original: The curvature bound K h is the same asin Singla and Feizi [2020].
Revised: The curvature bound K h is the same as in Singla and Feizi [2020].

Original: Also, since we apply adversarial perturbations in the penultimate layer, wealso need to backpropagate through this procedure to enable training of previous layers.
Revised: Also, since we apply adversarial perturbations in the penultimate layer, we also need to backpropagate through this procedure to enable training of previous layers.

Original: To this end,we simply use an identity map (same gradient output as the gradient input) to backpropagate throughthe adversarial training procedure and find that it works well in practice, achieving significantlybetter results compared to the state-of-the-art.
Revised: To this end, we simply use an identity map (same gradient output as the gradient input) to backpropagate through the adversarial training procedure and find that it works well in practice, achieving significantly better results compared to the state-of-the-art.

Original: It is possible that a more principled method ofbackpropagation may lead to better results and we leave that avenue open for future research.
Revised: It is possible that a more principled method of backpropagation may lead to better results and we leave that avenue open for future research.

Original:  Projection pooling layers
Revised:  Projection pooling layers

Original: In 1 -Lipschitz CNNs, pooling is usually carried out as follows: given input X ∈ R q × r × r ( r iseven), we first use rearrangement [Jacobsen et al., 2018] illustrated in Figure 1b to construct
Revised: In 1 -Lipschitz CNNs, pooling is usually carried out as follows: given input X ∈ R q × r × r ( r is even), we first use rearrangement [Jacobsen et al., 2018] illustrated in Figure 1b to construct

Original: X ′ ∈ R 4 q × ( r/ 2) ×
Revised: X ′ ∈ R 4 q × ( r/ 2) ×

Original: ( r/ 2) .
Revised: ( r/ 2) .

Original: Next, we apply an orthogonal convolution which gives an output of the samesize Z ∈ R 4 q × ( r/ 2) ×
Revised: Next, we apply an orthogonal convolution which gives an output of the same size Z ∈ R 4 q × ( r/ 2) ×

Original: ( r/ 2) and divide it into two tensors of equal sizes (along the channel dimension)giving Z :2 q ∈ R 2 q × ( r/ 2) × ( r/ 2) and Z 2 q : ∈ R 2 q × ( r/ 2) × ( r/ 2) .
Revised: ( r/ 2) and divide it into two tensors of equal sizes (along the channel dimension) giving Z :2 q ∈ R 2 q × ( r/ 2) × ( r/ 2) and Z 2 q : ∈ R 2 q × ( r/ 2) × ( r/ 2) .

Original: We then define max( Z :2 q , Z 2 q : ) (oreither one of Z :2 q , Z 2 q : ) as the output of the pooling layer.
Revised: We then define max( Z :2 q , Z 2 q : ) (or either one of Z :2 q , Z 2 q : ) as the output of the pooling layer.

Original: Although max is 1 -Lipschitz, its expressive power is limited.
Revised: Although max is 1 -Lipschitz, its expressive power is limited.

Original: For example, consider the 1 -Lipschitzfunction ∥ x, y ∥ 2{+
+} = (cid:112) x
Revised: For example, consider the 1 -Lipschitz function ∥ x, y ∥ {+
+} = (cid:112) x

Original: 2 + y 2 .
Revised: 2 + y 2 .

Original: It is easy to see that if x = y , the error between ∥ x, y ∥ 2 andmax( x, y ) can be arbitrarily large for large values of x, y .
Revised: It is easy to see that if x = y , the error between ∥ x, y ∥ 2 and max( x, y ) can be arbitrarily large for large values of x, y .

Original: To address such limitations, we constructexpressive pooling layers using the following mathematical property:
Revised: To address such limitations, we construct expressive pooling layers using the following mathematical property:

Original: Theorem 2 Given x ∈ R n and manifold M ⊂ R n , the distance function (in l 2 norm) is 1-Lipschitz:
Revised: Theorem 2 Given x ∈ R n and manifold M ⊂ R n , the distance function (in l 2 norm) is 1-Lipschitz:

Original: [Equation]
Revised: [Equation]

Original: The above theorem provides a powerful method for constructing a learnable pooling layer by selectinga learnable manifold M Θ ⊂ R r
Revised: The above theorem provides a powerful method for constructing a learnable pooling layer by selecting a learnable manifold M Θ ⊂ R r

Original: ( Θ denotes the set of learnable parameters).
Revised: ( Θ denotes the set of learnable parameters).

Original: To apply the pooling
Revised: To apply the pooling

Original: [Table]
Revised: [Table]

Original: operation to an input x ∈ R n , we simply output its l 2 distance to the manifold M Θ denoted byd M Θ ( x ) .
Revised: operation to an input x ∈ R n , we simply output its l 2 distance to the manifold M Θ denoted by d M Θ ( x ) .

Original: Since d M Θ ( x ) ∈ R while x ∈ R n , this operation reduces the input size by a factor of n .
Revised: Since d M Θ ( x ) ∈ R while x ∈ R n , this operation reduces the input size by a factor of n .

Original: As an example, let M u = { u } .
Revised: As an example, let M u = { u } .

Original: Here u ∈ R n is the learnable parameter and the distance functiond M u ( x ) =
Revised: Here u ∈ R n is the learnable parameter and the distance function d M u ( x ) =

Original: ∥ x − u ∥ 2 .
Revised: ∥ x − u ∥ 2 .

Original: Even in this very simple case, for u = 0 , d M u ( x ) =
Revised: Even in this very simple case, for u = 0 , d M u ( x ) =

Original: ∥ x ∥ 2 and for n = 2 , thisfunction can learn to represent the function ∥ x, y ∥ 2 discussed earlier exactly.
Revised: ∥ x ∥ 2 and for n = 2 , this function can learn to represent the function ∥ x, y ∥ 2 discussed earlier exactly.

Original: We also prove that asigned l 2 distance function can be defined when M satisfies certain properties:
Revised: We also prove that a signed l 2 distance function can be defined when M satisfies certain properties:

Original: Corollary 

Revised: Corollary

Original: If M is a connected manifold that divides R n into two nonempty connected sets A and
Revised: If M is a connected manifold that divides R n into two nonempty connected sets A and

Original: B such that A ∩ B = ϕ
Revised: B such that A ∩ B = ϕ

Original: and every path from a ∈ A and b ∈ B intersects a point on M , then thereexists a signed l 2 distance function with different signs in A and B .
Revised: and every path from a ∈ A and b ∈ B intersects a point on M , then there exists a signed l 2 distance function with different signs in A and B .

Original: Proofs of Theorems 2 and Corollary 1 are given in Appendix A.3 and A.4 respectively.
Revised: Proofs of Theorems 2 and Corollary 1 are given in Appendix A.3 and A.4 respectively.

Original: In practice, computing d M can be difficult for high-dimensional M .
Revised: In practice, computing d M can be difficult for high-dimensional M .

Original: To tackle this issue, we use 2 Dpooling layers where M is defined to be a piecewise linear curve in 2 D .
Revised: To tackle this issue, we use 2 D pooling layers where M is defined to be a piecewise linear curve in 2 D .

Original: The l 2 distance can thenbe computed by finding the minimum distance to all the line segments and connecting points.
Revised: The l 2 distance can then be computed by finding the minimum distance to all the line segments and connecting points.

Original: Weemphasize that even in this relatively simple case, M can have large number of parameters and d Mcan still be efficient to compute because these individual distances can be computed in parallel.
Revised: We emphasize that even in this relatively simple case, M can have large number of parameters and d M can still be efficient to compute because these individual distances can be computed in parallel.

Original: Weuse M θ defined below (illustrated in Figure 1b, lines L 1 , L 2 correspond to ϕ
Revised: We use M θ defined below (illustrated in Figure 1b, lines L 1 , L 2 correspond to ϕ

Original: =
Revised: =

Original: + θ, − θ ):
Revised: + θ, − θ ):

Original: [Equation]
Revised: [Equation]

Original: In each colored region (Figure 1b), d M θ can be computed using the following:
Revised: In each colored region (Figure 1b), d M θ can be computed using the following:

Original: [Equation]
Revised: [Equation]

Original: To apply projection pooling on Z :2 q , Z 2 q : discussed previously, we output d M θ ( Z :2 q , Z 2 q : ) .
Revised: To apply projection pooling on Z :2 q , Z 2 q : discussed previously, we output d M θ ( Z :2 q , Z 2 q : ) .

Original:  Experiments
Revised:  Experiments

Original: We perform experiments under the setting of provably robust image classification on CIFAR-10and CIFAR-100 datasets.
Revised: We perform experiments under the setting of provably robust image classification on CIFARand CIFAR-100 datasets.

Original: We use the LipConvnet -5, 10, 15, . . .
Revised: We use the LipConvnet -5, 10, 15, . . .

Original: , 40 architectures for comparison.
Revised: , 40 architectures for comparison.

Original: We use SOC as the orthogonal convolution and MaxMin as the activation in all architectures.
Revised: We use SOC as the orthogonal convolution and MaxMin as the activation in all architectures.

Original: Allexperiments were performed using 1 NVIDIA GeForce RTX 2080 Ti GPU.
Revised: All experiments were performed using 1 NVIDIA GeForce RTX 2080 Ti GPU.

Original: All networks were trainedfor 200 epochs with initial learning rate of 0.1, dropped by a factor of 0.1 after 100 and 150 epochs.
Revised: All networks were trained for 200 epochs with initial learning rate of 0.1, dropped by a factor of 0.1 after 100 and 150 epochs.

Original: For adversarial training with curvature regularization, we use ρ = 36 / 255 (0 . 1411) , γ = 0 .
Revised: For adversarial training with curvature regularization, we use ρ = 36 / 255 (0 . 1411) , γ = 0 .

Original: 5 for
Revised: 5 for

Original: CIFAR- 10 and ρ = 0 .
Revised: CIFAR- 10 and ρ = 0 .

Original: 2 , γ = 0 .
Revised: 2 , γ = 0 .

Original: 75 for CIFAR- 100 .
Revised: 75 for CIFAR- 100 .

Original: We find that certifying robustness using CRC - Lipis computationally expensive for CIFAR-100 due to large number of classes.
Revised: We find that certifying robustness using CRC - Lip is computationally expensive for CIFAR-100 due to large number of classes.

Original: To address this issue,we only use classes with top- 10 logits (instead of all 100 classes) for CIFAR-100 (Table 4).
Revised: To address this issue, we only use classes with top- 10 logits (instead of all 100 classes) for CIFAR-100 (Table 4).

Original: Since
Revised: Since

Original: CRC - Lip requires us to solve a convex optimization, we consider a certificate to be valid if the inputis correctly classified and gradient at the optimal solution is ≤ 10 − 6 ( 0 otherwise).
Revised: CRC - Lip requires us to solve a convex optimization, we consider a certificate to be valid if the input is correctly classified and gradient at the optimal solution is ≤ 10 − 6 ( 0 otherwise).

Original: All results are
Revised: All results are

Original: [Table]
Revised: [Table]

Original: reported using the complete test sets of CIFAR- 10 and CIFAR- 100 .
Revised: reported using the complete test sets of CIFAR- 10 and CIFAR- 100 .

Original: We compare the provable robustaccuracy using 3 different l 2 perturbation radii: 36 / 255 , 72 / 255 , 108 / 255 .
Revised: We compare the provable robust accuracy using 3 different l 2 perturbation radii: 36 / 255 , 72 / 255 , 108 / 255 .

Original: Table details: For the baseline ("Baseline" in Tables 3, 4), we use the standard max pooling with thecertificate based on LLN [Singla et al., 2022] due to their superior performance over prior works.
Revised: Table details: For the baseline ("Baseline" in Tables 3, 4), we use the standard max pooling with the certificate based on LLN [Singla et al., 2022] due to their superior performance over prior works.

Original: In
Revised: In

Original: Tables 3 and 4, for each architecture, " + Fast " adds faster gradient computation, " + CRC " replacesmax pooling with projection pooling (equation (9)) and replaces the last linear layer with a 1 -hiddenlayer MLP ( CRC - Lip certificate) while also using faster gradients.
Revised: Tables 3 and 4, for each architecture, " + Fast " adds faster gradient computation, " + CRC " replaces max pooling with projection pooling (equation (9)) and replaces the last linear layer with a 1 -hidden layer MLP ( CRC - Lip certificate) while also using faster gradients.

Original: For each architecture, the columns"Increase (Standard)" and "Increase (36/255)" denote the increase in standard and provable robustaccuracy relative to "Baseline" standard and provable robust accuracy ( 36 / 255 ).
Revised: For each architecture, the columns "Increase (Standard)" and "Increase (36/255)" denote the increase in standard and provable robust accuracy relative to "Baseline" standard and provable robust accuracy ( 36 / 255 ).

Original: Results where
Revised: Results where

Original: Projection pooling and CRC - Lip are added separately are given in Appendix Tables 5 and 6.
Revised: Projection pooling and CRC - Lip are added separately are given in Appendix Tables 5 and 6.

Original: LipConvnet Architecture: We use a 1 -Lipschitz CNN architecture called LipConvnet - n where n isa multiple of 5 and n + 1 is the total number of convolution layers.
Revised: LipConvnet Architecture: We use a 1 -Lipschitz CNN architecture called LipConvnet - n where n is a multiple of 5 and n + 1 is the total number of convolution layers.

Original: It consists of an initial SOC layerthat expands the number of channels from 3 to 32 .
Revised: It consists of an initial SOC layer that expands the number of channels from 3 to 32 .

Original: This is followed by 5 blocks that reduce spatialdimensions (height and width) by half while doubling the number of channels.
Revised: This is followed by 5 blocks that reduce spatial dimensions (height and width) by half while doubling the number of channels.

Original: The architecture issummarized in Tables 2a and 2b.
Revised: The architecture is summarized in Tables 2a and 2b.

Original: The last layer outputs the class logits and is either a linear layer inwhich case we use LLN to certify robustness or a single-hidden layer MLP where we use CRC - Lip .
Revised: The last layer outputs the class logits and is either a linear layer in which case we use LLN to certify robustness or a single-hidden layer MLP where we use CRC - Lip .

Original: Correcting the certificates : Since SOC is an approximation, the Lipschitz constant of each SOClayer can be slightly more than 1 and if we use a large number of SOC layers (e.g. 41 in LipConvnet40 ), the Lipschitz constant of the full network (Lip( f )) can be significantly larger than 1 .
Revised: Correcting the certificates : Since SOC is an approximation, the Lipschitz constant of each SOC layer can be slightly more than 1 and if we use a large number of SOC layers (e.g. 41 in LipConvnet40 ), the Lipschitz constant of the full network (Lip( f )) can be significantly larger than 1 .

Original: To mitigatethis issue, we take the following steps: (a) We use a large number of terms ( k = 15 ) during test timeto approximate the exponential which results in a small approximation error (using the error bound in
Revised: To mitigate this issue, we take the following steps: (a) We use a large number of terms ( k = 15 ) during test time to approximate the exponential which results in a small approximation error (using the error bound in

Original: Singla and Feizi [2021]) and (b) We compute Lip( f ) by multiplying the lipschitz constant of all SOClayers (using the power method) and then divide the certificate by Lip( f ) .
Revised: Singla and Feizi [2021]) and (b) We compute Lip( f ) by multiplying the lipschitz constant of all SOC layers (using the power method) and then divide the certificate by Lip( f ) .

Original: [Table]
Revised: [Table]

Original: 7.1 Results using Faster gradient computation
Revised: 7.1 Results using Faster gradient computation

Original: We show the reduction in training time per epoch (in seconds) on CIFAR-10 in Table 1 and CIFAR100 in Appendix Table 8.
Revised: We show the reduction in training time per epoch (in seconds) on CIFAR-10 in Table 1 and CIFAR100 in Appendix Table 8.

Original: In both Tables, we observe that for deeper networks ( ≥ 25 layers), thereduction in time per epoch is ≈ 30% .
Revised: In both Tables, we observe that for deeper networks ( ≥ 25 layers), the reduction in time per epoch is ≈ 30% .

Original: The corresponding standard and provable robust accuracynumbers are given in Tables 3 (CIFAR-10) and 4 (CIFAR-100) in the row " + Fast ".
Revised: The corresponding standard and provable robust accuracy numbers are given in Tables 3 (CIFAR-10) and 4 (CIFAR-100) in the row " + Fast ".

Original: From the columns"Increase (Standard)" and "Increase (36/255)", we observe that the performance is similar to thebaseline across all network architectures.
Revised: From the columns "Increase (Standard)" and "Increase (36/255)", we observe that the performance is similar to the baseline across all network architectures.

Original: For deeper networks: LipConvnet - 35 , 40 , we observe anincrease in performance for CIFAR- 10 and small decrease ( < 0 .
Revised: For deeper networks: LipConvnet - 35 , 40 , we observe an increase in performance for CIFAR- 10 and small decrease ( < 0 .

Original: 64% ) for CIFAR- 100 .
Revised: 64% ) for CIFAR- 100 .

Original: 7.2 Results using projection pooling and CRC-Lip
Revised: 7.2 Results using projection pooling and CRC-Lip

Original: We observe that using CRC and projection pooling (row " + CRC ") leads to significant improvementsin performance across all LipConvnet architectures.
Revised: We observe that using CRC and projection pooling (row " + CRC ") leads to significant improvements in performance across all LipConvnet architectures.

Original: On CIFAR- 10 , we observe significant improvements in both the standard ( ≥ 1 .
Revised: On CIFAR- 10 , we observe significant improvements in both the standard ( ≥ 1 .

Original: 63% , column "Increase (standard)") and provable robust accuracy( ≥ 3 .
Revised: 63% , column "Increase (standard)") and provable robust accuracy ( ≥ 3 .

Original: 14% , column "Increase (36/255)") across all architectures.
Revised: 14% , column "Increase (36/255)") across all architectures.

Original: On CIFAR-100, we also observesignificant improvements in the standard ( ≥ 2 .49% ) and provable robust accuracy ( ≥ 2 . 27% ).
Revised: On CIFAR-100, we also observe significant improvements in the standard ( ≥ 2 . and provable robust accuracy ( ≥ 2 . 27% ).

Original: Fordeeper networks ( LipConvnet - 35 , 40 ), on CIFAR- 10 , we observe even more significant gains in thestandard ( ≥ 5 .
Revised: For deeper networks ( LipConvnet - 35 , 40 ), on CIFAR- 10 , we observe even more significant gains in the standard ( ≥ 5 .

Original: and provable robust accuracy ( ≥ 8 .
Revised: and provable robust accuracy ( ≥ 8 .

Original: Similarly on CIFAR- 100 , we observegains of ≥ 8 . 

Revised: Similarly on CIFAR- 100 , we observe gains of ≥ 8 .

Original: % (standard) and ≥ 8 .
Revised: % (standard) and ≥ 8 .

Original: 65% (provable robust).
Revised: 65% (provable robust).

Original: Our results establish a new state-of-the-art for both the standard and provable robust accuracy acrossall attack radii.
Revised: Our results establish a new state-of-the-art for both the standard and provable robust accuracy across all attack radii.

Original: In Table 3 (CIFAR-10), the best "Baseline" standard and provable robust accuracy values (at 36 / 255 , 72 / 255 , 108 / 255 ) are 77 .
Revised: In Table 3 (CIFAR-10), the best "Baseline" standard and provable robust accuracy values (at 36 / 255 , 72 / 255 , 108 / 255 ) are 77 .

Original: 78% and 63 .
Revised: 78% and 63 .

Original: 53% respectively.
Revised: 53% respectively.

Original: The best" + CRC " values are 79 .
Revised: The best " + CRC " values are 79 .

Original: 57% and 67 .
Revised: 57% and 67 .

Original: This results in improvements of +1 .
Revised: This results in improvements of +1 .

Original: 79%and +3 .
Revised: 79% and +3 .

Original: 82% , +6 .
Revised: 82% , +6 .

Original: 75% , +7 .
Revised: 75% , +7 .

Original: 07% respectively.
Revised: 07% respectively.

Original: Similarly, in Table 4 (CIFAR-100), the best "Baseline"standard and provable robust values are 48 .
Revised: Similarly, in Table 4 (CIFAR-100), the best "Baseline" standard and provable robust values are 48 .

Original: 06% and 34 .
Revised: 06% and 34 .

Original: The best " + CRC "values are 51 .
Revised: The best " + CRC " values are 51 .

Original: 84% (+3 .
Revised: 84% (+3 .

Original: 78%) and 39 .
Revised: 78%) and 39 .

Original: 27% (+4 .
Revised: 27% (+4 .

Original: 94% (+4 .
Revised: 94% (+4 .

Original: 29% (+5 .
Revised: 29% (+5 .

Original:  Checklist
Revised:  Acknowledgements

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope? [Yes]

Original: (b) Did you describe the limitations of your work?

Original: [Yes] in Section 7, we discuss how CRC -

Original: Lip might lead to slower certification.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] (b) Did you specify all the training details (e
Revised: This project was supported in part by NSF CAREER AWARD 1942230, a grant from NIST 60NANB20D134, HR001119S0026 (GARD), ONR YIP award N00014-22-1-2271, Army Grant No.

Original: g., data splits, hyperparameters, how they were

Original: chosen)?

Original: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments

Original: multiple times)?

Original: [N/A] because it is expensive to run.

Original: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., type of

Original: GPUs, internal cluster, or cloud provider)?

Original: [Yes]

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]

Original: (b) Did you mention the license of the assets?

Original: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?

Original: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applica-

Original: ble?

Original: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board

Original: (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Revised: W911NF2120076 and the NSF award CCF2212458.

Original: AppendixA ProofsA.1 Proof of Theorem
Revised: Appendix A Proofs A.1 Proof of Theorem

Original: Since A is skew symmetric, ∀ j, k, A j,k = − A k,j .
Revised: Since A is skew symmetric, ∀ j, k, A j,k = − A k,j .

Original: Thus, we focus on computing ∇
Revised: Thus, we focus on computing ∇

Original: A j,k ℓ for j > k .
Revised: A j,k ℓ for j > k .

Original: A.1.1 Preliminaries
Revised: A.1.1 Preliminaries

Original: Using equation (1) in the main text, we can rewrite z as follows:z
Revised: Using equation (1) in the main text, we can rewrite z as follows:

Original: [Equation]
Revised: [Equation]

Original: z ( i ) i ! , where z ( i ) =
Revised: ! , where z ( i ) =

Original: A i x
Revised: A i x

Original: Using the chain rule, ∇ A j,k ℓ can be computed as follows:
Revised: Using the chain rule, ∇ A j,k ℓ can be computed as follows:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Moreover, since z (0) is independent of W j,k , we have: ∇ A j,k z (0) = 

Revised: Moreover, since z (0) is independent of W j,k , we have: ∇ A j,k z (0) =

Original: implying:
Revised: implying:

Original: [Equation]
Revised: [Equation]

Original: To compute each ∇ A j,k z ( i ) , consider a matrix E defined below:
Revised: To compute each ∇ A j,k z ( i ) , consider a matrix E defined below:

Original: [Equation]
Revised: [Equation]

Original: Now, let e ( j ) denote a vector that satisfies:
Revised: Now, let e ( j ) denote a vector that satisfies:

Original: [Equation]
Revised: [Equation]

Original: Clearly, E can be rewritten as follows:
Revised: Clearly, E can be rewritten as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: A.1.2 Computing ∇ A j,k z ( i )
Revised: A.1.2 Computing ∇ A j,k z ( i )

Original: Since z ( i ) =
Revised: Since z ( i ) =

Original: A i x , we can compute ∇ A j,k z ( i ) as follows:[Equation]
Revised: A i x [Equation] we can compute ∇ A j,k z ( i ) as follows:

Original: [Equation]
Revised: [Equation]

Original: Using equations (10) and (14)
Revised: Using equations (10) and (14)

Original: [Equation]
Revised: [Equation]

Original: we can simplify ∇ A j
Revised: we can simplify ∇ A j

Original: [Equation]
Revised: [Equation]

Original: k ℓ as follows:
Revised: k ℓ as follows:

Original: [Equation]
Revised: [Equation]

Original: Rewriting m
Revised: Rewriting m

Original: = i − l − 1 , we get:
Revised: = i − l − 1 , we get:

Original: [Equation]
Revised: [Equation]

Original: Note that the above equation can also be rewritten in the following equivalent forms:
Revised: Note that the above equation can also be rewritten in the following equivalent forms:

Original: [Equation]
Revised: [Equation]

Original: Let us now define the following quantities:
Revised: Let us now define the following quantities:

Original: [Equation]
Revised: [Equation]

Original: Note that they are similar to equations (2) and (3) in the maintext.
Revised: Note that they are similar to equations (2) and (3) in the maintext.

Original: Using equations (17) and (18)
Revised: Using equations (17) and (18)

Original: [Equation]
Revised: [Equation]

Original: we can rewrite (15) and (16) as follows:
Revised: we can rewrite (15) and (16) as follows:

Original: [Equation]
Revised: [Equation]

Original: Using the definition of E (equation (11)), we obtain the following equivalent expressions of ∇ A j,k ℓ :
Revised: Using the definition of E (equation (11)) [Equation] we obtain the following equivalent expressions of ∇ A j,k ℓ :

Original: [Equation]
Revised: [Equation]

Original: Using the above equation, we obtain the following equivalent expressions for ∇ ℓ :
Revised: Using the above equation [Equation] we obtain the following equivalent expressions for ∇ ℓ :

Original: [Equation]
Revised: [Equation]

Original: A.2 Proof of Proposition
Revised: A.2 Proof of Proposition

Original: Consider the inequality given in equation (6):
Revised: Consider the inequality given in equation (6):

Original: [Equation]
Revised: [Equation]

Original: The above inequality says that for the prediction of h to be different from l for some input y ∗ , the l 2distance ∥ y ∗ − g ( x ) ∥ 2 must be at least R .
Revised: The above inequality says that for the prediction of h to be different from l for some input y ∗ , the ldistance ∥ y ∗ − g ( x ) ∥ 2 must be at least R .

Original: We first take the contrapositive of this statement:
Revised: We first take the contrapositive of this statement:

Original: [Equation]
Revised: [Equation]

Original: y

Original: [Equation][Equation]
Revised: [Equation]

Original: But since g is 1 -Lipschitz, we have:
Revised: But since g is 1 -Lipschitz, we have:

Original: [Equation]
Revised: [Equation]

Original: Using both (25) and (26), we get:
Revised: Using both (25) and (26), we get:

Original: [Equation]
Revised: [Equation]

Original: Thus, in an l 2 ball of radius R , prediction l remains unchanged and the function is provably robust.
Revised: Thus, in an l 2 ball of radius R , prediction l remains unchanged and the function is provably robust.

Original: A.3 Proof of Theorem
Revised: A.3 Proof of Theorem

Original: Let x ∗ , y ∗ ∈ M
Revised: Let x ∗ , y ∗ ∈ M

Original: so that d M ( x ) and d M ( y ) satisfy:
Revised: so that d M ( x ) and d M ( y ) satisfy:

Original: [Equation]
Revised: [Equation]

Original: Since x ∗ and y ∗ minimize the l 2 distance of x and y respectively to M , we have:
Revised: Since x ∗ and y ∗ minimize the l 2 distance of x and y respectively to M , we have:

Original: [Equation]
Revised: [Equation]

Original: Using the triangle inequality for the expressions ∥
Revised: Using the triangle inequality for the expressions ∥

Original: y ∗ − x ∥ 2 and ∥ x ∗ − y ∥ 2 , we have:
Revised: y ∗ − x ∥ 2 and ∥ x ∗ − y ∥ 2 , we have:

Original: [Equation]
Revised: [Equation]

Original: Using inequalities (28) and (29), we get:
Revised: Using inequalities (28) and (29), we get:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: But then using equation (27) and inequality (30), we get:
Revised: But then using equation (27) and inequality (30), we get:

Original: [Equation]
Revised: [Equation]

Original: A.4 Proof of Corollary1
Revised: A.4 Proof of Corollary

Original: We define a function d M ( x ) that is (without loss of generality) positive in A and negative in B :
Revised: We define a function d M ( x ) that is (without loss of generality) positive in A and negative in B :

Original: [Equation]
Revised: [Equation]

Original: Using Theorem 2, it is easy to see that within each set A and B , d M is 1 -Lipschitz and continuous.
Revised: Using Theorem 2, it is easy to see that within each set A and B , d M is 1 -Lipschitz and continuous.

Original: Similarly, because every line segment connecting A and B crosses through some point on the M , itis evident that d M is continuous on each path connecting A to B .
Revised: Similarly, because every line segment connecting A and B crosses through some point on the M , it is evident that d M is continuous on each path connecting A to B .

Original: So, we next prove that d M is 1 -Lipschitz if x ∈ A and y ∈ B i.e.:
Revised: So, we next prove that d M is 1 -Lipschitz if x ∈ A and y ∈ B i.e.:

Original: [Equation]
Revised: [Equation]

Original: Let
Revised: Let

Original: p ∈
Revised: p ∈

Original: M be a point on the line segment connecting x and y such that:
Revised: M be a point on the line segment connecting x and y such that:

Original: [Equation]
Revised: [Equation]

Original: But then by the definitions of d M ( x ) and d M ( y ) :
Revised: But then by the definitions of d M ( x ) and d M ( y ) :

Original: [Equation]
Revised: [Equation]

Original: Using equation (31) and inequality (32), we get:
Revised: Using equation (31) and inequality (32), we get:

Original: [Equation]
Revised: [Equation]

Original: Since d M ( x ) and d M ( y ) have opposite signs, we have:
Revised: Since d M ( x ) and d M ( y ) have opposite signs, we have:

Original: [Equation]
Revised: [Equation]

Original: Using inequality (33) and equation (34), we prove:
Revised: Using inequality (33) and equation (34), we prove:

Original: [Equation]
Revised: [Equation]

Original: Hence, | d M ( x ) − d M ( y ) | is 1 -Lipschitz.
Revised: Hence, | d M ( x ) − d M ( y ) | is 1 -Lipschitz.

Original: B Approximating the convolution weight gradient 
Revised: B Approximating the convolution weight gradient B.1 Simplifying

Original: B.  Simplifying

Original: the 
Revised: the expression for ∇ A j,k ℓ Recall the expression for ∇ A j,k ℓ using equations (10) and (14): ∇ A j,k ℓ =

Original: expression
Revised: [Equation]

Original: for
Revised: i

Original:  ∇A j,k ℓ
Revised: [Equation] ∇

Original: Recall the expression for ∇ A j,k ℓ using equations (10) and (14):∇ A j,k ℓ = (cid:32) ∞ (cid:88) i = ∇ A j,k z ( i ) i !
Revised: [Equation] [Equation] i [Equation]

Original:  (cid:33) T ∇ z ℓ, where ∇ A j,k z ( i ) =
Revised: i ! (cid:33) [Equation] ∇ z ℓ, where ∇ A j,k z ( i ) =

Original: i − 1 (cid:88) l
Revised: i − 1 (cid:88) l

Original: = (cid:0)
Revised: = (cid:0)

Original: A l (cid:1) E (cid:0) A i − l − 1 (cid:1) x
Revised: A l (cid:1) E (cid:0) A i − l − 1 (cid:1) x

Original: We replace ∇ A j,k z ( i ) in ∇ A j,k ℓ and take x out of the transpose:∇ A
Revised: We replace ∇ A j,k z ( i ) in ∇ A j,k ℓ and take x out of the transpose:

Original: [Equation]
Revised: [Equation]

Original: k
Revised: We simplify the above as follows:

Original: [Equation]
Revised: [Equation]

Original: =

Original: [Equation]
Revised: [Equation]

Original: T (cid:32) ∞ (cid:88)

Original: [Equation]

Original: [Equation] (cid:19) T ∇ z ℓ We simplify the above as follows:∇ A j,k ℓ (35)
Revised: where R is defined as follows:

Original: [Equation][Equation]
Revised: [Equation]

Original: (37) where R is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: In the above simplification, R contains all terms of the form A l EA m where both l ≥ 1 and m ≥ 1 .
Revised: In the above simplification, R contains all terms of the form A l EA m where both l ≥ 1 and m ≥ 1 .

Original: B.2 Simplifying the expression for (cid:0) u (1) (cid:1) T E T v (1)
Revised: B.2 Simplifying the expression for (cid:0) u (1) (cid:1) T E T v (1)

Original: Recall that u (1) , v (1) are given by (equations (17) and (18)):[Equation]
Revised: Recall that u (1) , v (1) are given by (equations (17) and (18)):

Original: [Equation]
Revised: [Equation]

Original: We first replace v (1) using the above in (cid:0) u (1) (cid:1) T E T v (1) :[Equation] u
Revised: We first replace v (1) using the above in (cid:0) u (1) (cid:1) T E T v (1) :

Original: [Equation]
Revised: [Equation]

Original: Given 2 matrices B , C , we have (cid:0) B T (cid:1) i
Revised: Given 2 matrices B , C , we have (cid:0) B T (cid:1) i

Original: = (cid:0) B i (cid:1) T and ( B + C ) T =
Revised: = (cid:0) B i (cid:1) T and ( B + C ) T =

Original: B T + C T .
Revised: B T + C T .

Original: Thus: [Equation] ∞ (cid:88)
Revised: Thus: (cid:32)

Original: [Equation]
Revised: [Equation]

Original: Replacing the above in equation (40), we get:
Revised: Replacing the above in equation (40), we get:

Original: [Equation]
Revised: [Equation]

Original: Given 3 matrices, B , C , D , we have D T C T B T =
Revised: Given 3 matrices, B , C , D , we have D T C T B T =

Original: ( BCD ) T .
Revised: ( BCD ) T .

Original: Thus:
Revised: Thus:

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: we replace u (1) using equation (39) and take x out of the transpose:
Revised: we replace u (1) using equation (39) and take x out of the transpose:

Original: [Equation]
Revised: [Equation]

Original: Expanding the two series
Revised: Expanding the two series

Original: [Equation]
Revised: [Equation]

Original: we get:
Revised: we get:

Original: [Equation]
Revised: [Equation]

Original: We simplify the above as follows:
Revised: We simplify the above as follows:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where S is defined as follows:
Revised: where S is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: + A 2 EA3!3!

Original: + A 3 EA 4!2!

Original: (cid:19)
Revised: (cid:19)

Original: + . . .
Revised: + . . .

Original: B 3 Comparing the two expressions
Revised: B.3 Comparing the two expressions

Original: Comparing equations (35) and (41), the first part in the two are same.
Revised: Comparing equations (35) and (41), the first part in the two are same.

Original: The first part are the equations (36) and (42) respectively.
Revised: The first part are the equations (36) and (42) respectively.

Original: The two expressions only differ in the terms x T R T ( ∇ z ℓ ) and x T S T ( ∇ z ℓ ) .
Revised: The two expressions only differ in the terms x T R T ( ∇ z ℓ ) and x T S T ( ∇ z ℓ ) .

Original: Note that in R (equation (38)), the term A l EA m is multiplied by 1 / ( l + m + 1)! .
Revised: Note that in R (equation (38)), the term A l EA m is multiplied by 1 / ( l + m + 1)! .

Original: And in S (equation (44)), the term A l EA m is multiplied by 1 / ( l + 1)!
Revised: And in S (equation (44)), the term A l EA m is multiplied by 1 / ( l + 1)!

Original: ( m + 1)! .
Revised: ( m + 1)! .

Original: Thus, since the quantities R and S are multiplied by factorial in the denominator.
Revised: Thus, since the quantities R and S are multiplied by factorial in the denominator.

Original: For higher values of l or m , both the expressions converge to 0 .
Revised: For higher values of l or m , both the expressions converge to 0 .

Original: Moreover, S − R for smaller values of l, m can be directly computed using equations (38) and (44):
Revised: Moreover, S − R for smaller values of l, m can be directly computed using equations (38) and (44):

Original: [Equation]
Revised: [Equation]

Original: We can see that the denominator in the above terms are already very small even for l + m = 4 .
Revised: We can see that the denominator in the above terms are already very small even for l + m = 4 .

Original: Thus, we simply assume S − R ≈ 0 .
Revised: Thus, we simply assume S − R ≈ 0 .

Original: This results in the following approximate expression for ∇ A j,k ℓ :
Revised: This results in the following approximate expression for ∇ A j,k ℓ :

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Using the definition of E (equation (11)), we obtain the following expression of ∇ A j,k ℓ :
Revised: Using the definition of E (equation (11)), we obtain the following expression of ∇ A j,k ℓ :

Original: [Equation]
Revised: [Equation]

Original: Using equation (47), the gradient with respect to matrix A can be computed as follows:
Revised: Using equation (47), the gradient with respect to matrix A can be computed as follows:

Original: [Equation]
Revised: [Equation]

Original: C Iterations for z and ∇ z ℓ
Revised: C Iterations for z and ∇ z ℓ

Original: Recall that we use the following iterations to compute z during the forward pass:
Revised: Recall that we use the following iterations to compute z during the forward pass:

Original: [Equation]
Revised: [Equation]

Original: We have to prove that ∀ k, u (0) =
Revised: We have to prove that ∀ k, u (0) =

Original: z where:z
Revised: z where:

Original: [Equation]
Revised: [Equation]

Original: (cid:88) i =

Original: [Equation]

Original: i ! To prove the same, we use induction over k .
Revised: To prove the same, we use induction over k .

Original: For k = 1 , u (0) =
Revised: For k = 1 , u (0) =

Original: z = x by definition.
Revised: z = x by definition.

Original: For k ≥ 2 , we assume that the statement is true for j ≤ k − 1 and try to prove for k .
Revised: For k ≥ 2 , we assume that the statement is true for j ≤ k − 1 and try to prove for k .

Original: For k , we have the following expressions for u ( k − 1) and u ( k − 2) :
Revised: For k , we have the following expressions for u ( k − 1) and u ( k − 2) :

Original: [Equation]
Revised: [Equation]

Original: Next, we apply the iteration on u ( k − 2) till u (0) .
Revised: Next, we apply the iteration on u ( k − 2) till u (0) .

Original: By induction, the first term i.e. x simply results in (cid:80) k − 2 i =

Revised: By induction, the first term i.e. x simply results in (cid:80) k − 2 i =

Original: ( A i x ) /i ! .
Revised: ( A i x ) /i ! .

Original: The second term i.e. Ax /k in ( A k − 1 x ) / ( k − 1)! .
Revised: The second term i.e. Ax /k in ( A k − 1 x ) / ( k − 1)! .

Original: Thus, we have:
Revised: Thus, we have:

Original: [Equation]
Revised: [Equation]

Original: The same can proof follows for ∇ z ℓ except that we replace A with − A .
Revised: The same can proof follows for ∇ z ℓ except that we replace A with − A .

Original: D Review of Curvature-based Robustness Certificate
Revised: D Review of Curvature-based Robustness Certificate

Original: Given a binary classifier f : R d → R and input x (0) , we are interested in finding its shortest l 2distance to the decision boundary: min x ∥ x − x (0) ∥ 2 where f ( x ) = 0 .
Revised: Given a binary classifier f : R d → R and input x (0) , we are interested in finding its shortest ldistance to the decision boundary: min x ∥ x − x (0) ∥ 2 where f ( x ) = 0 .

Original: Using lagrange multiplier ηfor the constraint f ( x ) = 0 , the primal and dual problems can be written as follows:
Revised: Using lagrange multiplier η for the constraint f ( x ) = 0 , the primal and dual problems can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: Singla and Feizi [2020] prove that: If m I ≼ ∇ 2 x f ≼ M I , the dual minimization can be solved using 
Revised: Singla and Feizi [2020] prove that: If m I ≼ ∇ 2 x f ≼ M I , the dual minimization can be solved using convex optimization for − 1 /M ≤ η ≤ − 1 /m .

Revised: Furthermore, they show that if the solution x of the solution satisfies

Original: E Complete results on CIFAR-10 and CIFAR- 
Revised: E Complete results on CIFAR-10 and CIFAR-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: F
Revised: F

Original: Running time comparison for Faster SOC gradient computation

Original: [Table] [Table]
Revised: [Table] [Table]

Original: G Example of projection pooling layer
Revised: G Example of projection pooling layer