Original: LOT: Layer-wise Orthogonal Training on Improving ℓ 2 Certified Robustness
Revised: LOT: Layer-wise Orthogonal Training on Improving ℓ 2 Certified Robustness

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Recent studies show that training deep neural networks (DNNs) with Lipschitzconstraints are able to enhance adversarial robustness and other model propertiessuch as stability.
Revised: Recent studies show that training deep neural networks (DNNs) with Lipschitzconstraints are able to enhance adversarial robustness and other model propertiessuch as stability.

Original: In this paper, we propose a layer-wise orthogonal training method(LOT) to effectively train 1-Lipschitz convolution layers via parametrizing anorthogonal matrix with an unconstrained matrix.
Revised: In this paper, we propose a layer-wise orthogonal training method(LOT) to effectively train 1-Lipschitz convolution layers via parametrizing anorthogonal matrix with an unconstrained matrix.

Original: We then efficiently compute theinverse square root of a convolution kernel by transforming the input domain to the
Revised: We then efficiently compute theinverse square root of a convolution kernel by transforming the input domain to the

Original: Fourier frequency domain.
Revised: Fourier frequency domain.

Original: On the other hand, as existing works show that semisupervised training helps improve empirical robustness, we aim to bridge the gapand prove that semi-supervised learning also improves the certified robustness of
Revised: On the other hand, as existing works show that semisupervised training helps improve empirical robustness, we aim to bridge the gapand prove that semi-supervised learning also improves the certified robustness of

Original: Lipschitz-bounded models.
Revised: Lipschitz-bounded models.

Original: We conduct comprehensive evaluations for LOT underdifferent settings.
Revised: We conduct comprehensive evaluations for LOT underdifferent settings.

Original: We show that LOT significantly outperforms baselines regardingdeterministic ℓ 2 certified robustness, and scales to deeper neural networks.
Revised: We show that LOT significantly outperforms baselines regardingdeterministic ℓ 2 certified robustness, and scales to deeper neural networks.

Original: Underthe supervised scenario, we improve the state-of-the-art certified robustness forall architectures (e.g. from 59.04% to 63.50% on CIFAR-10 and from 32.57% to34.59% on CIFAR-100 at radius ρ = 36 / 255 for 40-layer networks).
Revised: Underthe supervised scenario, we improve the state-of-the-art certified robustness forall architectures (e.g. from 59.04% to 63.50% on CIFAR-10 and from 32.57% to34.59% on CIFAR-100 at radius ρ = 36 / 255 for 40-layer networks).

Original: With semisupervised learning over unlabelled data, we are able to improve state-of-the-artcertified robustness on CIFAR-10 at ρ = 108 / 255 from 36.04% to 42.39%.
Revised: With semisupervised learning over unlabelled data, we are able to improve state-of-the-artcertified robustness on CIFAR-10 at ρ = 108 / 255 from 36.04% to 42.39%.

Original: Inaddition, LOT consistently outperforms baselines on different model architectureswith only 1/3 evaluation time.
Revised: Inaddition, LOT consistently outperforms baselines on different model architectureswith only 1/3 evaluation time.

Original:  Introduction
Revised:  Introduction

Original: Given the wide applications of deep neural networks (DNNs), ensuring their robustness againstpotential adversarial attacks [7] is of great importance.
Revised: Given the wide applications of deep neural networks (DNNs), ensuring their robustness againstpotential adversarial attacks [7] is of great importance.

Original: There has been a line of research providingdefense approaches to improve the empirical robustness of DNNs [16] and certification methods tocertify DNN robustness [13].
Revised: There has been a line of research providingdefense approaches to improve the empirical robustness of DNNs [16] and certification methods tocertify DNN robustness [13].

Original: Existing certification techniques can be categorized as deterministicand probabilistic certifications [13], and in this work we will focus on improving the deterministic ℓ 2certified robustness by training 1-Lipschitz DNNs.
Revised: Existing certification techniques can be categorized as deterministicand probabilistic certifications [13], and in this work we will focus on improving the deterministic ℓ 2certified robustness by training 1-Lipschitz DNNs.

Original: Although different approaches have been proposed to empirically enforce the Lipschitz constant ofthe trained model [23], it is still challenging to strictly ensure the 1-Lipschitz, which can lead to tightrobustness certification.
Revised: Although different approaches have been proposed to empirically enforce the Lipschitz constant ofthe trained model [23], it is still challenging to strictly ensure the 1-Lipschitz, which can lead to tightrobustness certification.

Original: One recent work SOC [20] proposes to parametrize the orthogonal weightmatrices with the exponential of skew-symmetric matrices (i.e. W = exp( V − V ⊺ ) ).
Revised: One recent work SOC [20] proposes to parametrize the orthogonal weightmatrices with the exponential of skew-symmetric matrices (i.e. W = exp( V − V ⊺ ) ).

Original: However,such parametrization will be biased when the matrix norm is constrained and the expressivenessis limited, especially when they rescale V to be small to help with convergence.
Revised: However,such parametrization will be biased when the matrix norm is constrained and the expressivenessis limited, especially when they rescale V to be small to help with convergence.

Original: In this work, wepropose a layer-wise orthogonal training approach (LOT) by parameterizing the orthogonal weightmatrix with an unconstrained matrix W =
Revised: In this work, wepropose a layer-wise orthogonal training approach (LOT) by parameterizing the orthogonal weightmatrix with an unconstrained matrix W =

Original: ( V V ⊺ ) − 12 V .
Revised: ( V V ⊺ ) − 12 V .

Original: In order to calculate the inverse square rootfor convolution kernels, we will perform Fourier Transformation and calculate the inverse square rootof matrices on the frequency domain using Newton’s iteration.
Revised: In order to calculate the inverse square rootfor convolution kernels, we will perform Fourier Transformation and calculate the inverse square rootof matrices on the frequency domain using Newton’s iteration.

Original: In our parametrization, the output is agnostic to the input norm (i.e. scaling V = αV does not change the value of W ).
Revised: In our parametrization, the output is agnostic to the input norm (i.e. scaling V = αV does not change the value of W ).

Original: We show that suchparametrization achieves higher model expressiveness and robustness (Section 6.2), and providesmore meaningful representation vectors (Section 6.3).
Revised: We show that suchparametrization achieves higher model expressiveness and robustness (Section 6.2), and providesmore meaningful representation vectors (Section 6.3).

Original: In addition, several works have shown that semi-supervised learning will help improve the empiricalrobustness of models under adversarial training and other settings [4].
Revised: In addition, several works have shown that semi-supervised learning will help improve the empiricalrobustness of models under adversarial training and other settings [4].

Original: In this work, we take thefirst attempt to bridge semi-supervised training with certified robustness based on our 1-Lipschitz
Revised: In this work, we take thefirst attempt to bridge semi-supervised training with certified robustness based on our 1-Lipschitz

Original: DNNs.
Revised: DNNs.

Original: Theoretically, we show that semi-supervised learning can help improve the error bound of
Revised: Theoretically, we show that semi-supervised learning can help improve the error bound of

Original: Lipschitz-bounded models.
Revised: Lipschitz-bounded models.

Original: We also lower bound the certified radius as a function of the modelperformance and Lipschitz property.
Revised: We also lower bound the certified radius as a function of the modelperformance and Lipschitz property.

Original: Empirically, we indeed observe that including un-labelled datawill help with the certified robustness of 1-Lipschitz models, especially at a larger radius (e.g. from36.04% to 42.39% at ρ = 108 / 255 on CIFAR-10).
Revised: Empirically, we indeed observe that including un-labelled datawill help with the certified robustness of 1-Lipschitz models, especially at a larger radius (e.g. from36.04% to 42.39% at ρ = 108 / 255 on CIFAR-10).

Original: We conduct comprehensive experiments to evaluate our approach, and we show that LOT significantlyoutperforms the state-of-the-art in terms of the deterministic ℓ 2 certified robustness.
Revised: We conduct comprehensive experiments to evaluate our approach, and we show that LOT significantlyoutperforms the state-of-the-art in terms of the deterministic ℓ 2 certified robustness.

Original: We also conductdifferent ablation studies to show that (1) LOT can produce more meaningful features for visualization;(2) residual connections help to smoothify the training of LOT model.
Revised: We also conductdifferent ablation studies to show that (1) LOT can produce more meaningful features for visualization;(2) residual connections help to smoothify the training of LOT model.

Original: Technical contributions .
Revised: Technical contributions .

Original: In this work, we aim to train a certifiably robust 1-Lipschitz model andalso analyze the certified radius of the Lipschitz bounded model under semi-supervised learning.
Revised: In this work, we aim to train a certifiably robust 1-Lipschitz model andalso analyze the certified radius of the Lipschitz bounded model under semi-supervised learning.

Original: We propose
Revised: We propose

Original: a
Revised: a

Original: layer-wise orthogonal training
Revised: layer-wise orthogonal training

Original: method
Revised: method

Original: LOT
Revised: LOT

Original: for
Revised: for

Original: convolution layers
Revised: convolution layers

Original: to train
Revised: to train

Original: 1-Lipschitzmodels based on Newton’s iteration, and thus compute the deterministic certified robustness for themodel.
Revised: 1-Lipschitzmodels based on Newton’s iteration, and thus compute the deterministic certified robustness for themodel.

Original: We prove
Revised: We prove

Original: the convergence of
Revised: the convergence of

Original: Newton’s
Revised: Newton’s

Original: iteration used
Revised: iteration used

Original: in our algorithm.
Revised: in our algorithm.

Original: • We derive the certified robustness of lipschitz constrained model under semi-supervised setting,
Revised: • We derive the certified robustness of lipschitz constrained model under semi-supervised setting,

Original: and formally show how semi-supervised learning affects the certified radius.
Revised: and formally show how semi-supervised learning affects the certified radius.

Original: • We evaluate our LOT method under different settings (i
Revised: • We evaluate our LOT method under different settings (i

Original: e. supervised and semi-supervised) on
Revised: e. supervised and semi-supervised) on

Original: different models and
Revised: different models and

Original: datasets.
Revised: datasets.

Original: With supervised learning, we show that it significantly outperformsstate-of-the-art baselines, and on some deep architectures the performance gain is over 4%.
Revised: With supervised learning, we show that it significantly outperformsstate-of-the-art baselines, and on some deep architectures the performance gain is over 4%.

Original: Withsemi-supervised learning, we further improve certified robust accuracy by over 6% at a large radius.
Revised: Withsemi-supervised learning, we further improve certified robust accuracy by over 6% at a large radius.

Original:  Related Work
Revised:  Related Work

Original: Certified Robustness for Lipschitz Constrained Models Several studies have been conducted toexplore the Lipschitz-constrained models for certified robustness.
Revised: Certified Robustness for Lipschitz Constrained Models Several studies have been conducted toexplore the Lipschitz-constrained models for certified robustness.

Original: [23] first certify model robustnessbased on its Lipschitz constant and propose training algorithms to regularize the model Lipschitz.
Revised: [23] first certify model robustnessbased on its Lipschitz constant and propose training algorithms to regularize the model Lipschitz.

Original: Multiple works [5, 17, 18, 8] have been proposed to achieve 1-Lipschitz during training for linearnetworks by regularizing or normalizing the spectral norm of the weight matrix.
Revised: Multiple works [5, 17, 18, 8] have been proposed to achieve 1-Lipschitz during training for linearnetworks by regularizing or normalizing the spectral norm of the weight matrix.

Original: However, whenapplying these approaches on convolution layers, the spectral norm is bounded by unrolling theconvolution into linear operations, which leads to a loose Lipschitz bound [24].
Revised: However, whenapplying these approaches on convolution layers, the spectral norm is bounded by unrolling theconvolution into linear operations, which leads to a loose Lipschitz bound [24].

Original: Recently, [2] showthat the 1-Lipschitz requirement is not enough for a good robust model; rather, the gradient-normpreserving property is important.
Revised: Recently, [2] showthat the 1-Lipschitz requirement is not enough for a good robust model; rather, the gradient-normpreserving property is important.

Original: Besides these training-time techniques, different approaches havebeen proposed to calculate a tight Lipschitz bound during evaluation.
Revised: Besides these training-time techniques, different approaches havebeen proposed to calculate a tight Lipschitz bound during evaluation.

Original: [6] upper bounds the Lipschitzwith semi-definite programming while [11] upper bounds the Lipschitz with polynomial optimization.
Revised: [6] upper bounds the Lipschitzwith semi-definite programming while [11] upper bounds the Lipschitz with polynomial optimization.

Original: In this work we aim to effectively train 1-Lipschitz convolution models.
Revised: In this work we aim to effectively train 1-Lipschitz convolution models.

Original: Orthogonal Convolution Neural Networks [14] first proposes to directly construct orthogonalconvolution operations.
Revised: Orthogonal Convolution Neural Networks [14] first proposes to directly construct orthogonalconvolution operations.

Original: Such operations are not only 1-Lipschitz, but also gradient norm preserving,which provides a higher model capability and a smoother training process [2].
Revised: Such operations are not only 1-Lipschitz, but also gradient norm preserving,which provides a higher model capability and a smoother training process [2].

Original: BCOP [14] trainsorthogonal convolution by iteratively generating 2 × 2 orthogonal kernels from orthogonal matrices.[22] proposes to parametrize an orthogonal convolution with Cayley transformation W =
Revised: BCOP [14] trainsorthogonal convolution by iteratively generating 2 × 2 orthogonal kernels from orthogonal matrices.[22] proposes to parametrize an orthogonal convolution with Cayley transformation W =

Original: ( I − V +
Revised: ( I − V +

Original: V ⊺ )
Revised: V ⊺ )

Original: ( I + V − V ⊺ ) − 1 where the convolution inverse is calculated on the Fourier frequency domain.
Revised: ( I + V − V ⊺ ) − 1 where the convolution inverse is calculated on the Fourier frequency domain.

Original: ECO [26] proposes to explicitly normalize all the singular values [19] of convolution operations to be1.
Revised: ECO [26] proposes to explicitly normalize all the singular values [19] of convolution operations to be1.

Original: So far, the best-performing orthogonal convolution approach is SOC [20], where they parametrizethe orthogonal convolution with W = exp
Revised: So far, the best-performing orthogonal convolution approach is SOC [20], where they parametrizethe orthogonal convolution with W = exp

Original: ( V − V ⊺ ) where the exponential and transpose are definedwith the convolution operation.
Revised: ( V − V ⊺ ) where the exponential and transpose are definedwith the convolution operation.

Original: However, one major weakness of SOC is that it will rescale V to besmall so that the approximation of exp can converge soon, which will impose a bias on the resultingoutput space.
Revised: However, one major weakness of SOC is that it will rescale V to besmall so that the approximation of exp can converge soon, which will impose a bias on the resultingoutput space.

Original: For example, when V is very small W will be close to I .
Revised: For example, when V is very small W will be close to I .

Original: Such norm-dependentproperty is not desired, and thus we propose a parametrization that is invariant to rescaling.
Revised: Such norm-dependentproperty is not desired, and thus we propose a parametrization that is invariant to rescaling.

Original: Finally,[21] proposes several techniques for orthogonal CNNs, including a generalized Householder (HH)activation function, a certificate regularizer (CReg) loss, and a last layer normalization (LLN).
Revised: Finally,[21] proposes several techniques for orthogonal CNNs, including a generalized Householder (HH)activation function, a certificate regularizer (CReg) loss, and a last layer normalization (LLN).

Original: Theycan be integrated with our training approach to improve model robustness.
Revised: Theycan be integrated with our training approach to improve model robustness.

Original:  Problem Setup3.1 Lipschitz Constant of Neural Networks and Certified Robustness
Revised:  Problem Setup3.1 Lipschitz Constant of Neural Networks and Certified Robustness

Original: Let f : R d → R C denote a neural network for classification, where d is the input dimension and
Revised: Let f : R d → R C denote a neural network for classification, where d is the input dimension and

Original: C is the number of output classes.
Revised: C is the number of output classes.

Original: The model prediction is given by arg max c f ( x ) c , where f ( x ) crepresents the prediction probability for class c .
Revised: The model prediction is given by arg max c f ( x ) c , where f ( x ) crepresents the prediction probability for class c .

Original: The Lipschitz constant of the model under p -normis defined as: Lip p ( f ) = sup || f ( x 1 ) − f ( x 2 ) ||
Revised: The Lipschitz constant of the model under p -normis defined as: Lip p ( f ) = sup || f ( x 1 ) − f ( x 2 ) ||

Original: p || x
Revised: p || x

Original: 1 − x 2 || p ∀ x 1 , x 2 ∈ R d .
Revised: 1 − x 2 || p ∀ x 1 , x 2 ∈ R d .

Original: Unless specified, we will focus onℓ 2 -norm and use Lip ( f ) to denote Lip 2 ( f ) in this work.
Revised: Unless specified, we will focus onℓ 2 -norm and use Lip ( f ) to denote Lip 2 ( f ) in this work.

Original: We can observe that the definition of model
Revised: We can observe that the definition of model

Original: Lipschitz aligns with its robustness property - both require the model not to change much w.r.t.
Revised: Lipschitz aligns with its robustness property - both require the model not to change much w.r.t.

Original: inputchanges.
Revised: inputchanges.

Original: Formally speaking, define M f ( x ) = max i f ( x ) i − max j ̸ =argmax i f ( x ) i f ( x ) j to be theprediction gap of f on the input x , then we can guarantee that f ( x ) will not change its predictionwithin | x
Revised: Formally speaking, define M f ( x ) = max i f ( x ) i − max j ̸ =argmax i f ( x ) i f ( x ) j to be theprediction gap of f on the input x , then we can guarantee that f ( x ) will not change its predictionwithin | x

Original: ′ − x | < r , where
Revised: ′ − x | < r , where

Original: [Equation]
Revised: [Equation]

Original: Therefore, people have proposed to utilize the model Lipschitz to provide certified robustness andinvestigated training algorithms to train a model with a small Lipschitz constant.
Revised: Therefore, people have proposed to utilize the model Lipschitz to provide certified robustness andinvestigated training algorithms to train a model with a small Lipschitz constant.

Original: Note that the Lipschitz constant of a composed function f = f 1 ◦ f 2 satisfies Lip ( f ) ≤ Lip ( f 1 ) ×
Revised: Note that the Lipschitz constant of a composed function f = f 1 ◦ f 2 satisfies Lip ( f ) ≤ Lip ( f 1 ) ×

Original: Lip ( f 2 ) .
Revised: Lip ( f 2 ) .

Original: Since a neural network is usually composed of several layers, we can investigate the
Revised: Since a neural network is usually composed of several layers, we can investigate the

Original: Lipschitz of each layer to calculate the final upper bound.
Revised: Lipschitz of each layer to calculate the final upper bound.

Original: If we can restrict each layer to be1-Lipschitz, then the overall model will be 1-Lipschitz with an arbitrary number of layers.
Revised: If we can restrict each layer to be1-Lipschitz, then the overall model will be 1-Lipschitz with an arbitrary number of layers.

Original: 3.2 Orthogonal Linear and Convolution Operations
Revised: 3.2 Orthogonal Linear and Convolution Operations

Original: Consider a linear operation with equal input and output dimensions y
Revised: Consider a linear operation with equal input and output dimensions y

Original: =
Revised: =

Original: Wx , where x, y ∈ R nand W ∈ R n × n .
Revised: Wx , where x, y ∈ R nand W ∈ R n × n .

Original: We say W is an orthogonal matrix if WW ⊺ =
Revised: We say W is an orthogonal matrix if WW ⊺ =

Original: W ⊺ W
Revised: W ⊺ W

Original: =
Revised: =

Original: I and call y =
Revised: I and call y =

Original: Wxan orthogonal linear operation.
Revised: Wxan orthogonal linear operation.

Original: The orthogonal operation is not only 1-Lipschitz, but also normpreserving, i.e., ||
Revised: The orthogonal operation is not only 1-Lipschitz, but also normpreserving, i.e., ||

Original: Wx || 

Revised: Wx ||

Original: = ||
Revised: = ||

Original: x || 2 for all x ∈ R n .
Revised: x || 2 for all x ∈ R n .

Original: If the input and output dimensions do not match,i.e., W ∈ R m × n where n ̸ = m , we say W is semi-orthogonal if either WW ⊺ =
Revised: If the input and output dimensions do not match,i.e., W ∈ R m × n where n ̸ = m , we say W is semi-orthogonal if either WW ⊺ =

Original: I or W ⊺ W = I .
Revised: I or W ⊺ W = I .

Original: The semi-orthogonal operation is 1-Lipschitz and non-expansive
Revised: The semi-orthogonal operation is 1-Lipschitz and non-expansive

Original: , i.e., || Wx || 2 ≤ || x
Revised: , i.e., || Wx || 2 ≤ || x

Original: || 2 .
Revised: || 2 .

Original: The orthogonal convolution operation is defined in a similar way.
Revised: The orthogonal convolution operation is defined in a similar way.

Original: Let y
Revised: Let y

Original: =
Revised: =

Original: W ◦
Revised: W ◦

Original: X be an orthogonaloperation, where x, y ∈ R c × w × w and W ∈ R c × c × k × k .
Revised: X be an orthogonaloperation, where x, y ∈ R c × w × w and W ∈ R c × c × k × k .

Original: We say W is an orthogonal convolutionkernel if W ◦ W ⊺ =
Revised: We say W is an orthogonal convolutionkernel if W ◦ W ⊺ =

Original: W ⊺ ◦ W =
Revised: W ⊺ ◦ W =

Original: I conv where the transpose here refers to the convolution transposeand I conv denotes the identity convolution kernel.
Revised: I conv where the transpose here refers to the convolution transposeand I conv denotes the identity convolution kernel.

Original: Such orthogonal convolution is 1-Lipschitz andnorm-preserving.
Revised: Such orthogonal convolution is 1-Lipschitz andnorm-preserving.

Original: When the input and output channel numbers are different, the semi-orthogonalconvolution kernel W satisfies either W ◦ W ⊺ =
Revised: When the input and output channel numbers are different, the semi-orthogonalconvolution kernel W satisfies either W ◦ W ⊺ =

Original: I conv or W ⊺ ◦ W = I conv
Revised: I conv or W ⊺ ◦ W = I conv

Original: and it is 1-Lipschitz andnon-expansive.
Revised: and it is 1-Lipschitz andnon-expansive.

Original:  LOT: Layer-wise Orthogonal Training
Revised:  LOT: Layer-wise Orthogonal Training

Original: In this section, we propose our LOT framework to achieve certified robustness.
Revised: In this section, we propose our LOT framework to achieve certified robustness.

Original: We will first introducehow LOT layer works to achieve 1-Lipschitz.
Revised: We will first introducehow LOT layer works to achieve 1-Lipschitz.

Original: The key idea of our method is to parametrize anorthogonal convolution W with an un-constrained convolution kernel W =
Revised: The key idea of our method is to parametrize anorthogonal convolution W with an un-constrained convolution kernel W =

Original: ( V V ⊺ ) − 12 V .
Revised: ( V V ⊺ ) − 12 V .

Original: Next, wewill propose several techniques to improve the training and evaluation processes of our model.
Revised: Next, wewill propose several techniques to improve the training and evaluation processes of our model.

Original: Finally,we discuss how semi-supervised learning can help with the certified robustness of our model.
Revised: Finally,we discuss how semi-supervised learning can help with the certified robustness of our model.

Original: 4.1 1-Lipschitz Neural Networks via LOT
Revised: 4.1 1-Lipschitz Neural Networks via LOT

Original: Our key observation is that we can parametrize an orthogonal matrix W ∈ R n × n with an unconstrained matrix V ∈ R n × n by W =
Revised: Our key observation is that we can parametrize an orthogonal matrix W ∈ R n × n with an unconstrained matrix V ∈ R n × n by W =

Original: ( V V ⊺ ) − 12 V [9].
Revised: ( V V ⊺ ) − 12 V [9].

Original: In addition, this equation also holds truein the case of convolution kernel - given any convolution kernel V ∈ R c × c × k × k , where c denotesthe channel number and k denotes the kernel size, we can get an orthogonal convolution kernel by
Revised: In addition, this equation also holds truein the case of convolution kernel - given any convolution kernel V ∈ R c × c × k × k , where c denotesthe channel number and k denotes the kernel size, we can get an orthogonal convolution kernel by

Original: W =
Revised: W =

Original: ( V ◦ V ⊺ ) − 12 ◦ V , where transpose and inverse square root are with respect to the convolutionoperations.
Revised: ( V ◦ V ⊺ ) − 12 ◦ V , where transpose and inverse square root are with respect to the convolutionoperations.

Original: The orthogonality of W can be verified by W ◦ W ⊺ =
Revised: The orthogonality of W can be verified by W ◦ W ⊺ =

Original: W ◦ W ⊺ =
Revised: W ◦ W ⊺ =

Original: I conv , where I isthe identity convolution kernel.
Revised: I conv , where I isthe identity convolution kernel.

Original: This way, we can parametrize an orthogonal convolution layer bytraining over the un-constrained parameter V .
Revised: This way, we can parametrize an orthogonal convolution layer bytraining over the un-constrained parameter V .

Original: Formally speaking, we will parametrize an orthogonal convolution layer by Y =
Revised: Formally speaking, we will parametrize an orthogonal convolution layer by Y =

Original: ( V ◦ V ⊺ ) − 12 ◦ V ◦
Revised: ( V ◦ V ⊺ ) − 12 ◦ V ◦

Original: X ,where X, Y ∈ R c × w × w and w ≥ k is the input shape.
Revised: X ,where X, Y ∈ R c × w × w and w ≥ k is the input shape.

Original: The key obstacle here is how to calculate theinverse square root of a convolution kernel.
Revised: The key obstacle here is how to calculate theinverse square root of a convolution kernel.

Original: Inspired by [22], we can leverage the convolution theoremwhich states that the convolution in the input domain equals the element-wise multiplication in the
Revised: Inspired by [22], we can leverage the convolution theoremwhich states that the convolution in the input domain equals the element-wise multiplication in the

Original: Fourier frequency domain.
Revised: Fourier frequency domain.

Original: In the case of multi-channel convolution, the convolution correspondsto the matrix multiplication on each pixel location.
Revised: In the case of multi-channel convolution, the convolution correspondsto the matrix multiplication on each pixel location.

Original: That is, let FFT : R w × w → C w × w be the 2D
Revised: That is, let FFT : R w × w → C w × w be the 2D

Original: Discrete Fourier Transformation and FFT − 1 be the inverse Fourier Transformation.
Revised: Discrete Fourier Transformation and FFT − 1 be the inverse Fourier Transformation.

Original: We will zero-padthe input to w × w if the original shape is smaller than w .
Revised: We will zero-padthe input to w × w if the original shape is smaller than w .

Original: Let ˜ X
Revised: Let ˜ X

Original: i = FFT ( X i ) and ˜ V j,i = FFT ( V j,i )denote the input and kernel on frequency domain, then we have:
Revised: i = FFT ( X i ) and ˜ V j,i = FFT ( V j,i )denote the input and kernel on frequency domain, then we have:

Original: [Equation]
Revised: [Equation]

Original: in which multiplication, transpose and inverse square root operations are matrix-wise.
Revised: in which multiplication, transpose and inverse square root operations are matrix-wise.

Original: Therefore, wecan first calculate FFT ( Y ) on the frequency domain and perform the inverse Fourier transformationto get the final output.
Revised: Therefore, wecan first calculate FFT ( Y ) on the frequency domain and perform the inverse Fourier transformationto get the final output.

Original: We will use Newton’s iteration to calculate the inverse square root of the positive semi-definite matrix
Revised: We will use Newton’s iteration to calculate the inverse square root of the positive semi-definite matrix

Original: A = ˜ V : , : ,a,b ˜ V ⊺ : , : ,a,b in a differentiable way [15].
Revised: A = ˜ V : , : ,a,b ˜ V ⊺ : , : ,a,b in a differentiable way [15].

Original: If we initialize Y 

Revised: If we initialize Y

Original: = A and Z 

Revised: = A and Z

Original: =
Revised: =

Original: I and perform thefollowing update rule:
Revised: I and perform thefollowing update rule:

Original: [Equation]
Revised: [Equation]

Original: Z k will converge to A − 12 when || I − A
Revised: Z k will converge to A − 12 when || I − A

Original: || 2 < 1 .
Revised: || 2 < 1 .

Original: The condition can be satisfied by rescaling theparameter V = αV , noticing that the scaling factor will not change the resulting W .
Revised: The condition can be satisfied by rescaling theparameter V = αV , noticing that the scaling factor will not change the resulting W .

Original: In practice, weexecute a finite number of Newton’s iteration steps and we provide a rigorous error bound for thisfinite iteration scheme in Appendix D to show that the error will decay exponentially.
Revised: In practice, weexecute a finite number of Newton’s iteration steps and we provide a rigorous error bound for thisfinite iteration scheme in Appendix D to show that the error will decay exponentially.

Original: Circular Padding vs. Zero Padding When we apply the convolution theorem to calculate theconvolution Y
Revised: Circular Padding vs. Zero Padding When we apply the convolution theorem to calculate theconvolution Y

Original: =
Revised: =

Original: W ◦ X with Fourier Transformation, the result implicitly uses circular padding.
Revised: W ◦ X with Fourier Transformation, the result implicitly uses circular padding.

Original: However, in neural networks, zero padding is usually a better choice.
Revised: However, in neural networks, zero padding is usually a better choice.

Original: Therefore, we will firstperform zero padding on both sides of input X pad =
Revised: Therefore, we will firstperform zero padding on both sides of input X pad =

Original: zero_pad ( X ) and calculate the resulting
Revised: zero_pad ( X ) and calculate the resulting

Original: Y pad =
Revised: Y pad =

Original: W ◦ X pad with Fourier Transformation.
Revised: W ◦ X pad with Fourier Transformation.

Original: Thus, the implicit circular padding in this processwill actually pad the zeros which we padded beforehand.
Revised: Thus, the implicit circular padding in this processwill actually pad the zeros which we padded beforehand.

Original: Finally, we truncate the padded part and getour desired output Y = truncate ( Y pad ) .
Revised: Finally, we truncate the padded part and getour desired output Y = truncate ( Y pad ) .

Original: We empirically observe that this technique helps improvethe model robustness as shown in Appendix E.3.
Revised: We empirically observe that this technique helps improvethe model robustness as shown in Appendix E.3.

Original: When Input and Output Dimensions Differ In previous discussion, we assume that the convolution kernel V has equal input and output channels.
Revised: When Input and Output Dimensions Differ In previous discussion, we assume that the convolution kernel V has equal input and output channels.

Original: In the case W ∈ R c out × c in × k × k where c out ̸ = c in ,we aim to get a semi-orthogonal convolution kernel W (i.e. W ◦ W ⊺ =
Revised: In the case W ∈ R c out × c in × k × k where c out ̸ = c in ,we aim to get a semi-orthogonal convolution kernel W (i.e. W ◦ W ⊺ =

Original: I conv if c out < c in or
Revised: I conv if c out < c in or

Original: W ⊺ ◦ W =
Revised: W ⊺ ◦ W =

Original: I conv if c out > c in ).
Revised: I conv if c out > c in ).

Original: As pointed out in [9], calculating W with Newton’s iteration willnaturally lead to a semi-orthogonal convolution kernel when the input and output dimensions differ.
Revised: As pointed out in [9], calculating W with Newton’s iteration willnaturally lead to a semi-orthogonal convolution kernel when the input and output dimensions differ.

Original: Emulating a Larger Stride To emulate the case when the convolution stride is 2, we followprevious works [20] and use the invertible downsampling layer [10], in which the input dimensionc in will be increased by × 4 times.
Revised: Emulating a Larger Stride To emulate the case when the convolution stride is 2, we followprevious works [20] and use the invertible downsampling layer [10], in which the input dimensionc in will be increased by × 4 times.

Original: Strides larger than 2 can be emulated with similar techniques ifneeded.
Revised: Strides larger than 2 can be emulated with similar techniques ifneeded.

Original: Overall Algorithm Taking the techniques we discussed before, we can get our final LOT convolution layer.
Revised: Overall Algorithm Taking the techniques we discussed before, we can get our final LOT convolution layer.

Original: The detailed algorithm is shown in Appendix B. First, we will pad the input to prevent theimplicit circular padding mechanism and pad the kernel so that they are in the same shape.
Revised: The detailed algorithm is shown in Appendix B. First, we will pad the input to prevent theimplicit circular padding mechanism and pad the kernel so that they are in the same shape.

Original: Next, weperform the Fourier transformation and calculate the output on the frequency domain with Newton’siteration.
Revised: Next, weperform the Fourier transformation and calculate the output on the frequency domain with Newton’siteration.

Original: Finally, we perform inverse Fourier transformation and return the desired output.
Revised: Finally, we perform inverse Fourier transformation and return the desired output.

Original: Comparison with SOC Several works have been proposed on orthogonal convolution layers withre-parametrization[22, 20], among which the SOC approach via W = exp
Revised: Comparison with SOC Several works have been proposed on orthogonal convolution layers withre-parametrization[22, 20], among which the SOC approach via W = exp

Original: ( V − V ⊺ ) has achieved thebest performance.
Revised: ( V − V ⊺ ) has achieved thebest performance.

Original: Compared with SOC, LOT has the following advantages.
Revised: Compared with SOC, LOT has the following advantages.

Original: First, the parametrizationin LOT is norm-independent.
Revised: First, the parametrizationin LOT is norm-independent.

Original: Rescaling V to αV will not change the resulting W .
Revised: Rescaling V to αV will not change the resulting W .

Original: By comparison,
Revised: By comparison,

Original: V with a smaller norm in SOC will lead to a W closer to the identity transformation.
Revised: V with a smaller norm in SOC will lead to a W closer to the identity transformation.

Original: Consideringthat the norm of V will be regularized during training (e.g. SOC will re-scale V to have a small norm; people usually initialize weight to be small and impose l2-regularization during training),the orthogonal weight space in SOC may be biased.
Revised: Consideringthat the norm of V will be regularized during training (e.g. SOC will re-scale V to have a small norm; people usually initialize weight to be small and impose l2-regularization during training),the orthogonal weight space in SOC may be biased.

Original: Second, we can see that LOT is able to modelany orthogonal kernel W by noticing that ( WW ⊺ ) − 12 W is W itself; by comparison, SOC cannotparametrize all the orthogonal operations.
Revised: Second, we can see that LOT is able to modelany orthogonal kernel W by noticing that ( WW ⊺ ) − 12 W is W itself; by comparison, SOC cannotparametrize all the orthogonal operations.

Original: For example, in the case of orthogonal matrices, theexponential of a skew-symmetric matrix only models the special orthogonal group (i.e. the matriceswith +1 determinant).
Revised: For example, in the case of orthogonal matrices, theexponential of a skew-symmetric matrix only models the special orthogonal group (i.e. the matriceswith +1 determinant).

Original: Third, we directly handle the case when c in ̸ = c out , while SOC needs to doextra padding so that the channel numbers match.
Revised: Third, we directly handle the case when c in ̸ = c out , while SOC needs to doextra padding so that the channel numbers match.

Original: Finally, LOT is more efficient during evaluation,when we only need to perform the Fourier and inverse Fourier transformation as extra overhead,while SOC needs multiple convolution operations to calculate the exponential.
Revised: Finally, LOT is more efficient during evaluation,when we only need to perform the Fourier and inverse Fourier transformation as extra overhead,while SOC needs multiple convolution operations to calculate the exponential.

Original: We will further showquantitative comparisons with SOC in Section 6.2.
Revised: We will further showquantitative comparisons with SOC in Section 6.2.

Original: The major limitation of LOT is its large overhead during training, since we need to calculate Newton’siteration in each training step which takes more time and memory.
Revised: The major limitation of LOT is its large overhead during training, since we need to calculate Newton’siteration in each training step which takes more time and memory.

Original: In addition, we sometimes observethat Newton’s iteration is not stable when we perform many steps with 32-bit precision.
Revised: In addition, we sometimes observethat Newton’s iteration is not stable when we perform many steps with 32-bit precision.

Original: To overcomethis problem, we will pre-calculate Newton’s iteration with 64-bit precision during evaluation, as wewill introduce in Section 4.2.
Revised: To overcomethis problem, we will pre-calculate Newton’s iteration with 64-bit precision during evaluation, as wewill introduce in Section 4.2.

Original: 4.2 Training and Evaluation of LOT
Revised: 4.2 Training and Evaluation of LOT

Original: Smoothing the Training Stage In practice, we observe that the LOT layers are highly non-smoothwith respect to the parameter V (see Section 6.3).
Revised: Smoothing the Training Stage In practice, we observe that the LOT layers are highly non-smoothwith respect to the parameter V (see Section 6.3).

Original: Therefore, the model is difficult to convergeduring the training process especially when the model is deep.
Revised: Therefore, the model is difficult to convergeduring the training process especially when the model is deep.

Original: To smooth the training, we proposetwo techniques.
Revised: To smooth the training, we proposetwo techniques.

Original: First, we will initialize all except bottleneck layers where c in = c out with identityparameter V
Revised: First, we will initialize all except bottleneck layers where c in = c out with identityparameter V

Original: =
Revised: =

Original: I .
Revised: I .

Original: The bottleneck layers where c in = c out will still be randomly initialized.
Revised: The bottleneck layers where c in = c out will still be randomly initialized.

Original: Second, as pointed out in [12], residual connection helps with model smoothness.
Revised: Second, as pointed out in [12], residual connection helps with model smoothness.

Original: Therefore, for theintermediate layers, we will add the 1-Lipschitz residual connection y
Revised: Therefore, for theintermediate layers, we will add the 1-Lipschitz residual connection y

Original: = λx + (1 − λ )
Revised: = λx + (1 − λ )

Original: f ( x ) .
Revised: f ( x ) .

Original: Somework suggests that λ here can be trainable [22], while we observe that setting λ = 0 .
Revised: Somework suggests that λ here can be trainable [22], while we observe that setting λ = 0 .

Original: 5 is enough.
Revised: 5 is enough.

Original: Speeding up the Evaluation Stage Notice that after the model is trained, the orthogonal kernel ˜
Revised: Speeding up the Evaluation Stage Notice that after the model is trained, the orthogonal kernel ˜

Original: Wwill no longer change.
Revised: Wwill no longer change.

Original: Therefore, we can pre-calculate Newton’s iteration and use the result for eachevaluation step.
Revised: Therefore, we can pre-calculate Newton’s iteration and use the result for eachevaluation step.

Original: Thus, the only runtime overhead compared with a standard convolution layer duringevaluation is the Fourier transformation part.
Revised: Thus, the only runtime overhead compared with a standard convolution layer duringevaluation is the Fourier transformation part.

Original: In addition, we observe that using 64-bit precisioninstead of the commonly used 32-bit precision in Newton’s iteration will help with the numericalstability.
Revised: In addition, we observe that using 64-bit precisioninstead of the commonly used 32-bit precision in Newton’s iteration will help with the numericalstability.

Original: Therefore, when we pre-calculate ˜ W , we will first transform ˜ V into float64.
Revised: Therefore, when we pre-calculate ˜ W , we will first transform ˜ V into float64.

Original: After Newton’siterations, we transform the resulting ˜ W back to float32 type for efficiency.
Revised: After Newton’siterations, we transform the resulting ˜ W back to float32 type for efficiency.

Original: 4.3 Semi-supervised Learning
Revised: 4.3 Semi-supervised Learning

Original: Existing works have shown that semi-supervised learning helps improve empirical robustness [4],while the impact of semi-supervised learning on certified robustness has not been explored yet.
Revised: Existing works have shown that semi-supervised learning helps improve empirical robustness [4],while the impact of semi-supervised learning on certified robustness has not been explored yet.

Original: We aimto bridge this gap and investigate whether applying semi-supervised learning also benefits the certifiedrobustness of Lipschitz constrained models.
Revised: We aimto bridge this gap and investigate whether applying semi-supervised learning also benefits the certifiedrobustness of Lipschitz constrained models.

Original: As we will discuss in Section 5.1, we theoreticallyshow that the certified robustness of a Lipschitz-bounded model can be improved by semi-supervisedlearning.
Revised: As we will discuss in Section 5.1, we theoreticallyshow that the certified robustness of a Lipschitz-bounded model can be improved by semi-supervisedlearning.

Original: In practice, suppose we have a (small) labelled dataset D s = { ( x i , y i ) } and a (large)unlabelled dataset D u = { x ′ i } , we will first train a labeller model G pl over the labelled dataset D s .
Revised: In practice, suppose we have a (small) labelled dataset D s = { ( x i , y i ) } and a (large)unlabelled dataset D u = { x ′ i } , we will first train a labeller model G pl over the labelled dataset D s .

Original: Then we use G pl to assign pseudo-labels to the unlabelled dataset to get D ′ u
Revised: Then we use G pl to assign pseudo-labels to the unlabelled dataset to get D ′ u

Original: = { ( x ′ i , G pl ( x ′ i )) } .
Revised: = { ( x ′ i , G pl ( x ′ i )) } .

Original: Finally, we use the overall dataset D s + D ′ u to train the student model G as the final model.
Revised: Finally, we use the overall dataset D s + D ′ u to train the student model G as the final model.

Original: Weassume that both G pl and G will use the LOT model architecture.
Revised: Weassume that both G pl and G will use the LOT model architecture.

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: In this section, we will introduce our theoretical analysis to show that the certified robustnessof Lipschitz-constrained models will benefit from semi-supervised learning.
Revised: In this section, we will introduce our theoretical analysis to show that the certified robustnessof Lipschitz-constrained models will benefit from semi-supervised learning.

Original: We make similarassumptions with [25] which provides error bound of discrete models in semi-supervised learning,while we will analyze a (Lipschitz-bounded) continuous model and its certified accuracy.
Revised: We make similarassumptions with [25] which provides error bound of discrete models in semi-supervised learning,while we will analyze a (Lipschitz-bounded) continuous model and its certified accuracy.

Original: We first introduce data expansion proposed in [25], which we will adopt in our analysis.
Revised: We first introduce data expansion proposed in [25], which we will adopt in our analysis.

Original: Let T denotesome set of transformations and define B ≜ { x ′ |∃ T ∈ T such that || x ′ − T ( x ) ||
Revised: Let T denotesome set of transformations and define B ≜ { x ′ |∃ T ∈ T such that || x ′ − T ( x ) ||

Original: <
Revised: <

Original: r } to be the setof points within distance r from any of the transformation of x .
Revised: r } to be the setof points within distance r from any of the transformation of x .

Original: The neighbourhood of x is defined by N = { x ′ |B
Revised: The neighbourhood of x is defined by N = { x ′ |B

Original: ( x ) ∩ B ( x ′ ) ̸ = ∅} .
Revised: ( x ) ∩ B ( x ′ ) ̸ = ∅} .

Original: Let P denote the data distribution and P i denote the distribution ofdata in the i -th class.
Revised: Let P denote the data distribution and P i denote the distribution ofdata in the i -th class.

Original: The expansion property is defined as:
Revised: The expansion property is defined as:

Original: Definition 5.1 ( ( a, c ) -expansion, as in [25]) .
Revised: Definition 5.1 ( ( a, c ) -expansion, as in [25]) .

Original: We say that the class-conditional distribution P i satisfies( a, c ) -expansion if for all V ⊆ X with P
Revised: We say that the class-conditional distribution P i satisfies( a, c ) -expansion if for all V ⊆ X with P

Original: i ( V ) <
Revised: i ( V ) <

Original: a , the following holds:
Revised: a , the following holds:

Original: [Equation]
Revised: [Equation]

Original: if P i satisfies ( a, c ) -expansion for all classes in P , then we say P satisfies ( a, c ) -expansion.
Revised: if P i satisfies ( a, c ) -expansion for all classes in P , then we say P satisfies ( a, c ) -expansion.

Original: Next, we define the classifier and the loss function in the continuous case.
Revised: Next, we define the classifier and the loss function in the continuous case.

Original: For simplicity, we assumea binary classification task here.
Revised: For simplicity, we assumea binary classification task here.

Original: Definition 5.2 (Continuous model G and loss functions) .
Revised: Definition 5.2 (Continuous model G and loss functions) .

Original: We assume a binary classification task,where G ∗ ( x ) , G pl ( x ) ∈ { 0 , 1 } are discrete labels, and the trained model is G ( x ) ∈ [0 , 1] .
Revised: We assume a binary classification task,where G ∗ ( x ) , G pl ( x ) ∈ { 0 , 1 } are discrete labels, and the trained model is G ( x ) ∈ [0 , 1] .

Original: We definethe loss function L m ( G, G ∗ ) ∈ [0 , 1] to be:
Revised: We definethe loss function L m ( G, G ∗ ) ∈ [0 , 1] to be:

Original: [Equation]
Revised: [Equation]

Original: and the error of G is defined as Err m ( G ) =
Revised: and the error of G is defined as Err m ( G ) =

Original: L m ( G, G ∗ ) .
Revised: L m ( G, G ∗ ) .

Original: Remark.
Revised: Remark.

Original: Here we provide a general setting here, and if we assume G ( x ) ∈ { 0 , 1 } , L m ( G, G ∗ )becomes the 0-1 loss in [25].
Revised: Here we provide a general setting here, and if we assume G ( x ) ∈ { 0 , 1 } , L m ( G, G ∗ )becomes the 0-1 loss in [25].

Original: Then we define the marginal consistency set S m B ( G ; δ ) and loss R m B ( G ; δ ) in the continuous case.
Revised: Then we define the marginal consistency set S m B ( G ; δ ) and loss R m B ( G ; δ ) in the continuous case.

Original: The consistency set includes cases in which the model prediction will not change a lot within theneighbourhood, and the consistency loss measures the probability that the model is not consistentover the population.
Revised: The consistency set includes cases in which the model prediction will not change a lot within theneighbourhood, and the consistency loss measures the probability that the model is not consistentover the population.

Original: Definition 5.3 (Marginal consistency set and consistency loss) .
Revised: Definition 5.3 (Marginal consistency set and consistency loss) .

Original: We define the set S m B ( G ; δ ) in whichthe ratio of marginal prediction change on each input x is no larger than δ within its neighbourhood:
Revised: We define the set S m B ( G ; δ ) in whichthe ratio of marginal prediction change on each input x is no larger than δ within its neighbourhood:

Original: [Equation]
Revised: [Equation]

Original: and R m B ( G ; δ ) denotes the marginal consistency loss of the probability that x is not in S m B ( G ; δ ) :
Revised: and R m B ( G ; δ ) denotes the marginal consistency loss of the probability that x is not in S m B ( G ; δ ) :

Original: R m B ( G ; δ ) =
Revised: R m B ( G ; δ ) =

Original: P x ∈ P [ x / ∈ S
Revised: P x ∈ P [ x / ∈ S

Original: m B ( G ; δ )] .
Revised: m B ( G ; δ )] .

Original: Remark.
Revised: Remark.

Original: (1) We may use S m B ( G ) and R m B ( G ) for abbreviation when the choice of δ does not leadto confusion; (2) when δ = 0 , S m B ( G ) and R m B ( G ) requires that the prediction is exactly the samewithin the neighbourhood, which are reduced to S B ( G ) and R B ( G ) defined in [25].
Revised: (1) We may use S m B ( G ) and R m B ( G ) for abbreviation when the choice of δ does not leadto confusion; (2) when δ = 0 , S m B ( G ) and R m B ( G ) requires that the prediction is exactly the samewithin the neighbourhood, which are reduced to S B ( G ) and R B ( G ) defined in [25].

Original: Finally, we define the certified robustness radius CertR ( G ) as the average radius over the populationin which the model keeps its correct prediction:
Revised: Finally, we define the certified robustness radius CertR ( G ) as the average radius over the populationin which the model keeps its correct prediction:

Original: [Equation]
Revised: [Equation]

Original: Note that when G ( x ) ̸ =
Revised: Note that when G ( x ) ̸ =

Original: G ∗ ( x ) , the certified radius r will be 0.
Revised: G ∗ ( x ) , the certified radius r will be 0.

Original: 5.1 Error Bound and Certified Radius of Lipschitz-bounded Model
Revised: 5.1 Error Bound and Certified Radius of Lipschitz-bounded Model

Original: Let M ( G pl ) =
Revised: Let M ( G pl ) =

Original: { x : G pl ( x ) ̸ =
Revised: { x : G pl ( x ) ̸ =

Original: G ∗ ( x ) } denote the set in which the pseudolabel is wrong.
Revised: G ∗ ( x ) } denote the set in which the pseudolabel is wrong.

Original: We willhave similar separation and expansion assumptions on the data distribution as in [25].
Revised: We willhave similar separation and expansion assumptions on the data distribution as in [25].

Original: Assumption 5.1 (Separation) .
Revised: Assumption 5.1 (Separation) .

Original: We assume that P is separable with probability 1 − µ by ground-truthclassifier G ∗ , i.e., R m B ( G ∗ ; 0) ≤ µ .
Revised: We assume that P is separable with probability 1 − µ by ground-truthclassifier G ∗ , i.e., R m B ( G ∗ ; 0) ≤ µ .

Original: Assumption 5.2 (Expansion) .
Revised: Assumption 5.2 (Expansion) .

Original: Define a ≜ max i { P i ( M ( G pl )) } to be the maximum fraction ofincorrectly pseudolabeled examples in any class.
Revised: Define a ≜ max i { P i ( M ( G pl )) } to be the maximum fraction ofincorrectly pseudolabeled examples in any class.

Original: We assume that a < 1 / 3 and P satisfies ( a, c ) -expansion for c > 3 .
Revised: We assume that a < 1 / 3 and P satisfies ( a, c ) -expansion for c > 3 .

Original: We express our bounds in terms of c ≜ min(1 /a, c ) .
Revised: We express our bounds in terms of c ≜ min(1 /a, c ) .

Original: With these assumptions and let δ ∈ [0 , 1 c ] , we can compare the performance of G and G pl (i.e. theperformance with and without semi-supervised learning) with the following theorem:
Revised: With these assumptions and let δ ∈ [0 , 1 c ] , we can compare the performance of G and G pl (i.e. theperformance with and without semi-supervised learning) with the following theorem:

Original: Theorem 5.1 (Error bound with semi-supervised learning) .
Revised: Theorem 5.1 (Error bound with semi-supervised learning) .

Original: Suppose Assumption 5.1 and 5.2 holdsand let ˆ
Revised: Suppose Assumption 5.1 and 5.2 holdsand let ˆ

Original: G be a minimizer of the following loss function:
Revised: G be a minimizer of the following loss function:

Original: [Equation]
Revised: [Equation]

Original: Then we can upper bound the error of ˆ G by:
Revised: Then we can upper bound the error of ˆ G by:

Original: [Equation]
Revised: [Equation]

Original: In addition, we can bound the certified robustness radius w.r.t.
Revised: In addition, we can bound the certified robustness radius w.r.t.

Original: the Lipschitz constant as below:
Revised: the Lipschitz constant as below:

Original: Theorem 5.2 (Certified radius with semi-supervised learning) .
Revised: Theorem 5.2 (Certified radius with semi-supervised learning) .

Original: Suppose Assumption 5.2 holds.
Revised: Suppose Assumption 5.2 holds.

Original: Thenthe certified radius of a model G can be bounded by:
Revised: Thenthe certified radius of a model G can be bounded by:

Original: [Equation]
Revised: [Equation]

Original: The proofs of Theorem 5.1 and 5.2 are shown in Appendix C.
Revised: The proofs of Theorem 5.1 and 5.2 are shown in Appendix C.

Original: Discussion From Theorem 5.1, we observe that the benefits of semi-supervised learning dependon the error of G pl and data property.
Revised: Discussion From Theorem 5.1, we observe that the benefits of semi-supervised learning dependon the error of G pl and data property.

Original: If the data expands and separates well and G pl is accurate, cis large and µ is small, which means Err m ( ˆ G ) will be smaller than Err ( G pl ) .
Revised: If the data expands and separates well and G pl is accurate, cis large and µ is small, which means Err m ( ˆ G ) will be smaller than Err ( G pl ) .

Original: Note that µ is usuallysmall and we assume c > 3 , so
Revised: Note that µ is usuallysmall and we assume c > 3 , so

Original: Err m ( ˆ G ) <
Revised: Err m ( ˆ G ) <

Original: Err ( G pl ) will hold true in most cases, indicating thatsemi-supervised learning helps improve the model performance.
Revised: Err ( G pl ) will hold true in most cases, indicating thatsemi-supervised learning helps improve the model performance.

Original: From Theorem 5.2, we can observethat given the data distribution and G pl , the certified radius of the student model CertR ( G ) can belower-bounded by 1) how well G learns from G pl
Revised: From Theorem 5.2, we can observethat given the data distribution and G pl , the certified radius of the student model CertR ( G ) can belower-bounded by 1) how well G learns from G pl

Original: ( L m ( G, G pl ) ); 2) how consistent G is ( R m B ( G ; δ ) ).
Revised: ( L m ( G, G pl ) ); 2) how consistent G is ( R m B ( G ; δ ) ).

Original: These two factors also correspond to the losses in the actual training process - the cross-entropy lossmeasures how well G learns from G pl and the Lipschitz constraint measures local consistency for a
Revised: These two factors also correspond to the losses in the actual training process - the cross-entropy lossmeasures how well G learns from G pl and the Lipschitz constraint measures local consistency for a

Original: Lipschitz-bounded model 1 .
Revised: Lipschitz-bounded model 1 .

Original:  Evaluation
Revised:  Evaluation

Original: In this section, we will evaluate our method LOT on different datasets and models compared with thestate-of-the-art baselines based on the deterministic ℓ 2 certified robustness.
Revised: In this section, we will evaluate our method LOT on different datasets and models compared with thestate-of-the-art baselines based on the deterministic ℓ 2 certified robustness.

Original: We show that on both
Revised: We show that on both

Original: CIFAR-10 and CIFAR-100, LOT achieves significantly higher certified accuracy under different radiiand settings with more efficient evaluation time.
Revised: CIFAR-10 and CIFAR-100, LOT achieves significantly higher certified accuracy under different radiiand settings with more efficient evaluation time.

Original: In the meantime, we conduct a series of ablationstudies to analyze the representation power of our LOT, the error control of Newton’s iterations, andthe effects of residual connections.
Revised: In the meantime, we conduct a series of ablationstudies to analyze the representation power of our LOT, the error control of Newton’s iterations, andthe effects of residual connections.

Original: We show that under the semi-supervised setting, the certifiedaccuracy of LOT is further improved and outperforms the baselines.
Revised: We show that under the semi-supervised setting, the certifiedaccuracy of LOT is further improved and outperforms the baselines.

Original: 6.1 Experiment Setup
Revised: 6.1 Experiment Setup

Original: Baseline - SOC with CReg loss, HH activation and LLN technique
Revised: Baseline - SOC with CReg loss, HH activation and LLN technique

Original: In the evaluation, we willmainly compare with SOC [20], as SOC has been shown to outperform other 1-Lipschitz models interms of certified robustness.
Revised: In the evaluation, we willmainly compare with SOC [20], as SOC has been shown to outperform other 1-Lipschitz models interms of certified robustness.

Original: SOC parameterizes an orthogonal layer with W = exp( V − V ⊺ ) where
Revised: SOC parameterizes an orthogonal layer with W = exp( V − V ⊺ ) where

Original: V is the convolution kernel and the transpose and exponential are with respect to the convolutionoperation.
Revised: V is the convolution kernel and the transpose and exponential are with respect to the convolutionoperation.

Original: Following their setting, we will focus on the CIFAR-10 and CIFAR-100 datasets and themodel architecture varies from LipConvnet-5 to LipConvnet-40.[21] proposes three techniques to improve SOC networks .
Revised: Following their setting, we will focus on the CIFAR-10 and CIFAR-100 datasets and themodel architecture varies from LipConvnet-5 to LipConvnet-40.[21] proposes three techniques to improve SOC networks .

Original: First, they observe that adding a Certificate
Revised: First, they observe that adding a Certificate

Original: Regularization (CReg) loss can help with the certified performance at larger radius: ℓ CReg ( x, y )
Revised: Regularization (CReg) loss can help with the certified performance at larger radius: ℓ CReg ( x, y )

Original: =− γ ReLU (cid:0) (
Revised: =− γ ReLU (cid:0) (

Original: f ( x ) y − max i ̸
Revised: f ( x ) y − max i ̸

Original: = y f ( x ) i ) / √ 2 (cid:1) .
Revised: = y f ( x ) i ) / √ 2 (cid:1) .

Original: Second, they propose a Householder (HH) activation layer,which is a generalization of the standard GroupSort activation.
Revised: Second, they propose a Householder (HH) activation layer,which is a generalization of the standard GroupSort activation.

Original: Finally, for tasks with large number ofclasses (e.g. CIFAR-100), they propose Last Layer Normalization(LLN) to only normalize the finaloutput layer instead of orthogonalize.
Revised: Finally, for tasks with large number ofclasses (e.g. CIFAR-100), they propose Last Layer Normalization(LLN) to only normalize the finaloutput layer instead of orthogonalize.

Original: Since these techniques are agnostic to the type of convolutionoperation, we will further integrate them with LOT for comparison.
Revised: Since these techniques are agnostic to the type of convolutionoperation, we will further integrate them with LOT for comparison.

Original: Implementation Details We will train and evaluate the LOT network under supervised and unsupervised scenarios.
Revised: Implementation Details We will train and evaluate the LOT network under supervised and unsupervised scenarios.

Original: Following previous works [20], we focus on the CIFAR-10 and CIFAR-100datasets and the model architecture varies from LipConvnet-5 to LipConvnet-40.
Revised: Following previous works [20], we focus on the CIFAR-10 and CIFAR-100datasets and the model architecture varies from LipConvnet-5 to LipConvnet-40.

Original: In semi-supervisedlearning, we use the 500K data introduced in [4] as the unlabelled dataset.
Revised: In semi-supervisedlearning, we use the 500K data introduced in [4] as the unlabelled dataset.

Original: To train the LOT network,we will train the model for 200 epochs using a momentum SGD optimizer with an initial learningrate 0.1 and decay by 0.1 at the 100-th and 150-th epochs.
Revised: To train the LOT network,we will train the model for 200 epochs using a momentum SGD optimizer with an initial learningrate 0.1 and decay by 0.1 at the 100-th and 150-th epochs.

Original: We use Newton’s iteration with 10 stepswhich we observe is enough for convergence (see Appendix E.4).
Revised: We use Newton’s iteration with 10 stepswhich we observe is enough for convergence (see Appendix E.4).

Original: When CReg loss is applied, weuse γ = 0 .
Revised: When CReg loss is applied, weuse γ = 0 .

Original: 5 ; when HH activation is applied, we use the version of order 1.
Revised: 5 ; when HH activation is applied, we use the version of order 1.

Original: We add the residual
Revised: We add the residual

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: connection with a fixed λ = 0 .
Revised: connection with a fixed λ = 0 .

Original: 5 for LOT; for SOC, we use their original version, as we observethat residual connections even hurt their performance (see discussions in Section 6.3).
Revised: 5 for LOT; for SOC, we use their original version, as we observethat residual connections even hurt their performance (see discussions in Section 6.3).

Original: We show thecertified accuracy at radius ρ ∈ { 36255 , 72255 , 108255 } .
Revised: We show thecertified accuracy at radius ρ ∈ { 36255 , 72255 , 108255 } .

Original: For the evaluation time comparison, we show theruntime taken to do a full pass on the testing set evaluated on an NVIDIA RTX A6000 GPU.
Revised: For the evaluation time comparison, we show theruntime taken to do a full pass on the testing set evaluated on an NVIDIA RTX A6000 GPU.

Original: 6.2 Supervised Scenario
Revised: 6.2 Supervised Scenario

Original: We show the results of supervised learning on CIFAR-10 and CIFAR-100 in Table 1.
Revised: We show the results of supervised learning on CIFAR-10 and CIFAR-100 in Table 1.

Original: We can observethat LOT significantly outperforms baselines on both vanilla accuracy and certified accuracy fordifferent datasets.
Revised: We can observethat LOT significantly outperforms baselines on both vanilla accuracy and certified accuracy fordifferent datasets.

Original: In particular, we observe that the improvement is more significant for deepermodels.
Revised: In particular, we observe that the improvement is more significant for deepermodels.

Original: We owe it to the reason that our LOT layer has a better expressiveness and unbiasedparametrization.
Revised: We owe it to the reason that our LOT layer has a better expressiveness and unbiasedparametrization.

Original: “Mean evaluation time” column in Table 1 records the evaluation time per instanceaveraged between CIFAR-10 and CIFAR-100 models.
Revised: “Mean evaluation time” column in Table 1 records the evaluation time per instanceaveraged between CIFAR-10 and CIFAR-100 models.

Original: Since architectures of CIFAR-10 and CIFAR100 models differ only in the last linear year, the evaluation time is almost the same.
Revised: Since architectures of CIFAR-10 and CIFAR100 models differ only in the last linear year, the evaluation time is almost the same.

Original: We observe that
Revised: We observe that

Original: LOT has a better efficiency during the evaluation stage, since LOT pre-calculates Newton’s iterationwhile SOC calculates the exponential in each iteration.
Revised: LOT has a better efficiency during the evaluation stage, since LOT pre-calculates Newton’s iterationwhile SOC calculates the exponential in each iteration.

Original: A similar conclusion can be observed fromresults without CReg loss, HH activation and LLN, as shown in Table 5 and 6 in Appendix E.5.
Revised: A similar conclusion can be observed fromresults without CReg loss, HH activation and LLN, as shown in Table 5 and 6 in Appendix E.5.

Original: 6.3 Ablation Studies
Revised: 6.3 Ablation Studies

Original: In this section, we will perform several ablation studies for LOT.
Revised: In this section, we will perform several ablation studies for LOT.

Original: Unless specified, we will use thedeep model LipConvnet-20 for evaluation.
Revised: Unless specified, we will use thedeep model LipConvnet-20 for evaluation.

Original: We use the model without CReg loss and HH activationso that we can see the comparison under the standard CNN setting.
Revised: We use the model without CReg loss and HH activationso that we can see the comparison under the standard CNN setting.

Original: Representation analysis Recent works suggest that adversarially robust models will have a goodregularization-free feature visualization [1].
Revised: Representation analysis Recent works suggest that adversarially robust models will have a goodregularization-free feature visualization [1].

Original: We show the visualization of 4 neurons in the last hiddenlayer of LipConvNet-20 for both SOC and LOT in Figure 1.
Revised: We show the visualization of 4 neurons in the last hiddenlayer of LipConvNet-20 for both SOC and LOT in Figure 1.

Original: More figures are shown in Appendix E.

Revised: More figures are shown in Appendix E.

Original: The visualization process is as follows: given a chosen neuron, we will start from a random image andtake 500 gradient steps to maximize the neuron value with step size 1.0 and decay factor 0.1.
Revised: The visualization process is as follows: given a chosen neuron, we will start from a random image andtake 500 gradient steps to maximize the neuron value with step size 1.0 and decay factor 0.1.

Original: We cansee that LOT indeed generates more meaningful features.
Revised: We cansee that LOT indeed generates more meaningful features.

Original: We attribute it to the good expressivenesspower of our model.
Revised: We attribute it to the good expressivenesspower of our model.

Original: This indicates that our approach indeed leads to better representations.
Revised: This indicates that our approach indeed leads to better representations.

Original: Error Control of Newton’s Iterations
Revised: Error Control of Newton’s Iterations

Original: To see how our Newton’s iteration approximates the inversesquare root, we visualize the maximum singular value ( σ max ) of LOT layers in Appendix E.4.
Revised: To see how our Newton’s iteration approximates the inversesquare root, we visualize the maximum singular value ( σ max ) of LOT layers in Appendix E.4.

Original: Weobserve that all the resulting layers have a strictly < 1 Lipschitz.
Revised: Weobserve that all the resulting layers have a strictly < 1 Lipschitz.

Original: Therefore, we can safely concludethat the overall LOT network is 1-Lipschitz.
Revised: Therefore, we can safely concludethat the overall LOT network is 1-Lipschitz.

Original: [Table]
Revised: [Table]

Original: Results of different architectures are shown in Table 7 in Appendix E.6.
Revised: Results of different architectures are shown in Table 7 in Appendix E.6.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Effect of residual connections Empirically, we observe that LOT is non-smooth and therefore we needthe residual connection to smoothify the model.
Revised: Effect of residual connections Empirically, we observe that LOT is non-smooth and therefore we needthe residual connection to smoothify the model.

Original: Toverify this phenomenon, we adopt the visualizationapproach in [12] and show the loss surface in Figure 2.
Revised: Toverify this phenomenon, we adopt the visualizationapproach in [12] and show the loss surface in Figure 2.

Original: We can see that LOT without residual connection is highly non-smooth, while after adding residual connections the model becomes much smoother.
Revised: We can see that LOT without residual connection is highly non-smooth, while after adding residual connections the model becomes much smoother.

Original: By comparison, we observe that SOC networks withand without residual connection are similar (see Appendix E.2).
Revised: By comparison, we observe that SOC networks withand without residual connection are similar (see Appendix E.2).

Original: We hypothesize the reason to be that theexponential parametrization W = exp( A ) enablesresidual property implicitly, i.e.,
Revised: We hypothesize the reason to be that theexponential parametrization W = exp( A ) enablesresidual property implicitly, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where the first two terms are essentially the format of residual connection.
Revised: where the first two terms are essentially the format of residual connection.

Original: We show the resultsof models with and without residual connections in Table 2.
Revised: We show the resultsof models with and without residual connections in Table 2.

Original: We can observe that our LOT indeedimproves with the residual connection, while the performance remains similar or even worse for SOC.
Revised: We can observe that our LOT indeedimproves with the residual connection, while the performance remains similar or even worse for SOC.

Original: Therefore, in the evaluation, we use the LOT with residual connection and SOC without it.
Revised: Therefore, in the evaluation, we use the LOT with residual connection and SOC without it.

Original: 6.4 Semi-supervised Learning Scenario
Revised: 6.4 Semi-supervised Learning Scenario

Original: We show the results of semi-supervised learning for one architecture with CReg loss and HH activationin Table 3.
Revised: We show the results of semi-supervised learning for one architecture with CReg loss and HH activationin Table 3.

Original: The full results are shown in Table 7 and 8 in Appendix E.6.
Revised: The full results are shown in Table 7 and 8 in Appendix E.6.

Original: We can observe that, withsemi-supervised learning, the vanilla accuracy will drop slightly, which will impact the certifiedaccuracy at a small radius ( ρ = 36 / 255 ).
Revised: We can observe that, withsemi-supervised learning, the vanilla accuracy will drop slightly, which will impact the certifiedaccuracy at a small radius ( ρ = 36 / 255 ).

Original: However, the certified accuracy at a large radius willbe improved, and the gap is more significant at a larger radius (e.g. ρ = 108 / 255 ).
Revised: However, the certified accuracy at a large radius willbe improved, and the gap is more significant at a larger radius (e.g. ρ = 108 / 255 ).

Original: We improvethe previous state-of-the-art certified accuracy by over 6% at ρ = 108 / 255 .
Revised: We improvethe previous state-of-the-art certified accuracy by over 6% at ρ = 108 / 255 .

Original: This indeed showsthat semi-supervised learning can help improve certified robustness.
Revised: This indeed showsthat semi-supervised learning can help improve certified robustness.

Original: We empirically observe thatsemi-supervised learning does not help much on CIFAR-100.
Revised: We empirically observe thatsemi-supervised learning does not help much on CIFAR-100.

Original: We owe it to the reason that thevanilla accuracy of the teacher model on CIFAR-100 is low (less than 50%).
Revised: We owe it to the reason that thevanilla accuracy of the teacher model on CIFAR-100 is low (less than 50%).

Original: Thus, according to
Revised: Thus, according to

Original: Assumption 5.2 and Theorem 5.2, when c is low, the certified accuracy would be low.
Revised: Assumption 5.2 and Theorem 5.2, when c is low, the certified accuracy would be low.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we propose an orthogonal convolution layer LOT and build a 1-Lipschitz convolutionnetwork.
Revised: In this work, we propose an orthogonal convolution layer LOT and build a 1-Lipschitz convolutionnetwork.

Original: We show that LOT network outperforms the previous state-of-the-art in certified robustness.
Revised: We show that LOT network outperforms the previous state-of-the-art in certified robustness.

Original: We also show that semi-supervised learning can further help with the robustness both theoreticallyand experimentally.
Revised: We also show that semi-supervised learning can further help with the robustness both theoreticallyand experimentally.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes](b) Did you describe the limitations of your work?

Original: [Yes] We point out our limitations inthe comparison with previous works in Section 4.1.(c
Revised: [Yes] We point out our limitations inthe comparison with previous works in Section 4.1.(c

Original: ) Did you discuss any potential negative societal impacts of your work?
Revised: ) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] We discussthem in Appendix A.
Revised: [Yes] We discussthem in Appendix A.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] We state theassumptions in Section 5.(b)
Revised: [Yes] We state theassumptions in Section 5.(b)

Original: Did you include complete proofs of all theoretical results?
Revised: Did you include complete proofs of all theoretical results?

Original: [Yes] We include thecomplete proof in Appendix C.
Revised: [Yes] We include thecomplete proof in Appendix C.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] We upload it as
Revised: [Yes] We upload it as

Original: supplementary material.
Revised: supplementary material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] We specify them in Section 6.1.
Revised: [Yes] We specify them in Section 6.1.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running exper-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running exper-

Original: iments multiple times)?
Revised: iments multiple times)?

Original: [Yes] We tried multiple architectures and showed that the
Revised: [Yes] We tried multiple architectures and showed that the

Original: results are consistent across all these different settings in Section 6.2.
Revised: results are consistent across all these different settings in Section 6.2.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] We specify them in Section 6.1.
Revised: [Yes] We specify them in Section 6.1.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We cite the previousstate-of-the-art [20].
Revised: [Yes] We cite the previousstate-of-the-art [20].

Original: ) Did you mention the license of the assets?
Revised: ) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: ) Did you include any new assets either in the supplemental material or as a URL?
Revised: ) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: ) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: ) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: ) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: ) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: Did you include the full text of instructions given to participants and screenshots, if
Revised: Did you include the full text of instructions given to participants and screenshots, if

Original: applicable
Revised: applicable

Original: N
Revised: N

Original: Did you describe any potential participant risks, with links to Institutional Review
Revised: Did you describe any potential participant risks, with links to Institutional Review

Original: ) approvals, if applicable
Revised: ) approvals, if applicable

Original: [N
Revised: [N

Original: Did you include the estimated hourly wage paid to participants and the total amount
Revised: Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Broader Impact

Original: Adversarial attacks have been investigated a lot and people are worried that the vulnerability ofmachine learning models may affect their application in real world.

Original: This paper, as a potential countermeasure against the adversarial attacks, proposes a type of model architecture that can guarantee themodel robustness under certain adversarial attacks.

Original: This could help to secure the safe deploymentof ML models in real applications, thus promoting the development of various ML systems.

Original: Inaddition, better understanding of the Lipschitz property of machine learning models may help peopleunderstand and explain how the models work, which is a key concern when applying these algorithmsin practice.

Original: On the other hand, the misuse of our technique may also lead to negative impact.

Original: Forexample, if an attacker is fully familiar with this work, he/she may discover some specific attack tofool this model (beyond ℓ 2 attacks which we can certify against).

Original: Therefore, in real applications, it isrecommended that different defense techniques are applied together to secure the model safety.

Original: B Pseudocode of LOT Layer

Original: We show the detailed pseudocode of our LOT layer in Algorithm 1.

Original: Algorithm 1 LOT layer.

Original: Require:

Original: Unconstrained convolution kernel V ∈ R c out × c in × k × k ; Input tensor X ∈ R c in × w × w .

Original: 1: X pad =

Original: zero_pad ( X, ( k, k, k, k ))

Original: ∈ R c in × ( w +2 k ) × ( w +2 k ) .

Original: 2: V pad =

Original: zero_pad ( W, (0 , 0 , k + w, k + w ))

Original: ∈

Original: R c out × c in × ( w +2 k ) × ( w +2 k ) .

Original: 3: // Calculate the Fourier transformation: 4: for all i ∈ { 1 , . .

Original: , c in } do 5:

Original: ˜ X i = FFT ( X padi ) .

Original: 6: for all j ∈ { 1 , . . .

Original: , c out } do 7:

Original: ˜ V j,i =

Original: FFT ( V padj,i ) .

Original: 8: end for 9: end for 10: // Calculate the output on frequency domain: 11: for all a, b ∈ { 1 , . . .

Original: , w + 2 k } do 12:

Original: ˆ V =

Original: ˜

Original: V : , : ,a,b (cid:113) ||

Original: ˜ V : , : ,a,b ˜ V ⊺ : , : ,a,b || F // Rescale ˜ V .

Original: Calculate ˜ W : , : ,a,b =

Original: ( ˆ V ˆ V ⊺ ) − 12 ˆ V with Newton’s iteration.

Original: 14: ˜ Y : ,a,b = ˜ W : , : ,a,b ˜ X : ,a,b .

Original: 15: end for 16: // Get the final output: 17: for all i ∈ { 1 , . .

Original: , c out } do 18:

Original: Y i =

Original: FFT − 1 ( ˜ Y i ) .

Original: 19: end for 20: return ( Y : ,k : w + k,k : w + k ) .real

Original: C Proof of Theorem 5.1 and Theorem 5.2C.1 Proof

Original: To prove Theorem 5.1 and Theorem 5.2, we first define the loss on a subset, so that we can divide theloss into different subset losses.

Original: Definition C.1 (Subset loss) .

Original: Define conditional loss:

Original: L cond m ( G, G ∗ | S ) =

Original: E x ∈ S

Original: [ G ∗ ( x )(1 − G ( x ))

Original: + (1 − G ∗ ( x ))

Original: G ( x )] to bethe loss function calculated only over the subset S ⊆ X , and define the subset loss:

Original: L m ( G, G ∗ | S ) =

Original: L cond m ( G, G ∗ | S ) · P ( S ) .

Original: A property of the subset loss is that, if S = S 1 ∪ S 2 where S 1 and S 2 are disjoint, then L m ( G, G ∗ | S ) =

Original: L m ( G, G ∗ | S 1 ) + L m ( G, G ∗ | S 2 ) .

Original: First, we provide some facts on the relationship between subset loss L m ( G, G ∗ | S ) and L m ( G, G pl | S )when S ⊆ M ( G pl ) or

Original: S ⊆ M ( G pl ) .

Original: Fact C.1.

Original: When S ⊆ M ( G pl ) , then L m ( G, G ∗ | S ) =

Original: L m ( G, G pl | S ) ;

Original: when S ⊆ M ( G pl ) , then

Original: L m ( G, G ∗ | S ) + L m ( G, G pl | S )

Original: =

Original: P ( S ) .

Original: Proof.

Original: This is easy to see by noticing that when x ∈ M ( G pl ) , then G pl ( x ) =

Original: G ∗ ( x ) ; whenx ∈ M ( G pl ) , then G pl ( x ) ̸ =

Original: G ∗ ( x ) , so G pl ( x ) + G ∗ ( x ) = 1 .

Original: Now, we define S 1 =

Original: S m B ( G ) ∩ M ( G pl ) and S 2 =

Original: S m B ( G ) ∩ M ( G pl ) .

Original: Base on the fact, we will beable to derive the relationship between L m ( G, G ∗ | S 1 ) and L m ( G, G ∗ | S 2 ) :

Original: Lemma C.2.

Original: We have the following relationship between the losses on the sets in which G pl iscorrect vs. G pl is wrong:

Original: [Equation]

Original: And therefore,

Original: [Equation]

Original: Proof.

Original: For the first equation, note that S 2 ⊆ M ( G pl ) and S 1 ⊆ M ( G pl ) , so based on the previousfact, we have:

Original: [Equation]

Original: For the second inequation, notice M ( G pl ) \ S m B ( G ) ⊆

Original: S 1 , so P ( S 1 ) ≥ P ( M ( G pl ))

Original: − P ( S m B

Original: G ))

Original: =

Original: Err ( G pl ) − R m B ( G ) .

Original: So:

Original: [Equation]

Original: Fact C.3.

Original: Given δ ∈ [0 , 1 c ] and β ∈ (0 , c − 12 ] , we can verify that:

Original: [Equation]

Original: Proof.

Original: This can be verified by substituting δ = 1 c into LHS, noticing that LHS is monotonicallyincreasing w.r.t.

Original: δ .

Original: Lemma C.4.

Original: For any β ∈ (0 , c − 12 ] , define q = βc − 1 Err ( G pl ) and α = ( β − 1)

Original: Err ( G pl ) .

Original: If G fits thepseudolabels with suffcient accuracy and consistency:

Original: [Equation]

Original: Then G satisfies the following error bound:

Original: [Equation]

Original: The intuition of the proof is as follows.

Original: Lemma C.2 provides a relationship between the loss

Original: L m ( G, G ∗ | S 1 ) and L m ( G, G ∗ | S 2 ) .

Original: On the other hand, the expansion of S 1 is also related with S 2 by( N ( S 1 ) \ S 1 ) ∩ S m B ( G ) ⊆

Original: S 2 .

Original: Note that the expansion P ( N ( S 1 )) >

Original: c · P ( S 1 ) , so S 1 cannot be toolarge or otherwise N ( S 1 ) \ S 1 will be too large to be within S 2 .

Original: We will show that L m ( G, G ∗ | S 1 ) < q .

Original: Proof.

Original: Consider the expansion of S 1 , N ( S 1 ) .

Original: Since S 1 ⊆ M ( G pl ) and P ( G pl ) < 1 /c , we knowthat c · P ( S 1 ) < 1 , so by the assumption of expansion, P ( N ( S 1 )) ≥ c · P ( S 1 ) .

Original: In addition, noticethat ( N ( S 1 ) \ S 1 ) ∩ S m B ( G ) ⊆

Original: S 2 .

Original: Therefore, we have:

Original: [Equation]

Original: For the first term, we notice that N ( S 1 ) ∩ S m B ( G ) ⊆

Original: N ( S 1 ) and S 1 ⊆ S m B

Original: G ) , so the conditional losssatisfies L ( G, G ∗ |N

Original: ( S 1 ) ∩ S m B ( G )) /P

Original: ( N ( S 1 ) ∩ S m B ( G )) ≥ (1 − δ )

Original: L ( G, G ∗ | S 1 ) /P

Original: ( S 1 ) .

Original: Therefore,

Original: [Equation]

Original: |

Original: [Equation]

Original: L

Original: [Equation]

Original: ( G,

Original: [Equation]

Original: |N

Original: [Equation]

Original: ∩

Original: [Equation]

Original: ( G

Original: [Equation]

Original: Now, substituting L ( G, G ∗ | S 2 ) on the LHS with Lemma C.2 and noticing M ( G pl ) \ S m B

Original: ( G ) ⊆

Original: S 1 ,with simple transformation we have:

Original: [Equation]

Original: The last inequality comes from the condition in the lemma.

Original: Thus, with Fact C.3, we know

Original: L ( G, G ∗ | S 1 ) ≤ α/ (( c (1 − δ ) − 2)) ≤ q .

Original: Now, we can bound the overall error:

Original: [Equation]

Original: Now, we will prove our main lemma, based on which we will be able to derive Theorem 5.1 and

Original: Theorem 5.2.

Original: Lemma C.5.

Original: Suppose Assumption 5.2 holds true.

Original: Then we can bound:

Original: [Equation]

Original: for any δ ∈ [0 , 1 c ] .

Original: Proof.

Original: First, we consider the case where L m ( G, G pl ) + 2 R m B ( G ) ≤ c − 12 ·

Original: Err ( G pl ) .

Original: In this case, wecan find some β ∈ (0 , c − 12 ] such that.

Original: [Equation]

Original: Thus, by lemma C.4, we have:

Original: [Equation]

Original: Next, we consider the case where L m ( G, G pl ) + 2 R m B ( G ) >

Original: c − 12 · Err ( G pl ) .

Original: By triangle inequality,we have:

Original: [Equation]

Original: Now, we provide the proof of Theorem 5.1 and Theorem 5.2 based on Lemma C.5.

Original: Proof of Theorem 5.1.

Original: Since ˆ G is an optimizer of L ( G ) , we know Err m

Original: ( ˆ G ) ≤ L ( ˆ G ) ≤ L ( G ∗ ) .

Original: Substituting G ∗ into L ( G ) gives the bound in the theorem.

Original: Proof of Theorem 5.2.

Original: Note that CertR ( G ) ≥ 0 .

Original: 5 − Err ( G ) Lip ( G ) by its definition.

Original: Substituting Err ( G ) ≤

Original: L ( G ) gives the inequality in the theorem.

Original: D Error Control of Newton’s Method

Original: Recall that given an unconstrained matrix V ∈ R n × n , we know that W

Original: [-= ( V V T ) − 12 V is orthogonal,i.e., || W || 
-]
Revised: [-= ( V V T ) − 12 V is orthogonal,i.e., || W || 
-]

Original: = 1 , which provides certified robustness for the resulting model.

Original: In practice, we use afinite number of Newton’s iteration steps to approximate ( V V T ) − 12 .

Original: In this section, we provide thefollowing theorem which rigorously control the spectral norm under finite Newton’s iteration steps.

Original: Theorem D.1.

Original: Given a matrix V ∈ R n × n such that

Original: ||

Original: I − V V T

Original: [-|| 2 < 1 , if we use Newton’s iteratonfor k ⋆ steps following Equation (1) with initialization Y 
-]
Revised: [-|| 2 < 1 , if we use Newton’s iteratonfor k ⋆ steps following Equation (1) with initialization Y 
-]

Original: [-= V V T and Z 
-]
Revised: [-= V V T and Z 
-]

Original: =

Original: I , then we have||

Original: Z k ⋆ V |

Original: [Equation]

Original: +

Original: [Equation]

Original: where ρ min is the smallest eigenvalue of the matrix.

Original: Remark.

Original: We make sure the condition

Original: ||

Original: I − V V T

Original: || 2 < 1 is satisfied by rescaling as discussed in

Original: Section 4.1.

Original: As the theorem shows, along with the increase of Newton’s iteration step k ⋆ , the spectralnorm of Z k ⋆ V approaches 1 where the additional term

Original: ||

Original: I − V V T

Original: || 2 k 2 decays exponentially.

Original: Hence,we can rigorously bound the error in the orthogonalization process caused by finite steps and usethe bound to determine how many finite steps are needed.

Original: Indeed, in practice, we apply singularvalue decomposition to the computed matrix Z k ⋆ V , and find its maximum singular value alwaysapproaches 1 from the left side, i.e., the actual spectral norm is equal to or smaller than 1 .

Original: Detailexperimental verification is in Appendix E.4.

Original: D.1 Proof of Theorem D.

Original: For brevity, throughout this section, for V ∈ R n × n , we define A = V V T .

Original: Note that A is a realsymmetric matrix.

Original: Then, we recursively define

Original: [Equation]

Original: Before proving the main theorem, we first present the following three lemmas.

Original: Lemma D.1.

Original: For

Original: Proof of Lemma D.1.

Original: We prove the lemma by induction.

Original: [-Since B 0 = I , for k = 
-]
Revised: [-Since B 0 = I , for k = 
-]

Original: the lemma holds.

Original: Suppose that the lemma holds for k , then we have

Original: [Equation]

Original: In above equations, ( ∗ ) is due to the induction assumption that B k A = AB k or B k

Original: [-A 
-]
Revised: [-A 
-]

Original: =

Original: A 12 B k .

Original: Therefore, the lemma holds with k + 1 and by induction the lemma holds for any k ∈ N .

Original: Lemma D.2.

Original: For any k ∈ N , Y k

Original: = B k A and Z k = B k .

Original: Proof of Lemma D.2.

Original: We prove the lemma by induction.

Original: Since Y 0 =

Original: V V T =

Original: [-A , Z 
-]
Revised: [-A , Z 
-]

Original: = I , and

Original: [-B 
-]
Revised: [-B 
-]

Original: =

Original: [-I , for k = 
-]
Revised: [-I , for k = 
-]

Original: the lemma holds.

Original: Suppose that the lemma holds for k , then we have

Original: [Equation]

Original: where ( ∗ ) leverages B k A = AB k from Lemma D.1.

Original: Therefore, by induction the lemma holds forany k ∈ N .

Original: Lemma D.3.

Original: When || I − A

Original: || 2 < 1 , for any k ∈ N , the eigenvalue λ ∈ R of matrix

Original: A 12 B k is positive.

Original: Proof of Lemma D.3.

Original: We define C k

Original: =

Original: A 12 B k , then by leveraging the commutability between B kand

Original: A / A 12 (Lemma D.1)

Original: we have the following iteration:

Original: [Equation]

Original: Since || I − A

Original: [-|| 2 < 1 and A is a real symmetric matrix, any eigenvalue of C 0 , denoted by λ C 
-]
Revised: [-|| 2 < 1 and A is a real symmetric matrix, any eigenvalue of C 0 , denoted by λ C 
-]

Original: i ,∈ (0 , √ 2) .

Original: [-Denoting the diagonalization of C 0 by C 
-]
Revised: [-Denoting the diagonalization of C 0 by C 
-]

Original: = P − 1 D 0 P where D 0 = diag( λ C 0 1 , · · · , λ C 0 n ) .

Original: [-Then, from the iteration, we have C k +
-]
Revised: [-Then, from the iteration, we have C k +
-]

Original: =

Original: P − 1 (cid:0) 12 (3 D k − D 3 k ) (cid:1) P and therefore λ

Original: C k

Original: +1 i

Original: =

Original: λ C k i (cid:16) 32 − 12 ( λ C k i ) 2 (cid:17) .

Original: Define function f : R → R such that f ( x ) =

Original: x ( 32 − 12 x 2 ) .

Original: We findf ′

Original: [-( x ) = 
-]
Revised: [-( x ) = 
-]

Original: ⇒ x ∈ { 0 , 1 } .

Original: Thus, when x ∈ (0 , √ 2) , f ( x ) ∈ (0 , 1) ⊆ (0 , √ 2) .

Original: Now we applythe induction.

Original: When k = 0 , we have λ C k i ∈ (0 , √ 2) .

Original: Suppose λ C k i ∈ (0 , √ 2) , from above resultwe have λ C k

Original: +1 i

Original: = f ( λ C k i ) ∈

Original: (0 , √ 2) .

Original: Thus, for any k ∈ N , all eigenvalues of C k

Original: =

Original: A 12 B k arepositive.

Original: Now we are ready to prove the main theorem.

Original: Proof of Theorem D.1.

Original: Since Z k = B k for any k ∈ N acccording to Lemma D.2, we focus on

Original: B k and its expression of iteration (Equation (3))

Original: henceforth.

Original: According to [3, Section 6], define

Original: R k

Original: [-= I − B 2 k A , we have R k +
-]
Revised: [-= I − B 2 k A , we have R k +
-]

Original: = 34 R 2 k + 14 R 3 k .

Original: We have ||

Original: I − A

Original: [-|| 
-]
Revised: [-|| 
-]

Original: [-= || R 
-]
Revised: [-= || R 
-]

Original: || 2 < 1 .

Original: By induction,

Original: [Equation]

Original: induction

Original: [Equation]

Original: Therefore, || R k ⋆ || 2 ≤

Original: || I − A

Original: [-|| 
-]
Revised: [-|| 
-]

Original: k⋆ 2 , i.e., the eigenvalues of the symmetric matrix I − B 2 k ⋆ A are in therange [ −|| I − A

Original: || 2 k⋆ 2 , ||

Original: I − A

Original: || 2 k⋆ 2 ] .

Original: Given an eigenvalue λ ∈ R with eigenvector x ∈ R n of the real symmetric matrix A 12 B k ⋆ , we have

Original: [Equation]

Original: [Equation]

Original: Thus, (1 − λ 2 ) is an eigenvalue of matrix ( I − B 2 k ⋆ A ) .

Original: Since (1 − λ 2 ) ∈ [ −|| I − A

Original: || 2 k⋆ 2 , ||

Original: I − A

Original: || 2 k⋆ 2 ] ,we get

Original: [Equation]

Original: Then, according to Lemma D.3, we know that λ > 0 and hence λ ∈

Original: [Equation]

Original: Now we apply diagonalization to A 12 B k ⋆ :

Original: [Equation]

Original: As a result,

Original: [Equation]

Original: [Equation]

Original: where the last equality uses the fact A − 12 V is a orthogonal matrix with spectral norm 1 .

Original: Since any eigenvector with eigenvalue λ of A − 12 corresponds to the eigenvector with eigenvalue(1 /λ 2 ) of A = V V T , and V V T as a symmetric real matrix only has real eigenvalues,

Original: [Equation]

Original: [Equation]

Original: Plug it into Equation (11), we get

Original: [Equation]

Original: Notice that 1 − √ 1 − x ≤ x for x = ||

Original: I − A || 2 k⋆∈ [0 , 1] , we conclude the proof.

Original: E Addition Exp ResultsE.1 Visualization

Original: We show the representation visualization on 16 neurons for SOC and LOT in Figure 3.

Original: E.2 Residual Connection

Original: [-We show the loss landscape for both LOT and SOC in Figure 
-]
Revised: [-We show the loss landscape for both LOT and SOC in Figure 
-]

Original: E.3 Circular vs. Zero padding

Original: As we discussed in Section 4.1, the default parametrization leads to the convolution result withcircular-padding, while we will pre-process the input to get the final result with zero-padding.

Original: In

Original: Table 4, we show the performance comparison between circular and zero padding.

Original: We can see thatthe performance significantly improves when we do the pre-processing and use the zero-paddinginstead of default circular-padding.

Original: [Figure]

Original: E.4 Error Control of Newton’s Iterations

Original: As we discuss in Section D, we use a finite number of Newton’s iteration to approximate the inversesquare root of a matrix Z k ∗

Original: ≈ ( V V ⊺ ) − 12 .

Original: Theoretically, we have shown that the error of approximatedorthogonal matrix Z k ∗ V will decay exponentially with number of iterations.

Original: Furthermore, we observein practice that during the Newton’s iteration, the maximum singular value of Z k V will alwaysapproach 1 from the left side.

Original: As an example, we show the maximum singular value ( σ max ) for all

Original: LOT layers in LipConvnet-20 at different steps ( k ) during the Newton’s Iteration in Figure 5.

Original: Wecan see that σ max < 1 for all the layers, so we can safely assume that the Lipschitz bound is nolarger than 1.

Original: In addition, σ max > 1 − 10 − 4 = 0 .

Original: 9999 after k = 8 iterations for all the layers, whichindicates that the Newton’s iteration converges well.

Original: E.5 Supervised Learning without CReg Loss and HH Activation

Original: We show the results of semi-supervised learning under standard setting (without CReg Loss, HH

Original: Activation, and LLN on CIFAR-100) in Table 5 and Table 6.

Original: We can observe that we still achievea good performance compared with SOC, and the gap is sometimes larger than with the differentoptimization techniques.

Original: E.6 Full semi-supervised

Original: We show the results for all architectures with semi-supervised learning in Table 7 (with CReg and

Original: HH) and 8 (without CReg and HH).

Original: We can see observe that we still achieve a better performance

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: compared with SOC.

Original: In addition, these results also have a larger certified accuracy at larger radiuscompared with the supervised learning setting.

Original: E.7 Robustness against ℓ ∞ Empirical Attack

Original: To evaluate the robustness of our model against attacks in other ℓ p norms, we evaluate the modelagainst standard ℓ ∞ PGD attack[16] with ϵ = 8 ./ 255 .

Original: and 50 steps.

Original: We show the results on CIFAR10 with CReg loss and HH activation in Table 9.

Original: We can observe that we still achieve better empiricalrobustness on the LipConvNet compared with SOC.

Original: F Property of Output from FFT

Original: In our algorithm, we will first perform FFT to the input and calculate the matrix inverse square rooton the frequency domain.

Original: We are going to show that, although such operation is done on the complexnumbers, the results will still be real after the inverse FFT.

Original: Theorem F.1.

Original: Say J ∈ C m × m is unitary so that J ∗ J

Original: = I , and V =

Original: J ˜

Original: V J ∗ for V ∈ R m × m and˜ V ∈ C m × m .

Original: Let F ( V )

Original: =

Original: ( V V ∗ ) − 12 V be our transformation.

Original: Then F ( V ) =

Original: JF ( ˜ V ) J ∗ .

Original: Proof.

Original: First, notice that V T

Original: = V ∗ =

Original: J ˜ V ∗ J ∗ .

Original: Second, we have:

Original: [Equation]

Original: ∗

Original: [Equation]

Original: ∗ J ∗ )

Original: [Equation]

Original: J )

Original: [Equation]

Original: so that ( ˜ V ˜ V ∗ ) − 12 =

Original: J ∗ ( J ˜ V

Original: ˜

Original: V ∗ J ∗ ) − 12 J .

Original: Thus, we have

Original: [Equation]

Original: [Table]

Original: G Results on TinyImageNet

Original: To further validate the performance of LOT, we evaluate it on the TinyImageNet dataset whichconsists of 100,000 64 × 64 images in 200 classes.

Original: We use the same model architecture and trainingsetting as before and show the comparison between LOT and SOC in Table 10.

Original: We can observe that

Original: LOT still outperforms the existing SOC approach in most cases.

Original: Meanwhile, we observe that all1-Lipschitz models have a performance drop on the larger dataset compared with vanilla models, andwe leave the breakthrough as future work.

Original: H Discussion of Model Architectures

Original: We would like to emphasize that there are two major differences in designing 1-Lipschitz networks andstandard networks.

Original: First, standard architectures focus a lot on regularizing the network so that it doesnot overfit (e.g. with dropout), while in 1-Lipschitz networks we do not need strong regularizationbecause the 1-Lipschitz constraint is already strong enough.

Original: Second, standard architectures arecarefully designed so that the gradient will propagate in a proper manner (e.g. residual connection,

Original: BatchNorm), while 1-Lipschitz networks have an intrinsic gradient-norm-preserving property (see[14]).

Original: Thus, it may not be appropriate to directly utilize modern architectures in the design of1-Lipschitz networks.

Original: In Table 11, we show the results of a 1-Lipschitz ResNet18 model, where we(1) replace all the conv layers with 1-Lipschitz convolution, (2) replace all the residual connectionwith average and (3) enforce the variance parameter of BatchNorm layers to be 1 so that the model is1-Lipschitz.

Original: We can observe that LOT still outperforms SOC on both CIFAR-10 and TinyImageNet,while the performance of the ResNet18 architecture is not as good as that of LipConvNet.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]