Original: LOT: Layer-wise Orthogonal Training on Improving ℓ 2 Certified Robustness
Revised: LOT: Layer-wise Orthogonal Training on Improving ℓ 2 Certified Robustness

Original: Abstract
Revised: Abstract

Original: Recent studies show that training deep neural networks (DNNs) with Lipschitz constraints are able to enhance adversarial robustness and other model properties such as stability.
Revised: Recent studies show that training deep neural networks (DNNs) with Lipschitz constraints are able to enhance adversarial robustness and other model properties such as stability.

Original: In this paper, we propose a layer-wise orthogonal training method (LOT) to effectively train 1-Lipschitz convolution layers via parametrizing an orthogonal matrix with an unconstrained matrix.
Revised: In this paper, we propose a layer-wise orthogonal training method (LOT) to effectively train 1-Lipschitz convolution layers via parametrizing an orthogonal matrix with an unconstrained matrix.

Original: We then efficiently compute the inverse square root of a convolution kernel by transforming the input domain to the
Revised: We then efficiently compute the inverse square root of a convolution kernel by transforming the input domain to the

Original: Fourier frequency domain.
Revised: Fourier frequency domain.

Original: On the other hand, as existing works show that semisupervised training helps improve empirical robustness, we aim to bridge the gap and prove that semi-supervised learning also improves the certified robustness of
Revised: On the other hand, as existing works show that semisupervised training helps improve empirical robustness, we aim to bridge the gap and prove that semi-supervised learning also improves the certified robustness of

Original: Lipschitz-bounded models.
Revised: Lipschitz-bounded models.

Original: We conduct comprehensive evaluations for LOT under different settings.
Revised: We conduct comprehensive evaluations for LOT under different settings.

Original: We show that LOT significantly outperforms baselines regarding deterministic ℓ 2 certified robustness, and scales to deeper neural networks.
Revised: We show that LOT significantly outperforms baselines regarding deterministic ℓ 2 certified robustness, and scales to deeper neural networks.

Original: Under the supervised scenario, we improve the state-of-the-art certified robustness for all architectures (e.g. from 59.04% to 63.50% on CIFAR-10 and from 32.57% to 34.59% on CIFAR-100 at radius ρ = 36 / 255 for 40-layer networks).
Revised: Under the supervised scenario, we improve the state-of-the-art certified robustness for all architectures (e.g. from 59.04% to 63.50% on CIFAR-10 and from 32.57% to 34.59% on CIFAR-100 at radius ρ = 36 / 255 for 40-layer networks).

Original: With semisupervised learning over unlabelled data, we are able to improve state-of-the-art certified robustness on CIFAR-10 at ρ = 108 / 255 from 36.04% to 42.39%.
Revised: With semisupervised learning over unlabelled data, we are able to improve state-of-the-art certified robustness on CIFAR-10 at ρ = 108 / 255 from 36.04% to 42.39%.

Original: In addition, LOT consistently outperforms baselines on different model architectures with only 1/3 evaluation time.
Revised: In addition, LOT consistently outperforms baselines on different model architectures with only 1/3 evaluation time.

Original:  Introduction
Revised:  Introduction

Original: Given the wide applications of deep neural networks (DNNs), ensuring their robustness against potential adversarial attacks [7, 21, 31, 32] is of great importance.
Revised: Given the wide applications of deep neural networks (DNNs), ensuring their robustness against potential adversarial attacks [7, 21, 31, 32] is of great importance.

Original: There has been a line of research providing defense approaches to improve the empirical robustness of DNNs [18, 34, 30, 29] and certification methods to certify DNN robustness [14, 35, 15, 13, 33].
Revised: There has been a line of research providing defense approaches to improve the empirical robustness of DNNs [18, 34, 30, 29] and certification methods to certify DNN robustness [14, 35, 15, 13, 33].

Original: Existing certification techniques can be categorized as deterministic and probabilistic certifications [14], and in this work we will focus on improving the deterministic ℓ 2 certified robustness by training 1-Lipschitz DNNs.
Revised: Existing certification techniques can be categorized as deterministic and probabilistic certifications [14], and in this work we will focus on improving the deterministic ℓ 2 certified robustness by training 1-Lipschitz DNNs.

Original: Although different approaches have been proposed to empirically enforce the Lipschitz constant of the trained model [26], it is still challenging to strictly ensure the 1-Lipschitz, which can lead to tight robustness certification.
Revised: Although different approaches have been proposed to empirically enforce the Lipschitz constant of the trained model [26], it is still challenging to strictly ensure the 1-Lipschitz, which can lead to tight robustness certification.

Original: One recent work SOC [23] proposes to parametrize the orthogonal weight matrices with the exponential of skew-symmetric matrices (i.e. W = exp( V − V ⊺ ) ).
Revised: One recent work SOC [23] proposes to parametrize the orthogonal weight matrices with the exponential of skew-symmetric matrices (i.e. W = exp( V − V ⊺ ) ).

Original: However, such parametrization will be biased when the matrix norm is constrained and the expressiveness is limited, especially when they rescale V to be small to help with convergence.
Revised: However, such parametrization will be biased when the matrix norm is constrained and the expressiveness is limited, especially when they rescale V to be small to help with convergence.

Original: In this work, we propose a layer-wise orthogonal training approach (LOT) by parameterizing the orthogonal weight matrix with an unconstrained matrix W =
Revised: In this work, we propose a layer-wise orthogonal training approach (LOT) by parameterizing the orthogonal weight matrix with an unconstrained matrix W =

Original: ( V V ⊺ ) − 12 V .
Revised: ( V V ⊺ ) − 12 V .

Original: In order to calculate the inverse square root for convolution kernels, we will perform Fourier Transformation and calculate the inverse square root of matrices on the frequency domain using Newton’s iteration.
Revised: In order to calculate the inverse square root for convolution kernels, we will perform Fourier Transformation and calculate the inverse square root of matrices on the frequency domain using Newton’s iteration.

Original: In our parametrization, the output is agnostic to the input norm (i.e. scaling V = αV does not change the value of W ).
Revised: In our parametrization, the output is agnostic to the input norm (i.e. scaling V = αV does not change the value of W ).

Original: We show that such parametrization achieves higher model expressiveness and robustness (Section 6.2), and provides more meaningful representation vectors (Section 6.3).
Revised: We show that such parametrization achieves higher model expressiveness and robustness (Section 6.2), and provides more meaningful representation vectors (Section 6.3).

Original: In addition, several works have shown that semi-supervised learning will help improve the empirical robustness of models under adversarial training and other settings [4].
Revised: In addition, several works have shown that semi-supervised learning will help improve the empirical robustness of models under adversarial training and other settings [4].

Original: In this work, we take the first attempt to bridge semi-supervised training with certified robustness based on our 1-Lipschitz
Revised: In this work, we take the first attempt to bridge semi-supervised training with certified robustness based on our 1-Lipschitz

Original: DNNs.
Revised: DNNs.

Original: Theoretically, we show that semi-supervised learning can help improve the error bound of
Revised: Theoretically, we show that semi-supervised learning can help improve the error bound of

Original: Lipschitz-bounded models.
Revised: Lipschitz-bounded models.

Original: We also lower bound the certified radius as a function of the model performance and Lipschitz property.
Revised: We also lower bound the certified radius as a function of the model performance and Lipschitz property.

Original: Empirically, we indeed observe that including un-labelled data will help with the certified robustness of 1-Lipschitz models, especially at a larger radius (e.g. from 36.04% to 42.39% at ρ = 108 / 255 on CIFAR-10).
Revised: Empirically, we indeed observe that including un-labelled data will help with the certified robustness of 1-Lipschitz models, especially at a larger radius (e.g. from 36.04% to 42.39% at ρ = 108 / 255 on CIFAR-10).

Original: We conduct comprehensive experiments to evaluate our approach, and we show that LOT significantly outperforms the state-of-the-art in terms of the deterministic ℓ 2 certified robustness.
Revised: We conduct comprehensive experiments to evaluate our approach, and we show that LOT significantly outperforms the state-of-the-art in terms of the deterministic ℓ 2 certified robustness.

Original: We also conduct different ablation studies to show that (1) LOT can produce more meaningful features for visualization; (2) residual connections help to smoothify the training of LOT model.
Revised: We also conduct different ablation studies to show that (1) LOT can produce more meaningful features for visualization; (2) residual connections help to smoothify the training of LOT model.

Original: Technical contributions .
Revised: Technical contributions .

Original: In this work, we aim to train a certifiably robust 1-Lipschitz model and also analyze the certified radius of the Lipschitz bounded model under semi-supervised learning.
Revised: In this work, we aim to train a certifiably robust 1-Lipschitz model and also analyze the certified radius of the Lipschitz bounded model under semi-supervised learning.

Original: • We propose a layer-wise orthogonal training method LOT for convolution layers to train 1-Lipschitz models based on Newton’s iteration, and thus compute the deterministic certified robustness for the model.
Revised: • We propose a layer-wise orthogonal training method LOT for convolution layers to train 1-Lipschitz models based on Newton’s iteration, and thus compute the deterministic certified robustness for the model.

Original: We prove the convergence of Newton’s iteration used in our algorithm.
Revised: We prove the convergence of Newton’s iteration used in our algorithm.

Original: • We derive the certified robustness of lipschitz constrained model under semi-supervised setting, and formally show how semi-supervised learning affects the certified radius.
Revised: • We derive the certified robustness of lipschitz constrained model under semi-supervised setting, and formally show how semi-supervised learning affects the certified radius.

Original: • We evaluate our LOT method under different settings (i
Revised: • We evaluate our LOT method under different settings (i

Original: e. supervised and semi-supervised) on different models and datasets.
Revised: e. supervised and semi-supervised) on different models and datasets.

Original: With supervised learning, we show that it significantly outperforms state-of-the-art baselines, and on some deep architectures the performance gain is over 4%.
Revised: With supervised learning, we show that it significantly outperforms state-of-the-art baselines, and on some deep architectures the performance gain is over 4%.

Original: With semi-supervised learning, we further improve certified robust
Revised: With semi-supervised learning, we further improve certified robust

Original: accuracy by over 6% at a large radius.
Revised: accuracy by over 6% at a large radius.

Original:  Related Work
Revised:  Related Work

Original: Certified Robustness for Lipschitz Constrained Models Several studies have been conducted to explore the Lipschitz-constrained models for certified robustness.
Revised: Certified Robustness for Lipschitz Constrained Models Several studies have been conducted to explore the Lipschitz-constrained models for certified robustness.

Original: [26] first certifies model robustness based on its Lipschitz constant and propose training algorithms to regularize the model Lipschitz.
Revised: [26] first certifies model robustness based on its Lipschitz constant and propose training algorithms to regularize the model Lipschitz.

Original: Multiple works [5, 19, 20, 8] have been proposed to achieve 1-Lipschitz during training for linear networks by regularizing or normalizing the spectral norm of the weight matrix.
Revised: Multiple works [5, 19, 20, 8] have been proposed to achieve 1-Lipschitz during training for linear networks by regularizing or normalizing the spectral norm of the weight matrix.

Original: However, when applying these approaches on convolution layers, the spectral norm is bounded by unrolling the convolution into linear operations, which leads to a loose Lipschitz bound [27].
Revised: However, when applying these approaches on convolution layers, the spectral norm is bounded by unrolling the convolution into linear operations, which leads to a loose Lipschitz bound [27].

Original: Recently, [2] shows that the 1-Lipschitz requirement is not enough for a good robust model; rather, the gradient-normpreserving property is important.
Revised: Recently, [2] shows that the 1-Lipschitz requirement is not enough for a good robust model; rather, the gradient-normpreserving property is important.

Original: Besides these training-time techniques, different approaches have been proposed to calculate a tight Lipschitz bound during evaluation.
Revised: Besides these training-time techniques, different approaches have been proposed to calculate a tight Lipschitz bound during evaluation.

Original: [6] upper bounds the Lipschitz with semi-definite programming while [11] upper bounds the Lipschitz with polynomial optimization.
Revised: [6] upper bounds the Lipschitz with semi-definite programming while [11] upper bounds the Lipschitz with polynomial optimization.

Original: In this work we aim to effectively train 1-Lipschitz convolution models.
Revised: In this work we aim to effectively train 1-Lipschitz convolution models.

Original: Orthogonal Convolution Neural Networks [16] first proposes to directly construct orthogonal convolution operations.
Revised: Orthogonal Convolution Neural Networks [16] first proposes to directly construct orthogonal convolution operations.

Original: Such operations are not only 1-Lipschitz, but also gradient norm preserving, which provides a higher model capability and a smoother training process [2].
Revised: Such operations are not only 1-Lipschitz, but also gradient norm preserving, which provides a higher model capability and a smoother training process [2].

Original: BCOP [16] trains orthogonal convolution by iteratively generating 2 × 2 orthogonal kernels from orthogonal matrices.
Revised: BCOP [16] trains orthogonal convolution by iteratively generating 2 × 2 orthogonal kernels from orthogonal matrices.

Original: [25] proposes to parametrize an orthogonal convolution with Cayley transformation W
Revised: [25] proposes to parametrize an orthogonal convolution with Cayley transformation W

Original: =
Revised: =

Original: ( I − V +
Revised: ( I − V +

Original: V ⊺ )
Revised: V ⊺ )

Original: ( I + V − V ⊺ ) − 1 where the convolution inverse is calculated on the Fourier frequency domain.
Revised: ( I + V − V ⊺ ) − 1 where the convolution inverse is calculated on the Fourier frequency domain.

Original: ECO [36] proposes to explicitly normalize all the singular values [22] of convolution operations to be 1.
Revised: ECO [36] proposes to explicitly normalize all the singular values [22] of convolution operations to be 1.

Original: So far, the best-performing orthogonal convolution approach is SOC [23], where they parametrize the orthogonal convolution with W = exp
Revised: So far, the best-performing orthogonal convolution approach is SOC [23], where they parametrize the orthogonal convolution with W = exp

Original: ( V − V ⊺ ) where the exponential and transpose are defined with the convolution operation.
Revised: ( V − V ⊺ ) where the exponential and transpose are defined with the convolution operation.

Original: However, one major weakness of SOC is that it will rescale V to be small so that the approximation of exp can converge soon, which will impose a bias on the resulting output space.
Revised: However, one major weakness of SOC is that it will rescale V to be small so that the approximation of exp can converge soon, which will impose a bias on the resulting output space.

Original: For example, when V is very small W will be close to I .
Revised: For example, when V is very small W will be close to I .

Original: Such norm-dependent property is not desired, and thus we propose a parametrization that is invariant to rescaling.
Revised: Such norm-dependent property is not desired, and thus we propose a parametrization that is invariant to rescaling.

Original: Finally, [24] proposes several techniques for orthogonal CNNs, including a generalized Householder (HH) activation function, a certificate regularizer (CReg) loss, and a last layer normalization (LLN).
Revised: Finally, [24] proposes several techniques for orthogonal CNNs, including a generalized Householder (HH) activation function, a certificate regularizer (CReg) loss, and a last layer normalization (LLN).

Original: They can be integrated with our training approach to improve model robustness.
Revised: They can be integrated with our training approach to improve model robustness.

Original:  Problem Setup 3.1 Lipschitz Constant of Neural Networks and Certified Robustness
Revised:  Problem Setup 3.1 Lipschitz Constant of Neural Networks and Certified Robustness

Original: Let f : R d → R C denote a neural network for classification, where d is the input dimension and
Revised: Let f : R d → R C denote a neural network for classification, where d is the input dimension and

Original: C is the number of output classes.
Revised: C is the number of output classes.

Original: The model prediction is given by arg max c f ( x ) c , where f ( x ) c represents the prediction probability for class c .
Revised: The model prediction is given by arg max c f ( x ) c , where f ( x ) c represents the prediction probability for class c .

Original: The Lipschitz constant of the model under p -norm is defined as: Lip p ( f ) = sup
Revised: The Lipschitz constant of the model under p -norm is defined as: Lip p ( f ) = sup

Original: || f
Revised: || f

Original: ( x 1 ) − f ( x 2 )
Revised: ( x 1 ) − f ( x 2 )

Original: ||
Revised: ||

Original: p || x
Revised: p || x

Original: 1 − x 2 || p ∀ x 1 , x 2 ∈ R d .
Revised: 1 − x 2 || p ∀ x 1 , x 2 ∈ R d .

Original: Unless specified, we will focus on ℓ 2 -norm and use Lip ( f ) to denote Lip 2 ( f ) in this work.
Revised: Unless specified, we will focus on ℓ 2 -norm and use Lip ( f ) to denote Lip 2 ( f ) in this work.

Original: We can observe that the definition of model
Revised: We can observe that the definition of model

Original: Lipschitz aligns with its robustness property - both require the model not to change much w.r.t.
Revised: Lipschitz aligns with its robustness property - both require the model not to change much w.r.t.

Original: input changes.
Revised: input changes.

Original: Formally speaking, define M f ( x ) = max i f ( x ) i − max j ̸ =argmax i f ( x ) i f ( x ) j to be the prediction gap of f on the input x , then we can guarantee that f ( x ) will not change its prediction within | x
Revised: Formally speaking, define M f ( x ) = max i f ( x ) i − max j ̸ =argmax i f ( x ) i f ( x ) j to be the prediction gap of f on the input x , then we can guarantee that f ( x ) will not change its prediction within | x

Original: ′ − x | < r , where
Revised: ′ − x | < r , where

Original: [Equation]
Revised: [Equation]

Original: Therefore, people have proposed to utilize the model Lipschitz to provide certified robustness and investigated training algorithms to train a model with a small Lipschitz constant.
Revised: Therefore, people have proposed to utilize the model Lipschitz to provide certified robustness and investigated training algorithms to train a model with a small Lipschitz constant.

Original: Note that the Lipschitz constant of a composed function f = f 1 ◦ f 2 satisfies Lip ( f ) ≤ Lip ( f 1 ) ×
Revised: Note that the Lipschitz constant of a composed function f = f 1 ◦ f 2 satisfies Lip ( f ) ≤ Lip ( f 1 ) ×

Original: Lip ( f 2 ) .
Revised: Lip ( f 2 ) .

Original: Since a neural network is usually composed of several layers, we can investigate the
Revised: Since a neural network is usually composed of several layers, we can investigate the

Original: Lipschitz of each layer to calculate the final upper bound.
Revised: Lipschitz of each layer to calculate the final upper bound.

Original: If we can restrict each layer to be 1-Lipschitz, then the overall model will be 1-Lipschitz with an arbitrary number of layers.
Revised: If we can restrict each layer to be 1-Lipschitz, then the overall model will be 1-Lipschitz with an arbitrary number of layers.

Original: 3.2 Orthogonal Linear and Convolution Operations
Revised: 3.2 Orthogonal Linear and Convolution Operations

Original: Consider a linear operation with equal input and output dimensions y
Revised: Consider a linear operation with equal input and output dimensions y

Original: =
Revised: =

Original: Wx , where x, y ∈ R n and W ∈ R n × n .
Revised: Wx , where x, y ∈ R n and W ∈ R n × n .

Original: We say W is an orthogonal matrix if WW ⊺ =
Revised: We say W is an orthogonal matrix if WW ⊺ =

Original: W ⊺ W
Revised: W ⊺ W

Original: =
Revised: =

Original: I and call y
Revised: I and call y

Original: =
Revised: =

Original: Wx an orthogonal linear operation.
Revised: Wx an orthogonal linear operation.

Original: The orthogonal operation is not only 1-Lipschitz, but also normpreserving, i.e., ||
Revised: The orthogonal operation is not only 1-Lipschitz, but also normpreserving, i.e., ||

Original: Wx || 

Revised: Wx ||

Original: = ||
Revised: = ||

Original: x || 2 for all x ∈ R n .
Revised: x || 2 for all x ∈ R n .

Original: If the input and output dimensions do not match, i.e., W ∈ R m × n where n ̸ = m , we say W is semi-orthogonal if either WW ⊺ =
Revised: If the input and output dimensions do not match, i.e., W ∈ R m × n where n ̸ = m , we say W is semi-orthogonal if either WW ⊺ =

Original: I or W ⊺ W = I .
Revised: I or W ⊺ W = I .

Original: The semi-orthogonal operation is 1-Lipschitz and non-expansive
Revised: The semi-orthogonal operation is 1-Lipschitz and non-expansive

Original: , i.e., || Wx || 2 ≤ || x
Revised: , i.e., || Wx || 2 ≤ || x

Original: || 2 .
Revised: || 2 .

Original: The orthogonal convolution operation is defined in a similar way.
Revised: The orthogonal convolution operation is defined in a similar way.

Original: Let y
Revised: Let y

Original: =
Revised: =

Original: W ◦
Revised: W ◦

Original: X be an orthogonal operation, where x, y ∈ R c × w × w and W ∈ R c × c × k × k .
Revised: X be an orthogonal operation, where x, y ∈ R c × w × w and W ∈ R c × c × k × k .

Original: We say W is an orthogonal convolution kernel if W ◦ W ⊺ =
Revised: We say W is an orthogonal convolution kernel if W ◦ W ⊺ =

Original: W ⊺ ◦ W =
Revised: W ⊺ ◦ W =

Original: I conv where the transpose here refers to the convolution transpose and I conv denotes the identity convolution kernel.
Revised: I conv where the transpose here refers to the convolution transpose and I conv denotes the identity convolution kernel.

Original: Such orthogonal convolution is 1-Lipschitz and norm-preserving.
Revised: Such orthogonal convolution is 1-Lipschitz and norm-preserving.

Original: When the input and output channel numbers are different, the semi-orthogonal convolution kernel W satisfies either W ◦ W ⊺ =
Revised: When the input and output channel numbers are different, the semi-orthogonal convolution kernel W satisfies either W ◦ W ⊺ =

Original: I conv or W ⊺ ◦ W = I conv
Revised: I conv or W ⊺ ◦ W = I conv

Original: and it is 1-Lipschitz and non-expansive.
Revised: and it is 1-Lipschitz and non-expansive.

Original:  LOT: Layer-wise Orthogonal Training
Revised:  LOT: Layer-wise Orthogonal Training

Original: In this section, we propose our LOT framework to achieve certified robustness.
Revised: In this section, we propose our LOT framework to achieve certified robustness.

Original: 1 We will first introduce how LOT layer works to achieve 1-Lipschitz.
Revised: 1 We will first introduce how LOT layer works to achieve 1-Lipschitz.

Original: The key idea of our method is to parametrize an orthogonal convolution W with an un-constrained convolution kernel W =
Revised: The key idea of our method is to parametrize an orthogonal convolution W with an un-constrained convolution kernel W =

Original: ( V V ⊺ ) − 12 V .
Revised: ( V V ⊺ ) − 12 V .

Original: Next, we will propose several techniques to improve the training and evaluation processes of our model.
Revised: Next, we will propose several techniques to improve the training and evaluation processes of our model.

Original: Finally, we discuss how semi-supervised learning can help with the certified robustness of our model.
Revised: Finally, we discuss how semi-supervised learning can help with the certified robustness of our model.

Original: 4.1 1-Lipschitz Neural Networks via LOT
Revised: 4.1 1-Lipschitz Neural Networks via LOT

Original: Our key observation is that we can parametrize an orthogonal matrix W ∈ R n × n with an unconstrained matrix V ∈ R n × n by W =
Revised: Our key observation is that we can parametrize an orthogonal matrix W ∈ R n × n with an unconstrained matrix V ∈ R n × n by W =

Original: ( V V ⊺ ) − 12 V [9].
Revised: ( V V ⊺ ) − 12 V [9].

Original: In addition, this equation also holds true in the case of
Revised: In addition, this equation also holds true in the case of

Original: convolution kernel - given any convolution kernel V ∈ R c
Revised: convolution kernel - given any convolution kernel V ∈ R c

Original: ×
Revised: ×

Original: c
Revised: c

Original: × k × k , where c denotes the channel number and k denotes the kernel size, we can get an orthogonal convolution kernel by
Revised: × k × k , where c denotes the channel number and k denotes the kernel size, we can get an orthogonal convolution kernel by

Original: W =
Revised: W =

Original: ( V ◦ V ⊺ ) − 12 ◦ V , where transpose and inverse square root are with respect to the convolution operations.
Revised: ( V ◦ V ⊺ ) − 12 ◦ V , where transpose and inverse square root are with respect to the convolution operations.

Original: The orthogonality of W can be verified by W ◦ W ⊺ = 
Revised: The orthogonality of W can be verified by W ◦ W ⊺ = W ◦ W ⊺ =

Original: W ⊺ ◦ W = I conv , where I is the identity convolution kernel.
Revised: I conv , where I is the identity convolution kernel.

Original: This way, we can parametrize an orthogonal convolution layer by training over the un-constrained parameter V .
Revised: This way, we can parametrize an orthogonal convolution layer by training over the un-constrained parameter V .

Original: Formally speaking, we will parametrize an orthogonal convolution layer by Y =
Revised: Formally speaking, we will parametrize an orthogonal convolution layer by Y =

Original: ( V ◦ V ⊺ ) − 12 ◦ V ◦
Revised: ( V ◦ V ⊺ ) − 12 ◦ V ◦

Original: X , where X, Y ∈ R c × w × w and w ≥ k is the input shape.
Revised: X , where X, Y ∈ R c × w × w and w ≥ k is the input shape.

Original: The key obstacle here is how to calculate the inverse square root of a convolution kernel.
Revised: The key obstacle here is how to calculate the inverse square root of a convolution kernel.

Original: Inspired by [25], we can leverage the convolution theorem which states that the convolution in the input domain equals the element-wise multiplication in the
Revised: Inspired by [25], we can leverage the convolution theorem which states that the convolution in the input domain equals the element-wise multiplication in the

Original: Fourier frequency domain.
Revised: Fourier frequency domain.

Original: In the case of multi-channel convolution, the convolution corresponds to the matrix multiplication on each pixel location.
Revised: In the case of multi-channel convolution, the convolution corresponds to the matrix multiplication on each pixel location.

Original: That is, let FFT : R w × w → C w × w be the 2D
Revised: That is, let FFT : R w × w → C w × w be the 2D

Original: Discrete Fourier Transformation and FFT − 1 be the inverse Fourier Transformation.
Revised: Discrete Fourier Transformation and FFT − 1 be the inverse Fourier Transformation.

Original: We will zero-pad the input to w × w if the original shape is smaller than w .
Revised: We will zero-pad the input to w × w if the original shape is smaller than w .

Original: Let ˜ X
Revised: Let ˜ X

Original: i = FFT ( X i ) and ˜ V j,i = FFT ( V j,i ) denote the input and kernel on frequency domain, then we have:
Revised: i = FFT ( X i ) and ˜ V j,i = FFT ( V j,i ) denote the input and kernel on frequency domain, then we have:

Original: [Equation]
Revised: [Equation]

Original: in which multiplication, transpose and inverse square root operations are matrix-wise.
Revised: in which multiplication, transpose and inverse square root operations are matrix-wise.

Original: Therefore, we can first calculate FFT ( Y ) on the frequency domain and perform the inverse Fourier transformation to get the final output.
Revised: Therefore, we can first calculate FFT ( Y ) on the frequency domain and perform the inverse Fourier transformation to get the final output.

Original: We will use Newton’s iteration to calculate the inverse square root of the positive semi-definite matrix
Revised: We will use Newton’s iteration to calculate the inverse square root of the positive semi-definite matrix

Original: A = ˜ V : , : ,a,b ˜ V ∗ : , : ,a,b in a differentiable way [17].
Revised: A = ˜ V : , : ,a,b ˜ V ∗ : , : ,a,b in a differentiable way [17].

Original: If we initialize Y 

Revised: If we initialize Y

Original: = A and Z 

Revised: = A and Z

Original: =
Revised: =

Original: I and perform the following update rule:
Revised: I and perform the following update rule:

Original: [Equation]
Revised: [Equation]

Original: Z k will converge to A − 12 when || I − A
Revised: Z k will converge to A − 12 when || I − A

Original: || 2 < 1 .
Revised: || 2 < 1 .

Original: The condition can be satisfied by rescaling the parameter V = αV , noticing that the scaling factor will not change the resulting W .
Revised: The condition can be satisfied by rescaling the parameter V = αV , noticing that the scaling factor will not change the resulting W .

Original: In practice, we execute a finite number of Newton’s iteration steps and we provide a rigorous error bound for this finite iteration scheme in Appendix D to show that the error will decay exponentially.
Revised: In practice, we execute a finite number of Newton’s iteration steps and we provide a rigorous error bound for this finite iteration scheme in Appendix D to show that the error will decay exponentially.

Original: In addition, we show that although the operation is over complex number domain after the FFT, the resulting parameters will still be real domain, as shown below.
Revised: In addition, we show that although the operation is over complex number domain after the FFT, the resulting parameters will still be real domain, as shown below.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Say J ∈ C m × m is unitary
Revised: Say J ∈ C m × m is unitary

Original: so that J ∗ J
Revised: so that J ∗ J

Original: =
Revised: =

Original: I , and V =
Revised: I , and V =

Original: J ˜
Revised: J ˜

Original: V J ∗ for V ∈ R m × m and ˜ V ∈ C m × m .
Revised: V J ∗ for V ∈ R m × m and ˜ V ∈ C m × m .

Original: Let F ( V )
Revised: Let F ( V )

Original: =
Revised: =

Original: ( V V ∗ ) − 12 V be our transformation.
Revised: ( V V ∗ ) − 12 V be our transformation.

Original: Then F ( V ) =
Revised: Then F ( V ) =

Original: JF ( ˜ V ) J ∗ .
Revised: JF ( ˜ V ) J ∗ .

Original: Proof.
Revised: Proof.

Original: First, notice that V T
Revised: First, notice that V T

Original: = V ∗ =
Revised: = V ∗ =

Original: J ˜ V ∗ J ∗ .
Revised: J ˜ V ∗ J ∗ .

Original: Second, we have:
Revised: Second, we have:

Original: [Equation]
Revised: [Equation]

Original: so that ( ˜ V ˜ V ∗ ) − 12 =
Revised: so that ( ˜ V ˜ V ∗ ) − 12 =

Original: J ∗ ( J ˜ V
Revised: J ∗ ( J ˜ V

Original: ˜
Revised: ˜

Original: V ∗ J ∗ ) − 12 J .
Revised: V ∗ J ∗ ) − 12 J .

Original: Thus, we have
Revised: Thus, we have

Original: [Equation]
Revised: [Equation]

Original: Remark.
Revised: Remark.

Original: From this theorem it is clear that our returned value JF ( ˜ V )
Revised: From this theorem it is clear that our returned value JF ( ˜ V )

Original: J ∗ equals to the transformed version of the original matrix F ( V ) ∈
Revised: J ∗ equals to the transformed version of the original matrix F ( V ) ∈

Original: R m × m , and thus is guaranteed to be in the real domain.
Revised: R m × m , and thus is guaranteed to be in the real domain.

Original: Circular Padding vs. Zero Padding When we apply the convolution theorem to calculate the convolution Y
Revised: Circular Padding vs. Zero Padding When we apply the convolution theorem to calculate the convolution Y

Original: =
Revised: =

Original: W ◦ X with Fourier Transformation, the result implicitly uses circular padding.
Revised: W ◦ X with Fourier Transformation, the result implicitly uses circular padding.

Original: However, in neural networks, zero padding is usually a better choice.
Revised: However, in neural networks, zero padding is usually a better choice.

Original: Therefore, we will first perform zero padding on both sides of input X pad =
Revised: Therefore, we will first perform zero padding on both sides of input X pad =

Original: zero_pad ( X ) and calculate the resulting
Revised: zero_pad ( X ) and calculate the resulting

Original: Y pad =
Revised: Y pad =

Original: W ◦ X pad with Fourier Transformation.
Revised: W ◦ X pad with Fourier Transformation.

Original: Thus, the implicit circular padding in this process will actually pad the zeros which we padded beforehand.
Revised: Thus, the implicit circular padding in this process will actually pad the zeros which we padded beforehand.

Original: Finally, we truncate the padded part and get our desired output Y = truncate ( Y pad ) .
Revised: Finally, we truncate the padded part and get our desired output Y = truncate ( Y pad ) .

Original: We empirically observe that this technique helps improve the model robustness as shown in Appendix E.3.
Revised: We empirically observe that this technique helps improve the model robustness as shown in Appendix E.3.

Original: When Input and Output Dimensions Differ In previous discussion, we assume that the convolution kernel V has equal input and output channels.
Revised: When Input and Output Dimensions Differ In previous discussion, we assume that the convolution kernel V has equal input and output channels.

Original: In the case W ∈ R c out × c in × k × k where c out ̸ = c in , we aim to get a semi-orthogonal convolution kernel W (i.e. W ◦ W ⊺ =
Revised: In the case W ∈ R c out × c in × k × k where c out ̸ = c in , we aim to get a semi-orthogonal convolution kernel W (i.e. W ◦ W ⊺ =

Original: I conv if c out < c in or
Revised: I conv if c out < c in or

Original: W ⊺ ◦ W =
Revised: W ⊺ ◦ W =

Original: I conv if c out > c in ).
Revised: I conv if c out > c in ).

Original: As pointed out in [9], calculating W with Newton’s iteration will naturally lead to a semi-orthogonal convolution kernel when the input and output dimensions differ.
Revised: As pointed out in [9], calculating W with Newton’s iteration will naturally lead to a semi-orthogonal convolution kernel when the input and output dimensions differ.

Original: Emulating a Larger Stride To emulate the case when the convolution stride is 2, we follow previous works [23] and use the invertible downsampling layer [10], in which the input dimension c in will be increased by × 4 times.
Revised: Emulating a Larger Stride To emulate the case when the convolution stride is 2, we follow previous works [23] and use the invertible downsampling layer [10], in which the input dimension c in will be increased by × 4 times.

Original: Strides larger than 2 can be emulated with similar techniques if needed.
Revised: Strides larger than 2 can be emulated with similar techniques if needed.

Original: Overall Algorithm Taking the techniques we discussed before, we can get our final LOT convolution layer.
Revised: Overall Algorithm Taking the techniques we discussed before, we can get our final LOT convolution layer.

Original: The detailed algorithm is shown in Appendix B. First, we will pad the input to prevent the implicit circular padding mechanism and pad the kernel so that they are in the same shape.
Revised: The detailed algorithm is shown in Appendix B. First, we will pad the input to prevent the implicit circular padding mechanism and pad the kernel so that they are in the same shape.

Original: Next, we perform the Fourier transformation and calculate the output on the frequency domain with Newton’s iteration.
Revised: Next, we perform the Fourier transformation and calculate the output on the frequency domain with Newton’s iteration.

Original: Finally, we perform inverse Fourier transformation and return the desired output.
Revised: Finally, we perform inverse Fourier transformation and return the desired output.

Original: Comparison with SOC Several works have been proposed on orthogonal convolution layers with re-parametrization[25, 23], among which the SOC approach via W = exp
Revised: Comparison with SOC Several works have been proposed on orthogonal convolution layers with re-parametrization[25, 23], among which the SOC approach via W = exp

Original: ( V − V ⊺ ) has achieved the best performance.
Revised: ( V − V ⊺ ) has achieved the best performance.

Original: Compared with SOC, LOT has the following advantages.
Revised: Compared with SOC, LOT has the following advantages.

Original: First, the parametrization in LOT is norm-independent.
Revised: First, the parametrization in LOT is norm-independent.

Original: Rescaling V to αV will not change the resulting W .
Revised: Rescaling V to αV will not change the resulting W .

Original: By comparison,
Revised: By comparison,

Original: V with a smaller norm in SOC will lead to a W closer to the identity transformation.
Revised: V with a smaller norm in SOC will lead to a W closer to the identity transformation.

Original: Considering that the norm of V will be regularized during training (e.g. SOC will re-scale V to have a small norm; people usually initialize weight to be small and impose l2-regularization during training), the orthogonal weight space in SOC may be biased.
Revised: Considering that the norm of V will be regularized during training (e.g. SOC will re-scale V to have a small norm; people usually initialize weight to be small and impose l2-regularization during training), the orthogonal weight space in SOC may be biased.

Original: Second, we can see that LOT is able to model any orthogonal kernel W by noticing that ( WW ⊺ ) − 12 W is W itself; by comparison, SOC cannot parametrize all the orthogonal operations.
Revised: Second, we can see that LOT is able to model any orthogonal kernel W by noticing that ( WW ⊺ ) − 12 W is W itself; by comparison, SOC cannot parametrize all the orthogonal operations.

Original: For example, in the case of orthogonal matrices, the exponential of a skew-symmetric matrix only models the special orthogonal group (i.e. the matrices with +1 determinant).
Revised: For example, in the case of orthogonal matrices, the exponential of a skew-symmetric matrix only models the special orthogonal group (i.e. the matrices with +1 determinant).

Original: Third, we directly handle the case when c in ̸ = c out , while SOC needs to do extra padding so that the channel numbers match.
Revised: Third, we directly handle the case when c in ̸ = c out , while SOC needs to do extra padding so that the channel numbers match.

Original: Finally, LOT is more efficient during evaluation, when we only need to perform the Fourier and inverse Fourier transformation as extra overhead, while SOC needs multiple convolution operations to calculate the exponential.
Revised: Finally, LOT is more efficient during evaluation, when we only need to perform the Fourier and inverse Fourier transformation as extra overhead, while SOC needs multiple convolution operations to calculate the exponential.

Original: We will further show quantitative comparisons with SOC in Section 6.2.
Revised: We will further show quantitative comparisons with SOC in Section 6.2.

Original: The major limitation of LOT is its large overhead during training, since we need to calculate Newton’s iteration in each training step which takes more time and memory.
Revised: The major limitation of LOT is its large overhead during training, since we need to calculate Newton’s iteration in each training step which takes more time and memory.

Original: In addition, we sometimes observe that Newton’s iteration is not stable when we perform many steps with 32-bit precision.
Revised: In addition, we sometimes observe that Newton’s iteration is not stable when we perform many steps with 32-bit precision.

Original: To overcome this problem, we will pre-calculate Newton’s iteration with 64-bit precision during evaluation, as we will introduce in Section 4.2.
Revised: To overcome this problem, we will pre-calculate Newton’s iteration with 64-bit precision during evaluation, as we will introduce in Section 4.2.

Original: 4.2 Training and Evaluation of LOT
Revised: 4.2 Training and Evaluation of LOT

Original: Smoothing the Training Stage In practice, we observe that the LOT layers are highly non-smooth with respect to the parameter V (see Section 6.3).
Revised: Smoothing the Training Stage In practice, we observe that the LOT layers are highly non-smooth with respect to the parameter V (see Section 6.3).

Original: Therefore, the model is difficult to converge during the training process especially when the model is deep.
Revised: Therefore, the model is difficult to converge during the training process especially when the model is deep.

Original: To smooth the training, we propose two techniques.
Revised: To smooth the training, we propose two techniques.

Original: First, we will initialize all except bottleneck layers where c in = c out with identity parameter V
Revised: First, we will initialize all except bottleneck layers where c in = c out with identity parameter V

Original: =
Revised: =

Original: I .
Revised: I .

Original: The bottleneck layers where c in = c out will still be randomly initialized.
Revised: The bottleneck layers where c in = c out will still be randomly initialized.

Original: Second, as pointed out in [12], residual connection helps with model smoothness.
Revised: Second, as pointed out in [12], residual connection helps with model smoothness.

Original: Therefore, for the intermediate layers, we will add the 1-Lipschitz residual connection y
Revised: Therefore, for the intermediate layers, we will add the 1-Lipschitz residual connection y

Original: = λx + (1 − λ )
Revised: = λx + (1 − λ )

Original: f ( x ) .
Revised: f ( x ) .

Original: Some work suggests that λ here can be trainable [25], while we observe that setting λ
Revised: Some work suggests that λ here can be trainable [25], while we observe that setting λ

Original: = 0 .
Revised: = 0 .

Original: 5 is enough.
Revised: 5 is enough.

Original: Speeding up the Evaluation Stage Notice that after the model is trained, the orthogonal kernel ˜ W will no longer change.
Revised: Speeding up the Evaluation Stage Notice that after the model is trained, the orthogonal kernel ˜ W will no longer change.

Original: Therefore, we can pre-calculate Newton’s iteration and use the result for each evaluation step.
Revised: Therefore, we can pre-calculate Newton’s iteration and use the result for each evaluation step.

Original: Thus, the only runtime overhead compared with a standard convolution layer during evaluation is the Fourier transformation part.
Revised: Thus, the only runtime overhead compared with a standard convolution layer during evaluation is the Fourier transformation part.

Original: In addition, we observe that using 64-bit precision instead of the commonly used 32-bit precision in Newton’s iteration will help with the numerical stability.
Revised: In addition, we observe that using 64-bit precision instead of the commonly used 32-bit precision in Newton’s iteration will help with the numerical stability.

Original: Therefore, when we pre-calculate ˜ W , we will first transform ˜ V into float64.
Revised: Therefore, when we pre-calculate ˜ W , we will first transform ˜ V into float64.

Original: After Newton’s iterations, we transform the resulting ˜ W back to float32 type for efficiency.
Revised: After Newton’s iterations, we transform the resulting ˜ W back to float32 type for efficiency.

Original: 4.3 Semi-supervised Learning
Revised: 4.3 Semi-supervised Learning

Original: Existing works have shown that semi-supervised learning helps improve empirical robustness [4], while the impact of semi-supervised learning on certified robustness has not been explored yet.
Revised: Existing works have shown that semi-supervised learning helps improve empirical robustness [4], while the impact of semi-supervised learning on certified robustness has not been explored yet.

Original: We aim to bridge this gap and investigate whether applying semi-supervised learning also benefits the certified robustness of Lipschitz constrained models.
Revised: We aim to bridge this gap and investigate whether applying semi-supervised learning also benefits the certified robustness of Lipschitz constrained models.

Original: As we will discuss in Section 5.1, we theoretically show that the certified robustness of a Lipschitz-bounded model can be improved by semi-supervised learning.
Revised: As we will discuss in Section 5.1, we theoretically show that the certified robustness of a Lipschitz-bounded model can be improved by semi-supervised learning.

Original: In practice, suppose we have a (small) labelled dataset D s = { ( x i , y i ) } and a (large) unlabelled dataset D u = { x ′ i } , we will first train a labeller model G pl over the labelled dataset D s .
Revised: In practice, suppose we have a (small) labelled dataset D s = { ( x i , y i ) } and a (large) unlabelled dataset D u = { x ′ i } , we will first train a labeller model G pl over the labelled dataset D s .

Original: Then we use G pl to assign pseudo-labels to the unlabelled dataset to get D ′ u
Revised: Then we use G pl to assign pseudo-labels to the unlabelled dataset to get D ′ u

Original: = { ( x ′ i , G pl ( x ′ i )) } .
Revised: = { ( x ′ i , G pl ( x ′ i )) } .

Original: Finally, we use the overall dataset D s + D ′ u to train the student model G as the final model.
Revised: Finally, we use the overall dataset D s + D ′ u to train the student model G as the final model.

Original: We assume that both G pl and G will use the LOT model architecture.
Revised: We assume that both G pl and G will use the LOT model architecture.

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: In this section, we will introduce our theoretical analysis to show that the certified robustness of Lipschitz-constrained models will benefit from semi-supervised learning.
Revised: In this section, we will introduce our theoretical analysis to show that the certified robustness of Lipschitz-constrained models will benefit from semi-supervised learning.

Original: We make similar assumptions with [28] which provides error bound of discrete models in semi-supervised learning, while we will analyze a (Lipschitz-bounded) continuous model and its certified accuracy.
Revised: We make similar assumptions with [28] which provides error bound of discrete models in semi-supervised learning, while we will analyze a (Lipschitz-bounded) continuous model and its certified accuracy.

Original: We first introduce data expansion proposed in [28], which we will adopt in our analysis.
Revised: We first introduce data expansion proposed in [28], which we will adopt in our analysis.

Original: Let T denote some set of transformations and define B ≜ { x ′ |∃ T ∈ T such that || x ′ − T ( x ) ||
Revised: Let T denote some set of transformations and define B ≜ { x ′ |∃ T ∈ T such that || x ′ − T ( x ) ||

Original: <
Revised: <

Original: r } to be the set of points within distance r from any of the transformation of x .
Revised: r } to be the set of points within distance r from any of the transformation of x .

Original: The neighbourhood of x is defined by N = { x ′ |B
Revised: The neighbourhood of x is defined by N = { x ′ |B

Original: ( x ) ∩ B ( x ′ ) ̸ = ∅} .
Revised: ( x ) ∩ B ( x ′ ) ̸ = ∅} .

Original: Let P denote the data distribution and P i denote the distribution of data in the i -th class.
Revised: Let P denote the data distribution and P i denote the distribution of data in the i -th class.

Original: The expansion property is defined as:
Revised: The expansion property is defined as:

Original: Definition 5.1 ( ( a, c ) -expansion, as in [28]) .
Revised: Definition 5.1 ( ( a, c ) -expansion, as in [28]) .

Original: We say that the class-conditional distribution P i satisfies ( a, c ) -expansion if for all V ⊆ X with P
Revised: We say that the class-conditional distribution P i satisfies ( a, c ) -expansion if for all V ⊆ X with P

Original: i ( V ) <
Revised: i ( V ) <

Original: a , the following holds:
Revised: a , the following holds:

Original: [Equation]
Revised: [Equation]

Original: if P i satisfies ( a, c ) -expansion for all classes in P , then we say P satisfies ( a, c ) -expansion.
Revised: if P i satisfies ( a, c ) -expansion for all classes in P , then we say P satisfies ( a, c ) -expansion.

Original: Next, we define the classifier and the loss function in the continuous case.
Revised: Next, we define the classifier and the loss function in the continuous case.

Original: For simplicity, we assume a binary classification task here.
Revised: For simplicity, we assume a binary classification task here.

Original: Definition 5.2 (Continuous model G and loss functions) .
Revised: Definition 5.2 (Continuous model G and loss functions) .

Original: We assume a binary classification task, where G ∗ ( x ) , G pl ( x ) ∈ { 0 , 1 } are discrete labels, and the trained model is G ( x ) ∈ [0 , 1] .
Revised: We assume a binary classification task, where G ∗ ( x ) , G pl ( x ) ∈ { 0 , 1 } are discrete labels, and the trained model is G ( x ) ∈ [0 , 1] .

Original: We define the loss function L m ( G, G ∗ ) ∈
Revised: We define the loss function L m ( G, G ∗ ) ∈

Original: [0 , 1] to be:
Revised: [0 , 1] to be:

Original: [Equation]
Revised: [Equation]

Original: and the error of G is defined as Err m ( G ) =
Revised: and the error of G is defined as Err m ( G ) =

Original: L m ( G, G ∗ ) .
Revised: L m ( G, G ∗ ) .

Original: Remark.
Revised: Remark.

Original: Here we provide a general setting here, and if we assume G ( x ) ∈ { 0 , 1 } , L m ( G, G ∗ ) becomes the 0-1 loss in [28].
Revised: Here we provide a general setting here, and if we assume G ( x ) ∈ { 0 , 1 } , L m ( G, G ∗ ) becomes the 0-1 loss in [28].

Original: Then we define the marginal consistency set S m B ( G ; δ ) and loss R m B ( G ; δ ) in the continuous case.
Revised: Then we define the marginal consistency set S m B ( G ; δ ) and loss R m B ( G ; δ ) in the continuous case.

Original: The consistency set includes cases in which the model prediction will not change a lot within the neighbourhood, and the consistency loss measures the probability that the model is not consistent over the population.
Revised: The consistency set includes cases in which the model prediction will not change a lot within the neighbourhood, and the consistency loss measures the probability that the model is not consistent over the population.

Original: Definition 5.3 (Marginal consistency set and consistency loss) .
Revised: Definition 5.3 (Marginal consistency set and consistency loss) .

Original: We define the set S m B ( G ; δ ) in which the ratio of marginal prediction change on each input x is no larger than δ within its neighbourhood:
Revised: We define the set S m B ( G ; δ ) in which the ratio of marginal prediction change on each input x is no larger than δ within its neighbourhood:

Original: [Equation]
Revised: [Equation]

Original: and R m B ( G ; δ ) denotes the marginal consistency loss of the probability that x is not in S m B ( G ; δ ) :
Revised: and R m B ( G ; δ ) denotes the marginal consistency loss of the probability that x is not in S m B ( G ; δ ) :

Original: [Equation]
Revised: [Equation]

Original: Remark.
Revised: Remark.

Original: (1) We may use S m B ( G ) and R m B ( G ) for abbreviation when the choice of δ does not lead to confusion; (2) when δ = 0 , S m B ( G ) and R m B ( G ) requires that the prediction is exactly the same within the neighbourhood, which are reduced to S B ( G ) and R B ( G ) defined in [28].
Revised: (1) We may use S m B ( G ) and R m B ( G ) for abbreviation when the choice of δ does not lead to confusion; (2) when δ = 0 , S m B ( G ) and R m B ( G ) requires that the prediction is exactly the same within the neighbourhood, which are reduced to S B ( G ) and R B ( G ) defined in [28].

Original: Finally, we define the certified robustness radius CertR ( G ) as the average radius over the population in which the model keeps its correct prediction:
Revised: Finally, we define the certified robustness radius CertR ( G ) as the average radius over the population in which the model keeps its correct prediction:

Original: [Equation]
Revised: [Equation]

Original: Note that when G ( x ) ̸ =
Revised: Note that when G ( x ) ̸ =

Original: G ∗ ( x ) , the certified radius r will be 0.
Revised: G ∗ ( x ) , the certified radius r will be 0.

Original: 5.1 Error Bound and Certified Radius of Lipschitz-bounded Model
Revised: 5.1 Error Bound and Certified Radius of Lipschitz-bounded Model

Original: Let M ( G pl ) =
Revised: Let M ( G pl ) =

Original: { x : G pl ( x ) ̸ =
Revised: { x : G pl ( x ) ̸ =

Original: G ∗ ( x ) } denote the set in which the pseudolabel is wrong.
Revised: G ∗ ( x ) } denote the set in which the pseudolabel is wrong.

Original: We will have similar separation and expansion assumptions on the data distribution as in [28].
Revised: We will have similar separation and expansion assumptions on the data distribution as in [28].

Original: Assumption 5.1 (Separation) .
Revised: Assumption 5.1 (Separation) .

Original: We assume that P is separable with probability 1 − µ by ground-truth classifier G ∗ , i.e., R m B ( G ∗ ; 0) ≤ µ .
Revised: We assume that P is separable with probability 1 − µ by ground-truth classifier G ∗ , i.e., R m B ( G ∗ ; 0) ≤ µ .

Original: Assumption 5.2 (Expansion) .
Revised: Assumption 5.2 (Expansion) .

Original: Define a ≜ max i { P i ( M ( G pl )) } to be the maximum fraction of incorrectly pseudolabeled examples in any class.
Revised: Define a ≜ max i { P i ( M ( G pl )) } to be the maximum fraction of incorrectly pseudolabeled examples in any class.

Original: We assume that a < 1 / 3 and P satisfies ( a, c ) - expansion for c > 3 .
Revised: We assume that a < 1 / 3 and P satisfies ( a, c ) - expansion for c > 3 .

Original: We express our bounds in terms of c ≜ min(1 /a, c ) .
Revised: We express our bounds in terms of c ≜ min(1 /a, c ) .

Original: With these assumptions and let δ ∈ [0 , 1 c ] , we can compare the performance of G and G pl (i.e. the performance with and without semi-supervised learning) with the following theorem:
Revised: With these assumptions and let δ ∈ [0 , 1 c ] , we can compare the performance of G and G pl (i.e. the performance with and without semi-supervised learning) with the following theorem:

Original: Theorem 5.1 (Error bound with semi-supervised learning) .
Revised: Theorem 5.1 (Error bound with semi-supervised learning) .

Original: Suppose Assumption 5.1 and 5.2 holds and let ˆ G be a minimizer of the following loss function:
Revised: Suppose Assumption 5.1 and 5.2 holds and let ˆ G be a minimizer of the following loss function:

Original: [Equation]
Revised: [Equation]

Original: Then we can upper bound the error of ˆ G by:
Revised: Then we can upper bound the error of ˆ G by:

Original: [Equation]
Revised: [Equation]

Original: In addition, we can bound the certified robustness radius w.r.t.
Revised: In addition, we can bound the certified robustness radius w.r.t.

Original: the Lipschitz constant as below:
Revised: the Lipschitz constant as below:

Original: Theorem 5.2 (Certified radius with semi-supervised learning) .
Revised: Theorem 5.2 (Certified radius with semi-supervised learning) .

Original: Suppose Assumption 5.2 holds.
Revised: Suppose Assumption 5.2 holds.

Original: Then the certified radius of a model G can be bounded by:
Revised: Then the certified radius of a model G can be bounded by:

Original: [Equation]
Revised: [Equation]

Original: The proofs of Theorem 5.1 and 5.2 are shown in Appendix C.
Revised: The proofs of Theorem 5.1 and 5.2 are shown in Appendix C.

Original: Discussion From Theorem 5.1, we observe that the benefits of semi-supervised learning depend on the error of G pl and data property.
Revised: Discussion From Theorem 5.1, we observe that the benefits of semi-supervised learning depend on the error of G pl and data property.

Original: If the data expands and separates well and G pl is accurate, c is large and µ is small, which means Err m ( ˆ G ) will be smaller than Err ( G pl ) .
Revised: If the data expands and separates well and G pl is accurate, c is large and µ is small, which means Err m ( ˆ G ) will be smaller than Err ( G pl ) .

Original: Note that µ is usually small and we assume c > 3 , so
Revised: Note that µ is usually small and we assume c > 3 , so

Original: Err m ( ˆ G ) <
Revised: Err m ( ˆ G ) <

Original: Err ( G pl ) will hold true in most cases, indicating that semi-supervised learning helps improve the model performance.
Revised: Err ( G pl ) will hold true in most cases, indicating that semi-supervised learning helps improve the model performance.

Original: From Theorem 5.2, we can observe that given the data distribution and G pl , the certified radius of the student model CertR ( G ) can be lower-bounded by 1) how well G learns from G pl
Revised: From Theorem 5.2, we can observe that given the data distribution and G pl , the certified radius of the student model CertR ( G ) can be lower-bounded by 1) how well G learns from G pl

Original: ( L m ( G, G pl ) ); 2) how consistent G is ( R m B ( G ; δ ) ).
Revised: ( L m ( G, G pl ) ); 2) how consistent G is ( R m B ( G ; δ ) ).

Original: These two factors also correspond to the losses in the actual training process - the cross-entropy loss measures how well G learns from G pl and the Lipschitz constraint measures local consistency for a
Revised: These two factors also correspond to the losses in the actual training process - the cross-entropy loss measures how well G learns from G pl and the Lipschitz constraint measures local consistency for a

Original: Lipschitz-bounded model 2 .
Revised: Lipschitz-bounded model 2 .

Original:  Evaluation
Revised:  Evaluation

Original: In this section, we will evaluate our method LOT on different datasets and models compared with the state-of-the-art baselines based on the deterministic ℓ 2 certified robustness.
Revised: In this section, we will evaluate our method LOT on different datasets and models compared with the state-of-the-art baselines based on the deterministic ℓ 2 certified robustness.

Original: We show that on both
Revised: We show that on both

Original: CIFAR-10 and CIFAR-100, LOT achieves significantly higher certified accuracy under different radii and settings with more efficient evaluation time.
Revised: CIFAR-10 and CIFAR-100, LOT achieves significantly higher certified accuracy under different radii and settings with more efficient evaluation time.

Original: In the meantime, we conduct a series of ablation studies to analyze the representation power of our LOT, the error control of Newton’s iterations, and the effects of residual connections.
Revised: In the meantime, we conduct a series of ablation studies to analyze the representation power of our LOT, the error control of Newton’s iterations, and the effects of residual connections.

Original: We show that under the semi-supervised setting, the certified accuracy of LOT is further improved and outperforms the baselines.
Revised: We show that under the semi-supervised setting, the certified accuracy of LOT is further improved and outperforms the baselines.

Original: [Table]
Revised: [Table]

Original: 6.1 Experiment Setup
Revised: 6.1 Experiment Setup

Original: Baseline - SOC with CReg loss, HH activation and LLN technique
Revised: Baseline - SOC with CReg loss, HH activation and LLN technique

Original: In the evaluation, we will mainly compare with SOC [23]
Revised: In the evaluation, we will mainly compare with SOC [23]

Original: [Table]
Revised: [Table]

Original: as SOC has been shown to outperform other 1-Lipschitz models in terms of certified robustness.
Revised: as SOC has been shown to outperform other 1-Lipschitz models in terms of certified robustness.

Original: SOC parameterizes an orthogonal layer with W = exp( V − V ⊺ ) where
Revised: SOC parameterizes an orthogonal layer with W = exp( V − V ⊺ ) where

Original: V is the convolution kernel and the transpose and exponential are with respect to the convolution operation.
Revised: V is the convolution kernel and the transpose and exponential are with respect to the convolution operation.

Original: Following their setting
Revised: Following their setting

Original: [Table]
Revised: [Table]

Original: we will focus on the CIFAR-10 and CIFAR-100 datasets and the model architecture varies from LipConvnet-5 to LipConvnet-40.
Revised: we will focus on the CIFAR-10 and CIFAR-100 datasets and the model architecture varies from LipConvnet-5 to LipConvnet-40.

Original: [24] proposes three techniques to improve SOC networks .
Revised: [24] proposes three techniques to improve SOC networks .

Original: First, they observe that adding a Certificate
Revised: First, they observe that adding a Certificate

Original: Regularization (CReg) loss can help with the certified performance at larger radius: ℓ CReg ( x, y )
Revised: Regularization (CReg) loss can help with the certified performance at larger radius: ℓ CReg ( x, y )

Original: = − γ ReLU (cid:0) (
Revised: = − γ ReLU (cid:0) (

Original: f ( x ) y − max i ̸
Revised: f ( x ) y − max i ̸

Original: = y f ( x ) i ) / √ 2 (cid:1) .
Revised: = y f ( x ) i ) / √ 2 (cid:1) .

Original: Second, they propose a Householder (HH) activation layer, which is a generalization of the standard GroupSort activation.
Revised: Second, they propose a Householder (HH) activation layer, which is a generalization of the standard GroupSort activation.

Original: Finally, for tasks with large number of classes (e.g. CIFAR-100), they propose Last Layer Normalization(LLN) to only normalize the final output layer instead of orthogonalize.
Revised: Finally, for tasks with large number of classes (e.g. CIFAR-100), they propose Last Layer Normalization(LLN) to only normalize the final output layer instead of orthogonalize.

Original: Since these techniques are agnostic to the type of convolution operation, we will further integrate them with LOT for comparison.
Revised: Since these techniques are agnostic to the type of convolution operation, we will further integrate them with LOT for comparison.

Original: Implementation Details We will train and evaluate the LOT network under supervised and unsupervised scenarios.
Revised: Implementation Details We will train and evaluate the LOT network under supervised and unsupervised scenarios.

Original: Following previous works [23], we focus on the CIFAR-10 and CIFARdatasets and the model architecture varies from LipConvnet-5 to LipConvnet-40.
Revised: Following previous works [23], we focus on the CIFAR-10 and CIFARdatasets and the model architecture varies from LipConvnet-5 to LipConvnet-40.

Original: In semi-supervised learning, we use the 500K data introduced in [4] as the unlabelled dataset.
Revised: In semi-supervised learning, we use the 500K data introduced in [4] as the unlabelled dataset.

Original: To train the LOT network, we will train the model for 200 epochs using a momentum SGD optimizer with an initial learning rate 0.1 and decay by 0.1 at the 100-th and 150-th epochs.
Revised: To train the LOT network, we will train the model for 200 epochs using a momentum SGD optimizer with an initial learning rate 0.1 and decay by 0.1 at the 100-th and 150-th epochs.

Original: We use Newton’s iteration with 10 steps which we observe is enough for convergence (see Appendix E.4).
Revised: We use Newton’s iteration with 10 steps which we observe is enough for convergence (see Appendix E.4).

Original: When CReg loss is applied, we use γ = 0 .
Revised: When CReg loss is applied, we use γ = 0 .

Original: 5 ; when HH activation is applied, we use the version of order 1.
Revised: 5 ; when HH activation is applied, we use the version of order 1.

Original: We add the residual connection with a fixed λ = 0 .
Revised: We add the residual connection with a fixed λ = 0 .

Original: 5 for LOT; for SOC, we use their original version, as we observe that residual connections even hurt their performance (see discussions in Section 6.3).
Revised: 5 for LOT; for SOC, we use their original version, as we observe that residual connections even hurt their performance (see discussions in Section 6.3).

Original: We show the certified accuracy at radius ρ ∈ { 36255 , 72255 , 108255 } .
Revised: We show the certified accuracy at radius ρ ∈ { 36255 , 72255 , 108255 } .

Original: For the evaluation time comparison, we show the runtime taken to do a full pass on the testing set evaluated on an NVIDIA RTX A6000 GPU.
Revised: For the evaluation time comparison, we show the runtime taken to do a full pass on the testing set evaluated on an NVIDIA RTX A6000 GPU.

Original: 6.2 Supervised Scenario
Revised: 6.2 Supervised Scenario

Original: We show the results of supervised learning on CIFAR-10 and CIFAR-100 in Table 1.
Revised: We show the results of supervised learning on CIFAR-10 and CIFAR-100 in Table 1.

Original: We can observe that LOT significantly outperforms baselines on both vanilla accuracy and certified accuracy for different datasets.
Revised: We can observe that LOT significantly outperforms baselines on both vanilla accuracy and certified accuracy for different datasets.

Original: In particular, we observe that the improvement is more significant for deeper models.
Revised: In particular, we observe that the improvement is more significant for deeper models.

Original: We owe it to the reason that our LOT layer has a better expressiveness and unbiased parametrization.
Revised: We owe it to the reason that our LOT layer has a better expressiveness and unbiased parametrization.

Original: “Mean evaluation time” column in Table 1 records the evaluation time per instance averaged between CIFAR-10 and CIFAR-100 models.
Revised: “Mean evaluation time” column in Table 1 records the evaluation time per instance averaged between CIFAR-10 and CIFAR-100 models.

Original: Since architectures of CIFAR-10 and CIFAR100 models differ only in the last linear year, the evaluation time is almost the same.
Revised: Since architectures of CIFAR-10 and CIFAR100 models differ only in the last linear year, the evaluation time is almost the same.

Original: We observe that
Revised: We observe that

Original: LOT has a better efficiency during the evaluation stage, since LOT pre-calculates Newton’s iteration while SOC calculates the exponential in each iteration.
Revised: LOT has a better efficiency during the evaluation stage, since LOT pre-calculates Newton’s iteration while SOC calculates the exponential in each iteration.

Original: A similar conclusion can be observed from results without CReg loss, HH activation and LLN, as shown in Table 6 and 7 in Appendix E.5.
Revised: A similar conclusion can be observed from results without CReg loss, HH activation and LLN, as shown in Table 6 and 7 in Appendix E.5.

Original: [Table]
Revised: [Table]

Original: 6.3 Ablation Studies
Revised: 6.3 Ablation Studies

Original: In this section, we will perform several ablation studies for LOT.
Revised: In this section, we will perform several ablation studies for LOT.

Original: Unless specified, we will use the deep model LipConvnet-20 for evaluation.
Revised: Unless specified, we will use the deep model LipConvnet-20 for evaluation.

Original: We use the model without CReg loss and HH activation so that we can see the comparison under the standard CNN setting.
Revised: We use the model without CReg loss and HH activation so that we can see the comparison under the standard CNN setting.

Original: Representation Analysis Recent works suggest that adversarially robust models will have a good regularization-free feature visualization [1].
Revised: Representation analysis Recent works suggest that adversarially robust models will have a good regularization-free feature visualization [1].

Original: We show the visualization of 4 neurons in the last hidden layer of LipConvNet-20 for both SOC and LOT in Figure 1.
Revised: We show the visualization of 4 neurons in the last hidden layer of LipConvNet-20 for both SOC and LOT in Figure 1.

Original: More figures are shown in Appendix E.
Revised: More figures are shown in Appendix E.

Original: The visualization process is as follows: given a chosen neuron, we will start from a random image and take 500 gradient steps to maximize the neuron value with step size 1.0 and decay factor 0.1.
Revised: The visualization process is as follows: given a chosen neuron, we will start from a random image and take 500 gradient steps to maximize the neuron value with step size 1.0 and decay factor 0.1.

Original: We can see that LOT indeed generates more meaningful features.
Revised: We can see that LOT indeed generates more meaningful features.

Original: We attribute it to the good expressiveness power of our model.
Revised: We attribute it to the good expressiveness power of our model.

Original: This indicates that our approach indeed leads to better representations.
Revised: This indicates that our approach indeed leads to better representations.

Original: Error Control of Newton’s Iterations
Revised: Error Control of Newton’s Iterations

Original: To see how our Newton’s iteration approximates the inverse square root, we visualize the maximum singular value ( σ max ) of LOT layers in Appendix E.4.
Revised: To see how our Newton’s iteration approximates the inverse square root, we visualize the maximum singular value ( σ max ) of LOT layers in Appendix E.4.

Original: We observe that all the resulting layers have a strictly < 1 Lipschitz.
Revised: We observe that all the resulting layers have a strictly < 1 Lipschitz.

Original: Therefore, we can safely conclude that the overall LOT network is 1-Lipschitz.
Revised: Therefore, we can safely conclude that the overall LOT network is 1-Lipschitz.

Original: [Figure]
Revised: [Figure]

Original: Effect of Residual Connections Empirically, we observe that LOT is non-smooth and therefore we need the residual connection to smoothify the model.
Revised: Effect of residual connections Empirically, we observe that LOT is non-smooth and therefore we need the residual connection to smoothify the model.

Original: To verify this phenomenon, we adopt the visualization approach in [12] and show the loss surface in Figure 2.
Revised: To verify this phenomenon, we adopt the visualization approach in [12] and show the loss surface in Figure 2.

Original: We can see that LOT without residual connection is highly non-smooth, while after adding residual connections the model becomes much smoother.
Revised: We can see that LOT without residual connection is highly non-smooth, while after adding residual connections the model becomes much smoother.

Original: By comparison, we observe that SOC networks with and without residual connection are similar (see Appendix E.2).
Revised: By comparison, we observe that SOC networks with and without residual connection are similar (see Appendix E.2).

Original: We hypothesize the reason to be that the exponential parametrization W = exp( A ) enables residual property implicitly, i.e.,
Revised: We hypothesize the reason to be that the exponential parametrization W = exp( A ) enables residual property implicitly, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where the first two terms are essentially the format of residual connection.
Revised: where the first two terms are essentially the format of residual connection.

Original: We show the results of models with and without residual connections in Table 2.
Revised: We show the results of models with and without residual connections in Table 2.

Original: We can observe that our LOT indeed improves with the residual connection, while the performance remains similar or even worse for SOC.
Revised: We can observe that our LOT indeed improves with the residual connection, while the performance remains similar or even worse for SOC.

Original: Therefore, in the evaluation, we use the LOT with residual connection and SOC without it.
Revised: Therefore, in the evaluation, we use the LOT with residual connection and SOC without it.

Original: Robustness against ℓ ∞ Empirical Attack To evaluate the robustness of our model against attacks in other ℓ p norms, we evaluate the model against standard ℓ ∞ PGD attack[18] with ϵ = 8 / 255 and 50 steps.
Revised: Robustness against ℓ ∞ Empirical Attack To evaluate the robustness of our model against attacks in other ℓ p norms, we evaluate the model against standard ℓ ∞ PGD attack[18] with ϵ = 8 / 255 and 50 steps.

Original: We show the results on CIFAR-10 with CReg loss and HH activation in Table 3.
Revised: We show the results on CIFAR-10 with CReg loss and HH activation in Table 3.

Original: We can observe that we still achieve better empirical robustness on the LipConvNet compared with SOC.
Revised: We can observe that we still achieve better empirical robustness on the LipConvNet compared with SOC.

Original: [Table]
Revised: [Table]

Original: Results of different architectures are shown in Table 8 in Appendix E
Revised: Results of different architectures are shown in Table 8 in Appendix E

Original: [Table]
Revised: [Table]

Original: Model
Revised: Model

Original: [Table]
Revised: [Table]

Original: 6.4 Semi-supervised Learning Scenario
Revised: 6.4 Semi-supervised Learning Scenario

Original: We show the results of semi-supervised learning for one architecture with CReg loss and HH activation in Table 4.
Revised: We show the results of semi-supervised learning for one architecture with CReg loss and HH activation in Table 4.

Original: The full results are shown in Table 8 and 9 in Appendix E.6.
Revised: The full results are shown in Table 8 and 9 in Appendix E.6.

Original: We can observe that, with semi-supervised learning, the vanilla accuracy will drop slightly, which will impact the certified accuracy at a small radius ( ρ = 36 / 255 ).
Revised: We can observe that, with semi-supervised learning, the vanilla accuracy will drop slightly, which will impact the certified accuracy at a small radius ( ρ = 36 / 255 ).

Original: However, the certified accuracy at a large radius will be improved, and the gap is more significant at a larger radius (e.g. ρ = 108 / 255 ).
Revised: However, the certified accuracy at a large radius will be improved, and the gap is more significant at a larger radius (e.g. ρ = 108 / 255 ).

Original: We improve the previous state-of-the-art certified accuracy by over 6% at ρ = 108 / 255 .
Revised: We improve the previous state-of-the-art certified accuracy by over 6% at ρ = 108 / 255 .

Original: This indeed shows that semi-supervised learning can help improve certified robustness.
Revised: This indeed shows that semi-supervised learning can help improve certified robustness.

Original: We empirically observe that semi-supervised learning does not help much on CIFAR-100.
Revised: We empirically observe that semi-supervised learning does not help much on CIFAR-100.

Original: We owe it to the reason that the vanilla accuracy of the teacher model on CIFAR-100 is low (less than 50%).
Revised: We owe it to the reason that the vanilla accuracy of the teacher model on CIFAR-100 is low (less than 50%).

Original: Thus, according to
Revised: Thus, according to

Original: Assumption 5.2 and Theorem 5.2, when c is low, the certified accuracy would be low.
Revised: Assumption 5.2 and Theorem 5.2, when c is low, the certified accuracy would be low.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we propose an orthogonal convolution layer LOT and build a 1-Lipschitz convolution network.
Revised: In this work, we propose an orthogonal convolution layer LOT and build a 1-Lipschitz convolution network.

Original: We show that LOT network outperforms the previous state-of-the-art in certified robustness.
Revised: We show that LOT network outperforms the previous state-of-the-art in certified robustness.

Original: We also show that semi-supervised learning can further help with the robustness both theoretically and experimentally.
Revised: We also show that semi-supervised learning can further help with the robustness both theoretically and experimentally.

Original: Acknowledgements
Revised: Acknowledgements

Original: This work is partially supported by the NSF grant No.1910100, NSF CNS No.2046726, C3 AI, and the Alfred P. Sloan Foundation.A Broader Impact
Revised: This work is partially supported by the NSF grant No.1910100, NSF CNS No.2046726, C3 AI, and the Alfred P. Sloan Foundation.

Original: Adversarial attacks have been investigated a lot and people are worried that the vulnerability of machine learning models may affect their application in real world.

Original: This paper, as a potential countermeasure against the adversarial attacks, proposes a type of model architecture that can guarantee the model robustness under certain adversarial attacks.

Original: This could help to secure the safe deployment of ML models in real applications, thus promoting the development of various ML systems.

Original: In addition, better understanding of the Lipschitz property of machine learning models may help people understand and explain how the models work, which is a key concern when applying these algorithms in practice.

Original: On the other hand, the misuse of our technique may also lead to negative impact.

Original: For example, if an attacker is fully familiar with this work, he/she may discover some specific attack to fool this model (beyond ℓ 2 attacks which we can certify against).

Original: Therefore, in real applications, it is recommended that different defense techniques are applied together to secure the model safety.

Original: B Pseudocode of LOT Layer

Original: We show the detailed pseudocode of our LOT layer in Algorithm 1.

Original: Algorithm 1 LOT layer.

Original: Require:

Original: Unconstrained convolution kernel V ∈ R c out × c in × k × k ; Input tensor X ∈ R c in × w × w .

Original: 1: X pad =

Original: zero_pad ( X, ( k, k, k, k ))

Original: ∈ R c in × ( w +2 k ) × ( w +2 k ) .

Original: 2: V pad =

Original: zero_pad ( W, (0 , 0 , k + w, k + w ))

Original: ∈

Original: R c out × c in × ( w +2 k ) × ( w +2 k ) .

Original: 3: // Calculate the Fourier transformation: 4: for all i ∈ { 1 , . .

Original: , c in } do 5:

Original: ˜ X i = FFT ( X padi ) .

Original: 6: for all j ∈ { 1 , . . .

Original: , c out } do 7:

Original: ˜ V j,i =

Original: FFT ( V padj,i ) .

Original: 8: end for 9: end for 10: // Calculate the output on frequency domain: 11: for all a, b ∈ { 1 , . . .

Original: , w + 2 k } do 12:

Original: ˆ V =

Original: ˜

Original: V : , : ,a,b (cid:113) ||

Original: ˜ V : , : ,a,b ˜ V ⊺ : , : ,a,b || F // Rescale ˜ V .

Original: Calculate ˜ W : , : ,a,b =

Original: ( ˆ V ˆ V ∗ )

Original: − 12 ˆ V with Newton’s iteration.

Original: 14: ˜ Y : ,a,b = ˜ W : , : ,a,b ˜ X : ,a,b .

Original: 15: end for 16: // Get the final output: 17: for all i ∈ { 1 , . .

Original: , c out } do 18:

Original: Y i =

Original: FFT − 1 ( ˜ Y i ) .

Original: 19: end for 20: return ( Y : ,k : w + k,k : w + k ) .real

Original: C Proof of Theorem 5.1 and Theorem 5. C.1 Proof

Original: To prove Theorem 5.1 and Theorem 5.2, we first define the loss on a subset, so that we can divide the loss into different subset losses.

Original: Definition C.1 (Subset loss) .

Original: Define conditional loss:

Original: [Equation]

Original: ] to bethe loss function calculated only over the subset S ⊆ X , and define the subset loss:

Original: [Equation]

Original: A property of the subset loss is that, if S = S 1 ∪ S 2 where S 1 and S 2 are disjoint, then L m ( G, G ∗ | S ) =

Original: L m ( G, G ∗ | S 1 ) + L m ( G, G ∗ | S 2 ) .

Original: First, we provide some facts on the relationship between subset loss L m ( G, G ∗ | S ) and L m ( G, G pl | S ) when S ⊆ M ( G pl ) or

Original: S ⊆ M ( G pl ) .

Original: Fact C.1.

Original: When S ⊆ M ( G pl ) , then L m ( G, G ∗ | S ) =

Original: L m ( G, G pl | S ) ;

Original: when S ⊆ M ( G pl ) , then

Original: L m ( G, G ∗ | S ) + L m ( G, G pl | S )

Original: =

Original: P ( S ) .

Original: Proof.

Original: This is easy to see by noticing that when x ∈ M ( G pl ) , then G pl ( x ) =

Original: G ∗ ( x ) ; when x ∈ M ( G pl ) , then G pl ( x ) ̸ =

Original: G ∗ ( x ) , so G pl ( x ) + G ∗ ( x ) = 1 .

Original: Now, we define S 1 =

Original: S m B ( G ) ∩ M ( G pl ) and S 2 =

Original: S m B ( G ) ∩ M ( G pl ) .

Original: Base on the fact, we will be able to derive the relationship between L m ( G, G ∗ | S 1 ) and L m ( G, G ∗ | S 2 ) :

Original: Lemma C.2.

Original: We have the following relationship between the losses on the sets in which G pl is correct vs. G pl is wrong:

Original: [Equation]

Original: And therefore,

Original: [Equation]

Original: Proof.

Original: For the first equation, note that S 2 ⊆ M ( G pl ) and S 1 ⊆ M ( G pl ) , so based on the previous fact, we have:

Original: [Equation]

Original: For the second inequation, notice M ( G pl ) \ S m B ( G ) ⊆

Original: S 1 , so P ( S 1 ) ≥ P ( M ( G pl ))

Original: − P ( S m B

Original: G ))

Original: =

Original: Err ( G pl ) − R m B ( G ) .

Original: So:

Original: [Equation]

Original: Fact C.3.

Original: Given δ ∈ [0 , 1 c ] and β ∈ (0 , c − 12 ] , we can verify that:

Original: [Equation]

Original: Proof.

Original: This can be verified by substituting δ = 1 c into LHS, noticing that LHS is monotonically increasing w.r.t.

Original: δ .

Original: Lemma C.4.

Original: For any β ∈ (0 , c − 12 ] , define q = βc − 1 Err ( G pl ) and α = ( β − 1)

Original: Err ( G pl ) .

Original: If G fits the pseudolabels with suffcient accuracy and consistency:

Original: [Equation]

Original: Then G satisfies the following error bound:

Original: [Equation]

Original: The intuition of the proof is as follows.

Original: Lemma C.2 provides a relationship between the loss

Original: L m ( G, G ∗ | S 1 ) and L m ( G, G ∗ | S 2 ) .

Original: On the other hand, the expansion of S 1 is also related with S 2 by ( N ( S 1 ) \ S 1 ) ∩ S m B ( G ) ⊆

Original: S 2 .

Original: Note that the expansion P ( N ( S 1 )) >

Original: c · P ( S 1 ) , so S 1 cannot be too large or otherwise N ( S 1 ) \ S 1 will be too large to be within S 2 .

Original: We will show that L m ( G, G ∗ | S 1 ) < q .

Original: Proof.

Original: Consider the expansion of S 1 , N ( S 1 ) .

Original: Since S 1 ⊆ M ( G pl ) and P ( G pl ) < 1 /c , we know that c · P ( S 1 ) < 1 , so by the assumption of expansion, P ( N ( S 1 )) ≥ c · P ( S 1 ) .

Original: In addition, notice that ( N ( S 1 ) \ S 1 ) ∩ S m B ( G ) ⊆

Original: S 2 .

Original: Therefore, we have:

Original: [Equation]

Original: For the first term, we notice that N ( S 1 ) ∩ S m B ( G ) ⊆

Original: N ( S 1 ) and S 1 ⊆ S m B

Original: G ) , so the conditional loss satisfies L ( G, G ∗ |N

Original: ( S 1 ) ∩ S m B ( G )) /P

Original: ( N ( S 1 ) ∩ S

Original: [Equation]

Original: B ( G )) ≥ (1 − δ )

Original: L ( G, G ∗ | S 1 ) /P

Original: ( S 1 ) .

Original: Therefore,

Original: [Equation]

Original: G, G

Original: [Equation]

Original: Now, substituting L ( G, G ∗ | S 2 ) on the LHS with Lemma C.2 and noticing M ( G pl ) \ S m B

Original: ( G ) ⊆

Original: S 1 , with simple transformation we have:

Original: [Equation]

Original: The last inequality comes from the condition in the lemma.

Original: Thus, with Fact C.3, we know

Original: L ( G, G ∗ | S 1 ) ≤ α/

Original: [Equation]

Original: ( c (1 − δ ) − 2)) ≤ q .

Original: Now, we can bound the overall error:

Original: [Equation]

Original: Now, we will prove our main lemma, based on which we will be able to derive Theorem 5.1 and

Original: Theorem 5.2.

Original: Lemma C.5.

Original: Suppose Assumption 5.2 holds true.

Original: Then we can bound:

Original: [Equation]

Original: for any δ ∈ [0 , 1 c ] .

Original: Proof.

Original: First, we consider the case where L m ( G, G pl ) + 2 R m B ( G ) ≤ c − 12 ·

Original: Err ( G pl ) .

Original: In this case, we can find some β ∈ (0 , c − 12 ] such that.

Original: [Equation]

Original: Thus, by lemma C.4, we have:

Original: [Equation]

Original: Next, we consider the case where L m ( G, G pl ) + 2 R m B ( G ) >

Original: c − 12 · Err ( G pl ) .

Original: By triangle inequality, we have:

Original: [Equation]

Original: Now, we provide the proof of Theorem 5.1 and Theorem 5.2 based on Lemma C.5.

Original: Proof of Theorem 5.1.

Original: Since ˆ G is an optimizer of L ( G ) , we know Err m

Original: ( ˆ G ) ≤ L ( ˆ G ) ≤ L ( G ∗ ) .

Original: Substituting G ∗ into L ( G ) gives the bound in the theorem.

Original: Proof of Theorem 5.2.

Original: Note that CertR ( G ) ≥ 0 .

Original: 5 − Err ( G ) Lip ( G ) by its definition.

Original: Substituting Err ( G ) ≤

Original: L ( G ) gives the inequality in the theorem.

Original: D Error Control of Newton’s Method

Original: Recall that given an unconstrained matrix V ∈ R n × n , we know that W

Original: [-= ( V V T ) − 12 V is orthogonal, i.e., || W || 
-]
Revised: [-= ( V V T ) − 12 V is orthogonal, i.e., || W || 
-]

Original: = 1 , which provides certified robustness for the resulting model.

Original: In practice, we use a finite number of Newton’s iteration steps to approximate ( V V T ) − 12 .

Original: In this section, we provide the following theorem which rigorously control the spectral norm under finite Newton’s iteration steps.

Original: Theorem D.1.

Original: Given a matrix V ∈ R n × n such that

Original: ||

Original: I − V V T

Original: [-|| 2 < 1 , if we use Newton’s iteraton for k ⋆ steps following Equation (1) with initialization Y 
-]
Revised: [-|| 2 < 1 , if we use Newton’s iteraton for k ⋆ steps following Equation (1) with initialization Y 
-]

Original: [-= V V T and Z 
-]
Revised: [-= V V T and Z 
-]

Original: =

Original: I , then we have || Z k ⋆

Original: [Equation]

Original: where ρ min is the smallest eigenvalue of the matrix.

Original: Remark.

Original: We make sure the condition

Original: ||

Original: I − V V T

Original: || 2 < 1 is satisfied by rescaling as discussed in

Original: Section 4.1.

Original: As the theorem shows, along with the increase of Newton’s iteration step k ⋆ , the spectral norm of Z k ⋆ V approaches 1 where the additional term

Original: ||

Original: I − V V T

Original: || 2 k 2 decays exponentially.

Original: Hence, we can rigorously bound the error in the orthogonalization process caused by finite steps and use the bound to determine how many finite steps are needed.

Original: Indeed, in practice, we apply singular value decomposition to the computed matrix Z k ⋆ V , and find its maximum singular value always approaches 1 from the left side, i.e., the actual spectral norm is equal to or smaller than 1 .

Original: Detail experimental verification is in Appendix E.4.

Original: D.1 Proof of Theorem D.

Original: For brevity, throughout this section, for V ∈ R n × n , we define A = V V T .

Original: Note that A is a real symmetric matrix.

Original: Then, we recursively define

Original: [Equation]

Original: Before proving the main theorem, we first present the following three lemmas.

Original: Lemma D.1.

Original: [-For any k ∈ N , B k A = AB k and B k A 
-]
Revised: [-For any k ∈ N , B k A = AB k and B k A 
-]

Original: =

Original: A 12 B k .

Original: Proof of Lemma D.1.

Original: We prove the lemma by induction.

Original: [-Since B 0 = I , for k = 
-]
Revised: [-Since B 0 = I , for k = 
-]

Original: the lemma holds.

Original: Suppose that the lemma holds for k , then we have

Original: [Equation]

Original: In above equations, ( ∗ ) is due to the induction assumption that B k A = AB k or B k

Original: [-A 
-]
Revised: [-A 
-]

Original: =

Original: A 12 B k .

Original: Therefore, the lemma holds with k + 1 and by induction the lemma holds for any k ∈ N .

Original: Lemma D.2.

Original: For any k ∈ N , Y k

Original: = B k A and Z k = B k .

Original: Proof of Lemma D.2.

Original: We prove the lemma by induction.

Original: Since Y 0 =

Original: V V T =

Original: [-A , Z 
-]
Revised: [-A , Z 
-]

Original: = I , and

Original: [-B 
-]
Revised: [-B 
-]

Original: =

Original: [-I , for k = 
-]
Revised: [-I , for k = 
-]

Original: the lemma holds.

Original: Suppose that the lemma holds for k , then we have

Original: [Equation]

Original: where ( ∗ ) leverages B k A = AB k from Lemma D.1.

Original: Therefore, by induction the lemma holds for any k ∈ N .

Original: Lemma D.3.

Original: When || I − A

Original: || 2 < 1 , for any k ∈ N , the eigenvalue λ ∈ R of matrix

Original: A 12 B k is positive.

Original: Proof of Lemma D.3.

Original: We define C k

Original: =

Original: A 12 B k , then by leveraging the commutability between B k and A /

Original: A 12 (Lemma D.1)

Original: we have the following iteration:

Original: [Equation]

Original: Since || I − A

Original: [-|| 2 < 1 and A is a real symmetric matrix, any eigenvalue of C 0 , denoted by λ C 
-]
Revised: [-|| 2 < 1 and A is a real symmetric matrix, any eigenvalue of C 0 , denoted by λ C 
-]

Original: i , ∈ (0 , √ 2) .

Original: [-Denoting the diagonalization of C 0 by C 
-]
Revised: [-Denoting the diagonalization of C 0 by C 
-]

Original: = P − 1 D 0 P where D 0 = diag( λ C 0 1 , · · · , λ C 0 n ) .

Original: [-Then, from the iteration, we have C k +
-]
Revised: [-Then, from the iteration, we have C k +
-]

Original: =

Original: P − 1 (cid:0) 12 (3 D k − D 3 k ) (cid:1) P and therefore λ

Original: C k

Original: +1 i

Original: =

Original: λ C k i (cid:16) 32 − 12 ( λ C k i ) 2 (cid:17) .

Original: Define function f : R → R such that f ( x ) =

Original: x ( 32 − 12 x 2 ) .

Original: [-We find f ′ ( x ) = 
-]
Revised: [-We find f ′ ( x ) = 
-]

Original: ⇒ x ∈ { 0 , 1 } .

Original: Thus, when x ∈ (0 , √ 2) , f ( x ) ∈ (0 , 1) ⊆ (0 , √ 2) .

Original: Now we apply the induction.

Original: When k = 0 , we have λ C k i ∈ (0 , √ 2) .

Original: Suppose λ C k i ∈ (0 , √ 2) , from above result we have λ C k

Original: +1 i

Original: = f ( λ C k i ) ∈

Original: (0 , √ 2) .

Original: Thus, for any k ∈ N , all eigenvalues of C k

Original: =

Original: A 12 B k are positive.

Original: Now we are ready to prove the main theorem.

Original: Proof of Theorem D.1.

Original: Since Z k = B k for any k ∈ N acccording to Lemma D.2, we focus on

Original: B k and its expression of iteration (Equation (3))

Original: henceforth.

Original: According to [3, Section 6], define

Original: R k

Original: [-= I − B 2 k A , we have R k +
-]
Revised: [-= I − B 2 k A , we have R k +
-]

Original: = 34 R 2 k + 14 R 3 k .

Original: We have ||

Original: I − A

Original: [-|| 
-]
Revised: [-|| 
-]

Original: [-= || R 
-]
Revised: [-= || R 
-]

Original: || 2 < 1 .

Original: By induction,

Original: [Equation]

Original: by induction condition

Original: [Equation]

Original: Therefore, || R k ⋆ || 2 ≤

Original: || I − A

Original: [-|| 
-]
Revised: [-|| 
-]

Original: k⋆ 2 , i.e., the eigenvalues of the symmetric matrix I − B 2 k ⋆ A are in the range [ −|| I − A

Original: || 2 k⋆ 2 , ||

Original: I − A

Original: || 2 k⋆ 2 ] .

Original: Given an eigenvalue λ ∈ R with eigenvector x ∈ R n of the real symmetric matrix A 12 B k ⋆ , we have

Original: [Equation]

Original: Lemma

Original: [Equation]

Original: [Equation]

Original: Thus, (1 − λ 2 ) is an eigenvalue of matrix ( I − B 2 k ⋆ A ) .

Original: Since (1 − λ 2 ) ∈ [ −|| I − A

Original: || 2 k⋆ 2 , ||

Original: I − A

Original: || 2 k⋆ 2 ] , we get

Original: [Equation]

Original: Then, according to Lemma D.3, we know that λ > 0 and hence λ ∈ (cid:20)(cid:113)

Original: [Equation]

Original: Now we apply diagonalization to A 12 B k ⋆ :

Original: [Equation]

Original: As a result, ||

Original: [Equation]

Original: where the last equality uses the fact A − 12 V is a orthogonal matrix with spectral norm 1 .

Original: Since any eigenvector with eigenvalue λ of A − 12 corresponds to the eigenvector with eigenvalue (1 /λ 2 ) of A = V V T , and V V T as a symmetric real matrix only has real eigenvalues,

Original: [Equation]

Original: Plug it into Equation (11), we get

Original: [Equation]

Original: Notice that 1 − √ 1 − x ≤ x for x = ||

Original: I − A || 2 k⋆∈ [0 , 1] , we conclude the proof.

Original: E Addition Exp Results E.1 Visualization

Original: We show the representation visualization on 16 neurons for SOC and LOT in Figure 3.

Original: E.2 Residual Connection

Original: We show the loss landscape for both LOT and SOC in Figure

Original: E.3 Circular vs. Zero padding

Original: As we discussed in Section 4.1, the default parametrization leads to the convolution result with circular-padding, while we will pre-process the input to get the final result with zero-padding.

Original: In

Original: Table 5, we show the performance comparison between circular and zero padding.

Original: We can see that the performance significantly improves when we do the pre-processing and use the zero-padding instead of default circular-padding.

Original: [Figure]

Original: E.4 Error Control of Newton’s Iterations

Original: As we discuss in Section D, we use a finite number of Newton’s iteration to approximate the inverse square root of a matrix Z k ∗

Original: ≈ ( V V ⊺ ) − 12 .

Original: Theoretically, we have shown that the error of approximated orthogonal matrix Z k ∗ V will decay exponentially with number of iterations.

Original: Furthermore, we observe in practice that during the Newton’s iteration, the maximum singular value of Z k V will always approach 1 from the left side.

Original: As an example, we show the maximum singular value ( σ max ) for all

Original: LOT layers in LipConvnet-20 at different steps ( k ) during the Newton’s Iteration in Figure 5.

Original: We can see that σ max < 1 for all the layers, so we can safely assume that the Lipschitz bound is no larger than 1.

Original: In addition, σ max > 1 − 10 − 4 = 0 .

Original: 9999 after k = 8 iterations for all the layers, which indicates that the Newton’s iteration converges well.

Original: E.5 Supervised Learning without CReg Loss and HH Activation

Original: We show the results of semi-supervised learning under standard setting (without CReg Loss, HH

Original: Activation, and LLN on CIFAR-100) in Table 6 and Table 7.

Original: We can observe that we still achieve a good performance compared with SOC, and the gap is sometimes larger than with the different optimization techniques.

Original: E.6 Full semi-supervised

Original: We show the results for all architectures with semi-supervised learning in Table 8 (with CReg and

Original: HH) and 9 (without CReg and HH).

Original: We can see observe that we still achieve a better performance

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: compared with SOC.

Original: In addition, these results also have a larger certified accuracy at larger radius compared with the supervised learning setting.

Original: F Results on TinyImageNet

Original: To further validate the performance of LOT, we evaluate it on the TinyImageNet dataset which consists of 100,000 64 × 64 images in 200 classes.

Original: We use the same model architecture and training setting as before and show the comparison between LOT and SOC in Table 10.

Original: We can observe that

Original: LOT still outperforms the existing SOC approach in most cases.

Original: Meanwhile, we observe that all 1-Lipschitz models have a performance drop on the larger dataset compared with vanilla models, and we leave the breakthrough as future work.

Original: G Discussion of Model Architectures

Original: We would like to emphasize that there are two major differences in designing 1-Lipschitz networks and standard networks.

Original: First, standard architectures focus a lot on regularizing the network so that it does not overfit (e.g. with dropout), while in 1-Lipschitz networks we do not need strong regularization because the 1-Lipschitz constraint is already strong enough.

Original: Second, standard architectures are carefully designed so that the gradient will propagate in a proper manner (e.g. residual connection,

Original: BatchNorm), while 1-Lipschitz networks have an intrinsic gradient-norm-preserving property (see [16]).

Original: Thus, it may not be appropriate to directly utilize modern architectures in the design of 1-Lipschitz networks.

Original: In Table 11, we show the results of a 1-Lipschitz ResNet18 model, where we (1) replace all the conv layers with 1-Lipschitz convolution, (2) replace all the residual connection with average and (3) enforce the variance parameter of BatchNorm layers to be 1 so that the model is 1-Lipschitz.

Original: We can observe that LOT still outperforms SOC on both CIFAR-10 and TinyImageNet, while the performance of the ResNet18 architecture is not as good as that of LipConvNet.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]