Original: L IPS F ORMER : I NTRODUCING L IPSCHITZ C ONTINUITY TO V ISION T RANSFORMERS
Revised: L IPS F ORMER : I NTRODUCING L IPSCHITZ C ONTINUITY TO V ISION T RANSFORMERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present a Lipschitz continuous Transformer, called LipsFormer, to pursue training stability both theoretically and empirically for Transformer-based models.
Revised: We present a Lipschitz continuous Transformer, called LipsFormer, to pursue training stability both theoretically and empirically for Transformer-based models.

Original: In contrast to previous practical tricks that address training instability by learning rate warmup, layer normalization, attention formulation and weight initialization, we show that Lipschitz continuity is a more essential property to ensure training stability.
Revised: In contrast to previous practical tricks that address training instability by learning rate warmup, layer normalization, attention formulation and weight initialization, we show that Lipschitz continuity is a more essential property to ensure training stability.

Original: In LipsFormer, we replace unstable Transformer component modules with
Revised: In LipsFormer, we replace unstable Transformer component modules with

Original: Lipschitz continuous counterparts: CenterNorm instead of LayerNorm, spectral initialization instead of Xavier initialization, scaled cosine similarity attention instead of dot-product attention and weighted residual shortcut.
Revised: Lipschitz continuous counterparts: CenterNorm instead of LayerNorm, spectral initialization instead of Xavier initialization, scaled cosine similarity attention instead of dot-product attention, and weighted residual shortcut.

Original: We prove that these introduced modules are Lipschitz continuous and derive an upper bound on the Lipschitz constant of LipsFormer.
Revised: We prove that these introduced modules are Lipschitz continuous and derive an upper bound on the Lipschitz constant of LipsFormer.

Original: Our experiments show that LipsFormer allows stable training of deep Transformer architectures without the need of careful learning rate tuning such as warmup, yielding a faster convergence and better generalization.
Revised: Our experiments show that LipsFormer allows stable training of deep Transformer architectures without the need of careful learning rate tuning such as warmup, yielding a faster convergence and better generalization.

Original: As a result, on the ImageNet 1K dataset, LipsFormer-Tiny training for 100 epochs  without  learning ratewarmup attains a top-1 accuracy of 81.6% which is higher than Swin Transformer-Tiny training for 300 epochs with warmup.
Revised: As a result, on the ImageNet 1K dataset, LipsFormer-Swin-Tiny based on Swin Transformer training for 300 epochs can obtain 82.7% without any learning rate warmup.

Original: Moreover LipsFormer-Tiny training for 300 epochs achieves a top-1 accuracy of 83.5% with 4.7G FLOPs and 24M parameters.
Revised: Moreover, LipsFormer-CSwin-Tiny, based on CSwin, training for 300 epochs achieves a top-1 accuracy of 83.5% with 4.7G FLOPs and 24M parameters.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Transformer [43] has been widely adopted in natural language processing (NLP) [5, 24, 36] for its great capability of capturing long-range dependencies with self-attention .
Revised: Transformer [49] has been widely adopted in natural language processing (NLP) [6, 27, 40] for its great capability of capturing long-range dependencies with self-attention .

Original: Motivated by its success in NLP, Dosovitskiy et al.
Revised: Motivated by its success in NLP, Dosovitskiy et al.

Original: [14] introduced Vision Transformer (ViT) as a general backbone for computer vision tasks such as image classification [31, 47, 13], object detection [8, 53], and segmentation [9].
Revised: [17] introduced Vision Transformer (ViT) as a general backbone for computer vision tasks such as image classification [35, 53, 16], object detection [9, 59], and segmentation [12].

Original: Nowadays, Transformer [43] remains the dominant architecture for NLP [4, 5, 36], computer vision [52, 31, 47, 13] and many other AI applications [38, 37, 28].
Revised: Nowadays, Transformer [49] remains the dominant architecture for NLP [5, 6, 40], computer vision [58, 35, 53, 16] and many other AI applications [42, 41, 31].

Original: Despite its success, training Transformer remains challenging [29, 11] for practitioners: the training process can be prohibitively unstable, especially at the beginning of training.
Revised: Despite its success, training Transformer remains challenging [33, 14] for practitioners: the training process can be prohibitively unstable, especially at the beginning of training.

Original: To address the root cause for training instability, we resort to examining Lipschitz continuity of Transformer components.
Revised: To address the root cause for training instability, we resort to examining Lipschitz continuity of Transformer components.

Original: Intuitively, a Lipschitz continuous network is finite in the rate of change and its Lipschitz constant is an useful indicator for training stability.
Revised: Intuitively, a Lipschitz continuous network is finite in the rate of change and its Lipschitz constant is an useful indicator for training stability.

Original: As shown in [7, 6, 39], Lipschitz properties reveal intriguing behaviours of neural networks, such as robustness and generalization.
Revised: As shown in [8, 7, 44], Lipschitz properties reveal intriguing behaviours of neural networks, such as robustness and generalization.

Original: In this work, we focus on the trainability issue of Transformer architectures by explicitly enforcing Lipschitz continuity at network initialization.
Revised: In this work, we focus on the trainability issue of Transformer architectures by explicitly enforcing Lipschitz continuity at network initialization.

Original: Previous works for overcoming Transformer training instability usually focus on one or a combination of its components which can be divided into four categories: (1) improving normalization [48, 29, 45];
Revised: Previous works for overcoming Transformer training instability usually focus on one or a combination of its components which can be divided into four categories: (1) improving normalization [54, 33, 51];

Original: Xiong et al.
Revised: Xiong et al.

Original: [48] has shown that, for a Transformer architecture, Pre-LayerNorm (Pre-LN) is more stable than Post-LayerNorm (Post-LN).
Revised: [54] has shown that, for a Transformer architecture, Pre-LayerNorm (Pre-LN) is more stable than Post-LayerNorm (Post-LN).

Original: Liu et al.
Revised: Liu et al.

Original: [29] identified that Post-LN negatively influences training stability by amplifying parameter perturbations.
Revised: [33] identified that Post-LN negatively influences training stability by amplifying parameter perturbations.

Original: They introduced adaptive model initialization (Admin) to mitigate the amplification effect.
Revised: They introduced adaptive model initialization (Admin) to mitigate the amplification effect.

Original: Likewise, Wang et al.
Revised: Likewise, Wang et al.

Original: [45] introduced DeepNorm and a depth-specific initialization to stabilize Post-LN.
Revised: [51] introduced DeepNorm and a depth-specific initialization to stabilize Post-LN.

Original: However, even with normalization improvements such as Admin and DeepNorm, learning rate warmup [17] is still a necessity to stabilize training.
Revised: However, even with normalization improvements such as Admin and DeepNorm, learning rate warmup [20] is still a necessity to stabilize training.

Original: (2) more stable attention [25, 10]; Kim et al.
Revised: (2) more stable attention [28, 13]; Kim et al.

Original: [25] proved that the standard dot-product attention is not
Revised: [28] proved that the standard dot-product attention is not

Original: Lipschitz continuous and introduced an alternative L2 attention.
Revised: Lipschitz continuous and introduced an alternative L2 attention.

Original: (3) re-weighted residual shortcut ;
Revised: (3) re-weighted residual shortcut ;

Original: Bachlechner et al.
Revised: Bachlechner et al.

Original: [3] showed that a simple architecture change of gating each residual shortcut with a learnable zero-initialized parameter substantially stabilizes training.
Revised: [3] showed that a simple architecture change of gating each residual shortcut with a learnable zero-initialized parameter substantially stabilizes training.

Original: With ReZero, they were able to train extremely deep Transformers of 120 layers.
Revised: With ReZero, they were able to train extremely deep Transformers of 120 layers.

Original: (4) careful weight initialization ; To avoid gradient exploding or vanishing at the beginning of training, Zhang et al.
Revised: (4) careful weight initialization ; To avoid gradient exploding or vanishing at the beginning of training, Zhang et al.

Original: [54] proposed fixed-update initialization (Fixup) by rescaling a standard initialization.
Revised: [60] proposed fixed-update initialization (Fixup) by rescaling a standard initialization.

Original: They also proved that Fixup could enable stable training of residual networks without normalization.
Revised: They also proved that Fixup could enable stable training of residual networks without normalization.

Original: In this paper, we conduct a thorough analysis of Transformer architectures and propose a Lipschitz continuous Transformer called LipsFormer.
Revised: In this paper, we conduct a thorough analysis of Transformer architectures and propose a Lipschitz continuous Transformer called LipsFormer.

Original: In contrast to previous practical tricks that address training instability, we show that Lipschitz continuity is a more essential property to ensure training stability.
Revised: In contrast to previous practical tricks that address training instability, we show that Lipschitz continuity is a more essential property to ensure training stability.

Original: We focus our investigation on the following Transformer components: LayerNorm, dotproduct self-attention, residual shortcut, and weight initialization.
Revised: We focus our investigation on the following Transformer components: LayerNorm, dotproduct self-attention, residual shortcut, and weight initialization.

Original: For each analyzed module, we propose a Lipschitz continuous variant as a new building block for LipsFormer.
Revised: For each analyzed module, we propose a Lipschitz continuous variant as a new building block for LipsFormer.

Original: The final LipsFormer network has an upper bound Lipschitz constant at initial stages of training.
Revised: The final LipsFormer network has an upper bound Lipschitz constant at initial stages of training.

Original: Such a Lipschitz guarantee has two implications: 1) we can train LipsFormer without using the common trick of learning rate warmup, yielding a faster convergence and better generalization; 2) Transformer is more unstable at the beginning of training.
Revised: Such a Lipschitz guarantee has two implications: 1) we can train LipsFormer without using the common trick of learning rate warmup, yielding a faster convergence and better generalization; 2) Transformer is more unstable at the beginning of training.

Original: By ensuring initial network stability, we drastically increases the trainability of Transformer.
Revised: By ensuring initial network stability, we drastically increases the trainability of Transformer.

Original: Note that we could also enforce Lipschitz continuity during the whole training process by simply constraining updates on certain scaling parameters.
Revised: Note that we could also enforce Lipschitz continuity during the whole training process by simply constraining updates on certain scaling parameters.

Original: Our main contributions can be summarized as follows:
Revised: Our main contributions can be summarized as follows

Original:  • We give a thorough analysis of key Transformer components: LayerNorm, self-attention, residual shortcut, and weight initialization.
Revised: : • We give a thorough analysis of key Transformer components: LayerNorm, self-attention, residual shortcut, and weight initialization.

Original: More importantly, we identify potential instability problems each module brings to the training difficulty and propose their Lipschitz continuous counterparts: CenterNorm, scaled cosine similarity attention, scaled residual shortcut, and spectral-based initialization.
Revised: More importantly, we identify potential instability problems each module brings to the training difficulty and propose their Lipschitz continuous counterparts: CenterNorm, scaled cosine similarity attention, scaled residual shortcut, and spectral-based initialization.

Original: The proposed Lipschitz continuous modules can serve as drop-in replacements for a standard Transformer.
Revised: The proposed Lipschitz continuous modules can serve as drop-in replacements for a standard Transformer, such as Swin Transformer [35] and CSwin [16].

Original: • We propose a Lipschitz continuous Transformer (LipsFormer) that can be stably trained without the need of carefully tuning the learning rate schedule.
Revised: • We propose a Lipschitz continuous Transformer (LipsFormer) that can be stably trained without the need of carefully tuning the learning rate schedule.

Original: We derivea theoretical Lipschitz constant upper bound for  LipsFormer.
Revised: We derive theoretical Lipschitz constant upper bounds for both scaled cosine similarity attention and LipsFormer.

Original: The derivation provides a principled guidance for designing LipsFormer networks (e.g. choosing the weight of a residual shortcut according to the network depth).
Revised: The derivation provides a principled guidance for designing LipsFormer networks. We build LipsFormer-Swin and LipsFormer-CSwin based on Swin Transformer and CSwin individually.

Original: • We validate the efficacy of the LipsFormer on ImageNet classification.
Revised: • We validate the efficacy of the LipsFormer on ImageNet classification.

Original: We show empirically that LipsFormer can be trained smoothly without learning rate warmup.
Revised: We show empirically that LipsFormer can be trained smoothly without learning rate warmup.

Original: LipsFormer-Tiny training for 100 epochs without learning rate warmup attains a top-1 accuracy of 81.6% which is higher than Swin Transformer-Tiny training for 300 epochs with warmup.
Revised: As a result, on the ImageNet1K dataset, LipsFormer-Swin-Tiny training for 300 epochs can obtain a top-1 accuracy of 82.7% without any learning rate warmup.

Original: Moreover, LipsFormer-Tiny training for 300 epochs achieves a top-1 accuracy of 83.5% withonly 4.7G FLOPs and 24M parameters.
Revised: Moreover, LipsFormer-CSwin-Tiny training for 300 epochs achieves a top-1 accuracy of 83.5% with 4.7G FLOPs and 24M parameters.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In this section, we first define Lipschitz continuity and Lipschitz constant and then discuss several
Revised: In this section, we first define Lipschitz continuity and Lipschitz constant and then discuss several

Original: Lipschitz properties of a neural network.
Revised: Lipschitz properties of a neural network.

Original: We use the denominator-layout notation throughout this paper.
Revised: We use the denominator-layout notation throughout this paper.

Original: A sequence of N elements is denoted as X =
Revised: A sequence of N elements is denoted as X =

Original: [ x 1 ; . .
Revised: [ x 1 ; . .

Original: ; x N ] ⊤ ∈ R N × D , where each vector x i ∈ R D , i ∈ { 1 , ..., N } .
Revised: ; x N ] ⊤ ∈ R N × D , where each vector x i ∈ R D , i ∈ { 1 , ..., N } .

Original: Function transformation is parameterized by an associated weight matrix
Revised: Function transformation is parameterized by an associated weight matrix

Original: W and an affine transformation is denoted as f ( x ) =
Revised: W and an affine transformation is denoted as f ( x ) =

Original: W ⊤ x , where W ∈ R D × M .
Revised: W ⊤ x , where W ∈ R D × M .

Original: Definition 1.
Revised: Definition 1.

Original: A function f ( x , W ) :
Revised: A function f ( x , W ) :

Original: R D → R M is Lipschitz continuous (L-Lipschitz) under a choice of p-norm ∥ · ∥ p in the variable x
Revised: R D → R M is Lipschitz continuous (L-Lipschitz) under a choice of p-norm ∥ · ∥ p in the variable x

Original: if there exists a constant L such that for all ( x 1 , W ) and ( x 2 , W ) in the domain of f ,
Revised: if there exists a constant L such that for all ( x 1 , W ) and ( x 2 , W ) in the domain of f ,

Original: [Equation]
Revised: [Equation]

Original: where the smallest value of L that satisfies the inequality is called the Lipschitz constant of f .
Revised: where the smallest value of L that satisfies the inequality is called the Lipschitz constant of f .

Original: To emphasize that the Lipschitz constant with respect to x depends on W and the choice of p , we denote
Revised: To emphasize that the Lipschitz constant with respect to x depends on W and the choice of p , we denote

Original: L as Lip p ( f x ( W )) .
Revised: L as Lip p ( f x ( W )) .

Original: A function is generally referred to as expansive, non-expansive, and contractive in the variable x for Lip p ( f x ( W )) > 1 , Lip p ( f x ( W )) ≤ 1 , and Lip p ( f x ( W )) < 1 , respectively, exhibiting characteristic differences in the change rate of its output.
Revised: A function is generally referred to as expansive, non-expansive, and contractive in the variable x for Lip p ( f x ( W )) > 1 , Lip p ( f x ( W )) ≤ 1 , and Lip p ( f x ( W )) < 1 , respectively, exhibiting characteristic differences in the change rate of its output.

Original: Contemporary neural networks are rarely Lipschitz continuous under the influence of any constituent non-Lipschitz module.
Revised: Contemporary neural networks are rarely Lipschitz continuous under the influence of any constituent non-Lipschitz module.

Original: Even if a network is Lipschitz continuous, calculating its Lipschitz constant exactly is a challenging task [44].
Revised: Even if a network is Lipschitz continuous, calculating its Lipschitz constant exactly is a challenging task [50].

Original: According to Definition 1, the Lipschitz constant of f ( x , W ) with respect to x can be expressed as,
Revised: According to Definition 1, the Lipschitz constant of f ( x , W ) with respect to x can be expressed as,

Original: [Equation]
Revised: [Equation]

Original: Exact computation of the above equation is an NP-hard problem.
Revised: Exact computation of the above equation is an NP-hard problem.

Original: For subsequent analyses, We use p = 2 by default unless specified and suppress p to reduce clutter, but our conclusion can be easily extended to other choices of p .
Revised: For subsequent analyses, We use p = 2 by default unless specified and suppress p to reduce clutter, but our conclusion can be easily extended to other choices of p .

Original: Lemma 1.
Revised: Lemma 1.

Original: Given W , let f ( x , W ) :
Revised: Given W , let f ( x , W ) :

Original: R D → R be a continuously differentiable function and
Revised: R D → R be a continuously differentiable function and

Original: Lip( f x ( W )) be its Lipschitz constant with respect to x .
Revised: Lip( f x ( W )) be its Lipschitz constant with respect to x .

Original: According to the mean value theorem, we have the following inequality,
Revised: According to the mean value theorem, we have the following inequality,

Original: [Equation]
Revised: [Equation]

Original: where ∥∇ x f ( x , W ) ∥ is the gradient norm of f ( x , W ) with respect to x .
Revised: where ∥∇ x f ( x , W ) ∥ is the gradient norm of f ( x , W ) with respect to x .

Original: From Lemma 1, we can see that a practical method to compute the Lipschitz constant of a continuously differentiable function is to compute its maximum gradient norm.
Revised: From Lemma 1, we can see that a practical method to compute the Lipschitz constant of a continuously differentiable function is to compute its maximum gradient norm.

Original: To prove a function is not Lipschitz, it is sufficient to show that its gradient norm is not bounded.
Revised: To prove a function is not Lipschitz, it is sufficient to show that its gradient norm is not bounded.

Original: For example, f ( x ) = 1 x and f ( x ) = xare not Lipschitz continuous for x ∈ (0 , ∞ ) , because their gradient can be arbitrarily large as x approaches 0 and ∞ , respectively.
Revised: For example, f ( x ) = 1 x and f ( x ) = xare not Lipschitz continuous for x ∈ (0 , ∞ ) , because their gradient can be arbitrarily large as x approaches 0 and ∞ , respectively.

Original: Definition 2.
Revised: Definition 2.

Original: Let F ( x , { W l , l = 1 , . . .
Revised: Let F ( x , { W l , l = 1 , . . .

Original: , L } ) :
Revised: , L } ) :

Original: R D → R be an L-layer neural network defined as a composite function with L transformation functions:
Revised: R D → R be an L-layer neural network defined as a composite function with L transformation functions:

Original: [Equation]
Revised: [Equation]

Original: where { W l , l = 1 , . . .
Revised: where { W l , l = 1 , . . .

Original: , L } is the parameter set, and f l is the transformation function of the l -th layer.
Revised: , L } is the parameter set, and f l is the transformation function of the l -th layer.

Original: For an affine transformation f ( x , W ) =
Revised: For an affine transformation f ( x , W ) =

Original: W ⊤ x , its Lipschitz constant is,
Revised: W ⊤ x , its Lipschitz constant is,

Original: [Equation]
Revised: [Equation]

Original: where σ max ( W ) is the largest eigenvalue of W .
Revised: where σ max ( W ) is the largest eigenvalue of W .

Original: Many common activation functions such as Sigmoid, Tanh, ReLU, and GELU are 1-Lipschitz.
Revised: Many common activation functions such as Sigmoid, Tanh, ReLU, and GELU are 1-Lipschitz.

Original: See
Revised: See

Original: Appendix A.1 for a simple illustration.
Revised: Appendix A.1 for a simple illustration.

Original: Lemma 2.
Revised: Lemma 2.

Original: Given the Lipschitz constant of each transformation function in a network F , the following inequality holds
Revised: Given the Lipschitz constant of each transformation function in a network F , the following inequality holds

Original: [Equation]
Revised: [Equation]

Original: From Lemma 2, the Lipschitz constant of a network is upper bounded by the product of each layer’s Lipschitz constant.
Revised: From Lemma 2, the Lipschitz constant of a network is upper bounded by the product of each layer’s Lipschitz constant.

Original: This multiplicative nature gives us an insight into understanding why deeper networks usually suffer more severe training instability: if a network’s constituent layers are expansive, the upper bound of its Lipschitz constant increases monotonically with its network depth.
Revised: This multiplicative nature gives us an insight into understanding why deeper networks usually suffer more severe training instability: if a network’s constituent layers are expansive, the upper bound of its Lipschitz constant increases monotonically with its network depth.

Original: We refer the interested readers to [15, 27] for estimating tighter bounds of deep neural networks.
Revised: We refer the interested readers to [18, 30] for estimating tighter bounds of deep neural networks.

Original:  A N A SSUMPTION FOR T RAINING S TABILITY
Revised:  A N A SSUMPTION FOR T RAINING S TABILITY

Original: Our design philosophy for LipsFormer is based on the following assumption.
Revised: Our design philosophy for LipsFormer is based on the following assumption.

Original: Assumption 1.
Revised: Assumption 1.

Original: A network should satisfy the following Lipschitz conditions for training stability,
Revised: A network should satisfy the following Lipschitz conditions for training stability,

Original: [Equation]
Revised: [Equation]

Original: The first inequality focuses on the forward process and assumes that a stable network should satisfy
Revised: The first inequality focuses on the forward process and assumes that a stable network should satisfy

Original: Lipschitz continuity with respect to its input x : a small perturbation of its input should not lead to a drastic change of its output.
Revised: Lipschitz continuity with respect to its input x : a small perturbation of its input should not lead to a drastic change of its output.

Original: Guaranteeing smoothness is vital for guarding a network’s generalization ability.
Revised: Guaranteeing smoothness is vital for guarding a network’s generalization ability.

Original: For the second inequality, recall that the forward process of a typical neural network propagates computation as x l +

Revised: For the second inequality, recall that the forward process of a typical neural network propagates computation as x l +

Original: =
Revised: =

Original: ( W l +1 )
Revised: ( W l +1 )

Original: ⊤ x l , where x l and W l +1 are the input and weight matrix of Layer l +1 .
Revised: ⊤ x l , where x l and W l +1 are the input and weight matrix of Layer l +1 .

Original: Since common non-linearities are 1-Lipschitz, we drop non-linear activations here for simplicity.
Revised: Since common non-linearities are 1-Lipschitz, we drop non-linear activations here for simplicity.

Original: To backpropagate the network loss L , we have
Revised: To backpropagate the network loss L , we have

Original: [Equation]
Revised: [Equation]

Original: Gradient descent updates network weights according to W ← W − lr × ∂ L ∂ W .
Revised: Gradient descent updates network weights according to W ← W − lr × ∂ L ∂ W .

Original: As demonstrated above, any value explosion will propagate with the chain derivation: if ∂ L ∂ x l +1 is unbounded, ∂ L ∂ x l and ∂ L ∂ W l +1 will consequently be unbounded.
Revised: As demonstrated above, any value explosion will propagate with the chain derivation: if ∂ L ∂ x l +1 is unbounded, ∂ L ∂ x l and ∂ L ∂ W l +1 will consequently be unbounded.

Original: Meanwhile, if ∂ L ∂ W l +1 is not bound, it will largely influence the back-propagation chain in the next iteration.
Revised: Meanwhile, if ∂ L ∂ W l +1 is not bound, it will largely influence the back-propagation chain in the next iteration.

Original: This justifies the second inequality for the purpose of training stability.
Revised: This justifies the second inequality for the purpose of training stability.

Original: Intuitively, guaranteeing that a network’s output does not change too much under small perturbations of either its input or network weights induces a more stable training process.
Revised: Intuitively, guaranteeing that a network’s output does not change too much under small perturbations of either its input or network weights induces a more stable training process.

Original: In this work, we focus on satisfying the first inequality in Assumption 1 for Transformer architectures.
Revised: In this work, we focus on satisfying the first inequality in Assumption 1 for Transformer architectures.

Original:  L IPS F ORMER
Revised:  L IPS F ORMER

Original: A Lipschitz continuous Transformer (LipsFormer) requires all of its constituent modules to be
Revised: A Lipschitz continuous Transformer (LipsFormer) requires all of its constituent modules to be

Original: Lipschitz continuous according to Lemma 2.
Revised: Lipschitz continuous according to Lemma 2.

Original: In this section, we analyze key Transformer components and introduce their Lipschitz continuous counterparts when any Lipschitz continuity is violated.
Revised: In this section, we analyze key Transformer components and introduce their Lipschitz continuous counterparts when any Lipschitz continuity is violated.

Original: 4. L IPSCHITZ C ONTINUOUS M ODULES 4.1. C ENTER N ORM I NSTEAD OF L AYER N ORM
Revised: 4. L IPSCHITZ C ONTINUOUS M ODULES 4.1. C ENTER N ORM I NSTEAD OF L AYER N ORM

Original: LayerNorm [2] is the most widely used normalization method in Transformer.
Revised: LayerNorm [2] is the most widely used normalization method in Transformer.

Original: It is defined as
Revised: It is defined as

Original: LN( x ) =
Revised: LN( x ) =

Original: γ ⊙ z + β , where z = y
Revised: γ ⊙ z + β , where z = y

Original: Std( y ) and y = (cid:18)
Revised: Std( y ) and y = (cid:18)

Original: [Equation]
Revised: [Equation]

Original:  Std( y ) is the standard deviation of the mean-subtracted input y .
Revised: where x , y ∈ R D , Std( y ) is the standard deviation of the mean-subtracted input y , and ⊙ is an element-wise product.

Original: γ and β are initialized to 1 andrespectively.
Revised: γ and β are initialized to 1 and 0 respectively.

Original: For simplicity, we drop γ and β from analysis because they can be explicitly constrained within any pre-defined range.
Revised: For simplicity, we drop γ and β from analysis because they can be explicitly constrained within any pre-defined range.

Original: By taking partial derivatives, the Jacobian matrix of z with respect to x is,
Revised: By taking partial derivatives, the Jacobian matrix of z with respect to x is,

Original: [Equation]
Revised: [Equation]

Original: The equation above shows that LayerNorm is not Lipschitz continuous because when Std( y ) approaches 0, the values in the Jacobian matrix will approach ∞ , causing severe training instability.
Revised: The equation above shows that LayerNorm is not Lipschitz continuous because when Std( y ) approaches 0, the values in the Jacobian matrix will approach ∞ , causing severe training instability.

Original: On the other end, when Std( y ) is very large, training will be hindered by LayerNorm as gradients become extremely small.
Revised: On the other end, when Std( y ) is very large, training will be hindered by LayerNorm as gradients become extremely small.

Original: Also note that backpropagating through LayerNorm is slow due to poor parallelization when computing the Jacobian matrix, especially for the term I −
Revised: Also note that backpropagating through LayerNorm is slow due to poor parallelization when computing the Jacobian matrix, especially for the term I −

Original: yy ⊤ ∥
Revised: yy ⊤ ∥

Original: y ∥ 22 .
Revised: y ∥ 22 .

Original: In practice, we notice that a single LayerNorm operation could cause severe training instability without learning rate warmup.
Revised: In practice, we notice that a single LayerNorm operation could cause severe training instability without learning rate warmup.

Original: The underlying reason is that LayerNorm is not Lipschitz continuous and some ill-defined input with zero variance will lead to a Jacobian matrix filled with infinity.
Revised: The underlying reason is that LayerNorm is not Lipschitz continuous and some ill-defined input with zero variance will lead to a Jacobian matrix filled with infinity.

Original: To stabilize training by enforcing Lipschitz continuity, we introduce CenterNorm as,
Revised: To stabilize training by enforcing Lipschitz continuity, we introduce CenterNorm as,

Original: [Equation]
Revised: [Equation]

Original: where D is the dimension of x .
Revised: where D is the dimension of x .

Original: The Jacobian matrix ∂ CN( x ) ∂ x contains a term DD − 1 (cid:0) I − 1 D 11 ⊤ (cid:1) where DD − 1 is a heuristic to avoid the eigenvalue contraction from (cid:0) I − 1 D 11 ⊤ (cid:1) .
Revised: The Jacobian matrix ∂ CN( x ) ∂ x contains a term DD − 1 (cid:0) I − 1 D 11 ⊤ (cid:1) where DD − 1 is a heuristic to avoid the eigenvalue contraction from (cid:0) I − 1 D 11 ⊤ (cid:1) .

Original: It is easy to verify that,
Revised: It is easy to verify that,

Original: [Equation]
Revised: [Equation]

Original:  where Lip(CN x ) = DD − 1 for γ = 1 and β = 0 .
Revised: − CN( x 2 ) ∥ ≤ Lip(CN x ) ∥ x 1 − x 2 ∥ , where Lip(CN x ) = DD − 1 for γ = 1 and β = 0 .

Original: As most deep neural networks are dealing with high dimensional data with D ≫ 1 , we make a simplification that Lip(CN x ) is 1-Lipschitz for later discussions.
Revised: As most deep neural networks are dealing with high dimensional data with D ≫ 1 , we make a simplification that Lip(CN x ) is 1-Lipschitz for later discussions.

Original: CenterNorm is by design Lipschitz continuous at initialization.
Revised: CenterNorm is by design Lipschitz continuous at initialization.

Original: To guarantee its
Revised: To guarantee its

Original: Lipschitz continuity through training we could simply constraint γ and β to a pre-defined range.
Revised: Lipschitz continuity through training we could simply constraint γ and β to a pre-defined range.

Original: 4.1. S CALED C OSINE S IMILARITY A TTENTION
Revised: 4.1. S CALED C OSINE S IMILARITY A TTENTION

Original: Self-attention [43] is a key component of Transformer, helping capture long-range relationships within data.
Revised: Self-attention [49] is a key component of Transformer, helping capture long-range relationships within data.

Original: In practice, people use multi-head attention to effectively capture such relationships under different contexts.
Revised: In practice, people use multi-head attention to effectively capture such relationships under different contexts.

Original: Since multi-head attention is a linear combination of multiple single-head attention outputs, for simplicity, we focus our analysis on single-head attention, which is defined as,
Revised: Since multi-head attention is a linear combination of multiple single-head attention outputs, for simplicity, we focus our analysis on single-head attention, which is defined as,

Original: [Equation]
Revised: [Equation]

Original: where W Q , W K , W V are the projection matrices to transform X into query, key, and value matrices, respectively.
Revised: where W Q , W K , W V are the projection matrices to transform X into query, key, and value matrices, respectively.

Original: Intuitively, every token aggregates information from all the visible tokens by computing a weighted sum of the values of the visible tokens according to the similarity between its query and each visible token’s key.
Revised: Intuitively, every token aggregates information from all the visible tokens by computing a weighted sum of the values of the visible tokens according to the similarity between its query and each visible token’s key.

Original: The similarity between the i -th query q i and j -th key k j is denoted as
Revised: The similarity between the i -th query q i and j -th key k j is denoted as

Original: P ij ∝ x i ⊤ W Q ( W K )
Revised: P ij ∝ x i ⊤ W Q ( W K )

Original: ⊤ x j .
Revised: ⊤ x j .

Original: In [25], Kim et al.
Revised: In [28], Kim et al.

Original: proved that the standard dot-product self-attention is not Lipschitz continuous and introduced an alternative L2 self-attention that is Lipschitz continuous.
Revised: proved that the standard dot-product self-attention is not Lipschitz continuous and introduced an alternative L2 self-attention that is Lipschitz continuous.

Original: Here we use a scaled cosine similarity attention, which is defined as,
Revised: Here we use a scaled cosine similarity attention, which is defined as,

Original: [Equation]
Revised: [Equation]

Original: where P =
Revised: where P =

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: − ..
Revised: − ..

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: where ν and τ are predefined or learnable scalars; Q , K , V are ℓ 2 row-normalized: q i , k i , v i =
Revised: where ν and τ are predefined or learnable scalars; Q , K , V are ℓ 2 row-normalized: q i , k i , v i =

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ϵ is a smoothing factor to guarantee the validity of cosine similarity computation even when ∥ x i ⊤ W Q ∥ = 0 .
Revised: ϵ is a smoothing factor to guarantee the validity of cosine similarity computation even when ∥ x i ⊤ W Q ∥ = 0 .

Original: For arbitrary pair of rows of
Revised: For arbitrary pair of rows of

Original: Q and K denoted as q i and k j , the cosine similarity on their ℓ 2 -normalized vectors is proportional to their L2 dot product.
Revised: Q and K denoted as q i and k j , the cosine similarity on their ℓ 2 -normalized vectors is proportional to their L2 dot product.

Original: The upper bound of SCSA ’s Lipschitz constant with respect to ∥ · ∥ 2 and ∥ · ∥ ∞ is the following,
Revised: The upper bound of SCSA ’s Lipschitz constant with respect to ∥ · ∥ 2 and ∥ · ∥ ∞ is the following,

Original: Theorem 1.
Revised: Theorem 1.

Original: Single-head scaled cosine similarity attention is Lipschitz continuous, its Lip ∞ and
Revised: Single-head scaled cosine similarity attention is Lipschitz continuous, its Lip ∞ and

Original: Lip 2 are upper bounded by the following inequalities,
Revised: Lip 2 are upper bounded by the following inequalities,

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 1 can be found in Appendix D. For multi-head attention, we heuristically scale head feature concatenation by 1 K where K is the number of heads.
Revised: Proof of Theorem 1 can be found in Appendix H. For multi-head attention, we heuristically scale head feature concatenation by 1 K where K is the number of heads.

Original: Please refer to Appendix A.2 for more details.
Revised: Please refer to Appendix A.2 for more details.

Original: 4.1. W EIGHTED R ESIDUAL S HORTCUT
Revised: 4.1. W EIGHTED R ESIDUAL S HORTCUT

Original: Residual block [21] is a common component of contemporary neural networks [31, 47, 43].
Revised: Residual block [24] is a common component of contemporary neural networks [35, 53, 49].

Original: It has been proven effective in avoiding gradient vanishing, especially when training deep networks.
Revised: It has been proven effective in avoiding gradient vanishing, especially when training deep networks.

Original: A standard residual shortcut block is defined as,
Revised: A standard residual shortcut block is defined as,

Original: [Equation]
Revised: [Equation]

Original: The Lipschitz constant of a residual shortcut block with respect to x is Lip( RS x ( W ))
Revised: The Lipschitz constant of a residual shortcut block with respect to x is Lip( RS x ( W ))

Original: = 1 +
Revised: = 1 +

Original: Lip( f x ( W )) .
Revised: Lip( f x ( W )) .

Original: For any non-degenerate Lipschitz continuous function f ( x , W ) , its Lipschitz constant is greater than 0, hence a residual block is strictly expansive.
Revised: For any non-degenerate Lipschitz continuous function f ( x , W ) , its Lipschitz constant is greater than 0, hence a residual block is strictly expansive.

Original: According to Lemma 2, stacking
Revised: According to Lemma 2, stacking

Original: L identical residual blocks alone will grow the upper bound of a network’s Lipschitz constant exponentially to Lip( RS x ( W ))
Revised: L identical residual blocks alone will grow the upper bound of a network’s Lipschitz constant exponentially to Lip( RS x ( W ))

Original: L , causing substantial vulnerability to forward value explosion.
Revised: L , causing substantial vulnerability to forward value explosion.

Original: One way to mitigate such an instability is to constraint the Lipschitz constant of the residual path to be much smaller than 1, especially at the beginning of training when the network is undergoing fast changes via learning.
Revised: One way to mitigate such an instability is to constraint the Lipschitz constant of the residual path to be much smaller than 1, especially at the beginning of training when the network is undergoing fast changes via learning.

Original: In this paper, we explicitly multiply the residual path with a scale factor initialized to a small value such as 0.1 and 0.2.
Revised: In this paper, we explicitly multiply the residual path with a scale factor initialized to a small value such as 0.1 and 0.2.

Original: We define the weighted residual shortcut as,
Revised: We define the weighted residual shortcut as,

Original: [Equation]
Revised: [Equation]

Original: where α is a learnable parameter vector with the same dimension as the channel size of x .
Revised: where α is a learnable parameter vector with the same dimension as the channel size of x .

Original: It is easy to verify that
Revised: It is easy to verify that

Original: [Equation]
Revised: [Equation]

Original: where Lip(WRS x ( W ))
Revised: where Lip(WRS x ( W ))

Original: = 1 + max( α )
Revised: = 1 + max( α )

Original: when Lip( f x ( W ))
Revised: when Lip( f x ( W ))

Original: = 1 .
Revised: = 1 .

Original: As training progresses, α changes as part of the learning process.
Revised: As training progresses, α changes as part of the learning process.

Original: We could easily constrain α to a pre-defined range to ensure the Lipschitz continuity of a network during the whole training process.
Revised: We could easily constrain α to a pre-defined range to ensure the Lipschitz continuity of a network during the whole training process.

Original: Note that re-weighting shortcut and residual path has been explored before: in [45, 29], the authors redefine a residual block as α ⊙ x + f ( x , W ) to alleviate the LayerNorm instability; ReZero [3] uses a similar formulation as Equation 4 to speed up convergence, where α is a scalar instead of a vector.
Revised: Note that re-weighting shortcut and residual path has been explored before: in [51, 33], the authors redefine a residual block as α ⊙ x + f ( x , W ) to alleviate the LayerNorm instability; ReZero [3] uses a similar formulation as Equation 4 to speed up convergence, where α is a scalar instead of a vector.

Original: Our formulation is motivated by decreasing the Lipschitz constant of a network, instead of being a practical trick.
Revised: Our formulation is motivated by decreasing the Lipschitz constant of a network, instead of being a practical trick.

Original: It provides a more principled guidance to network design.
Revised: It provides a more principled guidance to network design.

Original: For example, when training a very deep network, a smaller α would be justified for the purpose of training stabilization.
Revised: For example, when training a very deep network, a smaller α would be justified for the purpose of training stabilization.

Original: 4.1. S PECTRAL I NITIALIZATION FOR C ONVOLUTION AND F EED - FORWARD C ONNECTION
Revised: 4.1. S PECTRAL I NITIALIZATION FOR C ONVOLUTION AND F EED - FORWARD C ONNECTION

Original: Both convolution and feed-forward connection are compositions of affine transformations.
Revised: Both convolution and feed-forward connection are compositions of affine transformations.

Original: As shown in Equation 1, affine transformation is Lipschitz continuous, hence by Lemma 2 both convolution and feed-forward connection are Lipschitz continuous.
Revised: As shown in Equation 1, affine transformation is Lipschitz continuous, hence by Lemma 2 both convolution and feed-forward connection are Lipschitz continuous.

Original: Note that a careful initialization is important for successfully training a neural network.
Revised: Note that a careful initialization is important for successfully training a neural network.

Original: Many initialization methods have been proposed before such as Xavier [16] and Kaiming [20] initialization.
Revised: Many initialization methods have been proposed before such as Xavier [19] and Kaiming [23] initialization.

Original: Inspired by spectral norm regularization [50], we introduce a 1-Lipschitz initialization called spectral initialization,
Revised: Inspired by spectral norm regularization [56], we introduce a 1-Lipschitz initialization called spectral initialization,

Original: [Equation]
Revised: [Equation]

Original: where W is Xavier-norm initialized and σ max ( W ) is its largest eigenvalue.
Revised: where W is Xavier-norm initialized and σ max ( W ) is its largest eigenvalue.

Original: For affine transformation f ( x , W si ) =
Revised: For affine transformation f ( x , W si ) =

Original: W si ⊤ x , its Lipschitz constant satisfies the following inequality,
Revised: W si ⊤ x , its Lipschitz constant satisfies the following inequality,

Original: [Equation]
Revised: [Equation]

Original: where Lip( f x ( W si ))
Revised: where Lip( f x ( W si ))

Original: = 1 at initialization.
Revised: = 1 at initialization.

Original: We use spectral initialization on all convolutions and feed-forward connections.
Revised: We use spectral initialization on all convolutions and feed-forward connections.

Original: 4. L IPS F ORMER 4.2. L IPS F ORMER B LOCK
Revised: 4. L IPS F ORMER 4.2. L IPS F ORMER B LOCK

Original: We start by introducing the main building block for LipsFormer.
Revised: We start by introducing the main building block for LipsFormer.

Original: As shown in Figure 1, each
Revised: As shown in Figure 1, each

Original: LipsFormer block (LipsBlock) is composed of three sub-modules: convolution blocks (lightweight depth-wise and element-wise convolution), scaled cosine similarity attention, and feed-forward connection.
Revised: LipsFormer block (LipsBlock) is composed of three sub-modules: convolution blocks (lightweight depth-wise and element-wise convolution), scaled cosine similarity attention, and feed-forward connection.

Original: CenterNorm operator is optional after each sub-module.
Revised: CenterNorm operator is optional after each sub-module.

Original: In this work we apply CenterNorm after scaled cosine similarity attention and feed-forward connection.
Revised: In this work we apply CenterNorm after scaled cosine similarity attention and feed-forward connection.

Original: Within each residual block, residual path is re-weighted with a learnable α and randomly dropped with probability p during training as indicated by dashed lines.
Revised: Within each residual block, residual path is re-weighted with a learnable α and randomly dropped with probability p during training as indicated by dashed lines.

Original: For the convolution blocks, we use a 7 × 7 depth-wise and a 1 × 1 element-wise convolution.
Revised: For the convolution blocks, we use a 7 × 7 depth-wise and a 1 × 1 element-wise convolution.

Original: Ablation study on each component can be found in Sec.
Revised: Ablation study on each component can be found in Sec.

Original: In Table 1 we compare the LipsFormer residual block with commonly used Post-Norm and PreNorm residual blocks.
Revised: In Table 1 we compare the LipsFormer residual block with commonly used Post-Norm and Pre-Norm residual blocks.

Original: CenterNorm and scaled cosine similarity attention are Lipschitz continuous counterparts for LayerNorm and dot-product attention.
Revised: CenterNorm and scaled cosine similarity attention are Lipschitz continuous counterparts for LayerNorm and dot-product attention.

Original: Weighted residual connection and DropPath  are used to constrain the Lipschitz constant of a deep LipsFormer network.
Revised: Weighted residual connection and DropPath [29] are used to constrain the Lipschitz constant of a deep LipsFormer network.

Original: [Figure]
Revised: [Figure]

Revised: р

Original: [Figure] 
Revised: [Figure] [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: T ABLE 1: Various forms of residual blocks for Transformer architectures.
Revised: T ABLE 1: Various forms of residual blocks for Transformer architectures.

Original: As illustrated in Figure 1, f represents a transformation function ∈ { self-attention, feed-forward } .
Revised: As illustrated in Figure 1, f represents a transformation function ∈ { self-attention, feed-forward } .

Original: For LipsFormer, f ∈ { scaled cosine similarity attention, feed-forward, convolution blocks } .
Revised: For LipsFormer, f ∈ { scaled cosine similarity attention, feed-forward, convolution blocks } .

Original: 4.2. O VERALL A RCHITECTURE OF L IPS F ORMER
Revised: 4.2. O VERALL A RCHITECTURE OF L IPS F ORMER

Original: In general, LipsFormer follows the architecture of Swin Transformer v1.
Revised: In general, LipsFormer follows the architecture of Swin Transformer v1.

Original: We start by processing an input image with non-overlapped convolutional token embedding ( 4 × 4 convolution with stride 4) to obtain a feature representation with resolution H 4 × W 4 .
Revised: We start by processing an input image with non-overlapped convolutional token embedding ( 4 × 4 convolution with stride 4) to obtain a feature representation with resolution H 4 × W 4 .

Original: Then the main computation passes four stages where each stage consists of a pre-defined number of LipsFormer blocks as shown in
Revised: Then the main computation passes four stages where each stage consists of a pre-defined number of LipsFormer blocks as shown in

Original: Figure 1.
Revised: Figure 1.

Original: Between consecutive stages, we reduce the output resolution by 2 and double the size of output channels by a 2 × 2 non-overlapped convolution with stride 2.
Revised: Between consecutive stages, we reduce the output resolution by 2 and double the size of output channels by a 2 × 2 non-overlapped convolution with stride 2.

Original: We build three variants of LipsFormer in correspondence with CSwin Transformer [13] as detailed in Appendix Table 4: LipsFormer-Tiny (LipsFormer-T) , LipsFormer-Small (LipsFormer-S), and LipsFormer-Base (LipsFormer-B).
Revised: We build three variants of LipsFormer in correspondence with CSwin Transformer [16] as detailed in Appendix Table 4: LipsFormer-CSwin-Tiny (LipsFormer-CSwin-T) , LipsFormer-CSwin-Small (LipsFormer-CSwin-S), and LipsFormer-CSwin-Base (LipsFormer-CSwin-B).

Original: The number of LipsFormer blocks within the four computation stages are [1, 2, 21, 1] for LipsFormer-T, [2, 4, 32, 2] for LipsFormer-S and
Revised: The number of LipsFormer blocks within the four computation stages are [1, 2, 21, 1] for LipsFormer-CSwin-T, [2, 4, 32, 2] for LipsFormer-CSwin-S and

Original: LipsFormer-B.
Revised: LipsFormer-CSwin-B.

Original: The overall architecture of LipsFormer is illustrated in Figure 3 of Appendix B.
Revised: The overall architecture of LipsFormer is illustrated in Figure 3 of Appendix B.

Revised: We can also build LipsFormer on Swin Transformer, more experiments about LipsFormer-Swin can be found in Appendix.

Original:  L IPSCHITZ C ONSTANT OF L IPS F ORMER
Revised: 4.2. L IPSCHITZ C ONSTANT OF L IPS F ORMER

Original: As illustrated in Figure 3, LipsFormer includes four computation stages, each starting with patch merging followed by a pre-defined number of LipsFormer blocks.
Revised: As illustrated in Figure 3, LipsFormer includes four computation stages, each starting with patch merging followed by a pre-defined number of LipsFormer blocks.

Original: Feed-forward connection, convolution, and patch merging are Lipschitz continuous operators.
Revised: Feed-forward connection, convolution, and patch merging are Lipschitz continuous operators.

Original: With spectral initialization these affine transformations are 1-Lipschitz at the beginning of training, hence are dropped from analysis.
Revised: With spectral initialization these affine transformations are 1-Lipschitz at the beginning of training, hence are dropped from analysis.

Original: For the
Revised: For the

Original: Lipschitz constant of the LipsFormer, we have the following theorem.
Revised: Lipschitz constant of the LipsFormer, we have the following theorem.

Original: Theorem 2.
Revised: Theorem 2.

Original: For a LipsFormer with S stages where the s -th stage has M s residual blocks, when α is set to 1 (cid:80) Ss =1 M s , the Lipschitz constant of the LipsFormer is upper bounded by exp( κ ) , whereκ = max( { Lip( f i ) :
Revised: For a LipsFormer with S stages where the s -th stage has M s residual blocks, when α is set to [Equation] the Lipschitz constant of the LipsFormer is upper bounded by exp( κ ) , where

Original: i = 1 , . .
Revised: [Equation]

Original: , (cid:80) Ss =1 M s } ) .

Original: Proof of Theorem 2 can be found in Appendix E. Theorem 2 suggests that 1) Deeper networks with more residual blocks should initialize with a smaller α to avoid exponential growth of its Lipschitz constant; 2) To control the Lipschitz constant of Lipsformer we should focus on constraining the 
Revised: Proof of Theorem 2 is in Appendix I. Theorem 2 suggests that 1) Deeper networks with more residual blocks should initialize with a smaller α to avoid exponential growth of its Lipschitz constant; 2) To control the Lipschitz constant of Lipsformer we should focus on constraining the Lipschitz constant

Original: Lipschitz constant Lip( f i ) of each constituent layer, especially the one with the largest Lipschitz constant.
Revised: Lip( f i ) of each constituent layer, especially the one with the largest Lipschitz constant.

Original:  E XPERIMENTS 5. D ATASET AND T RAINING S ETUP
Revised:  E XPERIMENTS 5. D ATASET AND T RAINING S ETUP

Original: We evaluate LipsFormer on the standard ImageNet-1K [12] dataset, which consists of 1.28M images and 1,000 classes.
Revised: We evaluate LipsFormer-CSwin on the standard ImageNet-1K [15] dataset, which consists of 1.28M images and 1,000 classes.

Original: We adopt a similar training strategy as in CSwin Transformer [13] for a fair comparison.
Revised: We adopt a similar training strategy as in CSwin Transformer [16] for a fair comparison.

Original: Specifically, we use the AdamW [34] optimizer with weight decay 0.05 for LipsFormerT/S and 0.1 for LipsFormer-B.
Revised: Specifically, we use the AdamW [38] optimizer with weight decay 0.05 for LipsFormerCSwin-T/S and 0.1 for LipsFormer-CSwin-B.

Original: By default, all our models are trained for 300 epochs with an input image size of 224 × 224 .
Revised: By default, all our models are trained for 300 epochs with an input image size of 224 × 224 .

Original: For LipsFormer, the training batch size is 2048 and the initial learning rate is 0.002 with a standard cosine learning rate decay [33] without learning rate warmup [33].
Revised: For LipsFormer-CSwin, the training batch size is 2048 and the initial learning rate is 0.002 with a standard cosine learning rate decay [37] without learning rate warmup [37].

Original: We apply stochastic depth [23] for LipsFormer-T, LipsFormer-S, andLipsFormer-B, with a maximum
Revised: We apply stochastic depth [26] for LipsFormer-CSwin-T, LipsFormer-CSwin-S, and

Original:  DropPath rate of 0.2, 0.4, and 0.5, respectively.
Revised: LipsFormer-CSwin-B, with a maximum DropPath rate of 0.2, 0.4, and 0.5, respectively.

Original: For ablation study, we train each model forepochs for efficiency.
Revised: For ablation study, we train each model for 100 epochs for efficiency.

Original: See Appendix C for more details.
Revised: See Appendix C for more details.

Original: 5. C OMPARISON WITH S TATE - OF - THE -A RT M ODELS
Revised: 5. C OMPARISON WITH S TATE - OF - THE -A RT M ODELS

Original: Table 2 reports the LipsFormer results compared with state-of-the-art CNN and Transformer models.
Revised: Table 2 reports the LipsFormer-CSwin results compared with state-of-the-art CNN and Transformer models.

Original: We evaluate all three variants of LipsFormer against state-of-the-art models of similar sizes: Tiny ( < 32M parameters), Small (32-64M parameters), and Base (64-96M parameters).
Revised: We evaluate all three variants of LipsFormer-CSwin against state-of-the-art models of similar sizes: Tiny ( < 32M parameters), Small (31-64M parameters), and Base (56-96M parameters).

Original: [Table]
Revised: [Table]

Original: Compared with previous state-of-the-art Vision Transformer models
Revised: Compared with previous state-of-the-art Vision Transformer models

Original: [Table]
Revised: [Table]

Original: LipsFormer attains a higher classification accuracy on all its model variants.
Revised: LipsFormer-CSwin attains a higher classification accuracy on all its model variants.

Original: For instance [Table] LipsFormer-T obtains a 83.5% Topaccuracy that outperforms CSwin-T by 0.8%, ViT-S by 2.1% and NAT-T by 0.3%.
Revised: For instance, LipsFormer-CSwin-T obtains a 83.5% Top-1 accuracy that outperforms CSwin-T by 0.8%, ViT-S by 2.1% and NAT-T by 0.3%.

Original: LipsFormer-T also outperforms recently improved CNN architectures, such as ConvNeXt-T and EffNet-B4 by 1.4% and 0.6%, respectively.
Revised: LipsFormer-CSwin-T also outperforms recently improved CNN architectures, such as ConvNeXt-T and EffNet-B4 by 1.4% and 0.6%, respectively.

Original: LipsFormer-T has fewer parameters than NAT-T and ConvNeXt-T. LipsFormer-B also outperforms its counterparts, including Swin-B, CrossFormer-L, ConvNeXt-B, DeiT-B, ViT-B, and NAT-B with fewer parameters.
Revised: LipsFormer-CSwin-T has fewer parameters than NAT-T and ConvNeXt-T. LipsFormer-CSwin-B also outperforms its counterparts, including Swin-B, CrossFormer-L, ConvNeXt-B, DeiT-B, ViT-B, and NAT-B with fewer parameters.

Original: Also note that all the other Transformer models use learning rate warmup, but LipsFormer does not.
Revised: Also note that all the other Transformer models use learning rate warmup, but LipsFormer-CSwin does not.

Original: 5. A BLATION S TUDY
Revised: 5. A BLATION S TUDY

Original: We conduct extensive ablation study on each key component of LipsFormer as shown in Table 3.
Revised: We conduct extensive ablation study on each key component of LipsFormer-CSwin as shown in Table 3.

Original: We use LipsFormer-T for ablation study and all results in this comparison are trained for 100 epochs without learning rate warmup, except for ablation on warmup.
Revised: We use LipsFormer-CSwin-T for ablation study and all results in this comparison are trained for 100 epochs without learning rate warmup, except for ablation on warmup.

Original: Warmup.
Revised: Warmup.

Original: In previous experiments we do not use learning rate warmup when training LipsFormer.
Revised: In previous experiments we do not use learning rate warmup when training LipsFormerCSwin.

Original: Theoretically, warmup is not needed given LipsFormer’s appealing stabilization guarantee.
Revised: Theoretically, warmup is not needed given LipsFormer’s appealing stabilization guarantee.

Original: According to the results in Table 3, 5 epochs of warmup does not bring in further improvement for LipsFormer.
Revised: According to the results in Table 3, 5 epochs of warmup does not bring in further improvement.

Original: CenterNorm.
Revised: CenterNorm.

Original: We compare CenterNorm against no-Norm (as in ReZero [3]) and the standard
Revised: We compare CenterNorm against no-Norm (as in ReZero [3]) and the standard

Original: LayerNorm.
Revised: LayerNorm.

Original: Results show that: 1) Lipsformer with LayerNorm becomes unstable and does not converge, but with CenterNorm and no-Norm, LipsFormer can successfully converge; 2)Using
Revised: Results show that: 1) Lipsformer-CSwin with LayerNorm becomes unstable and does not converge, but with CenterNorm and no-Norm, LipsFormer-CSwin can successfully converge; 2)

Original:  CenterNorm significantly outperforms no-Norm by 1.3%.
Revised: Using CenterNorm significantly outperforms no-Norm by 1.3%.

Original: [Table]
Revised: [Table]

Original: we
Revised: normal LipsFormer-CSwin with any of the three initializations converges. Spectral initialization and Xavier initialization slightly outperforms truncated normal , but

Original:  We 
Revised: Scaled Cosine Similarity Attention. To validate the effectiveness of scaled cosine similarity attention, we compare it with the standard dot-product attention and the L2 distance attention [28]. We find that the standard dot-product self-attention leads to forward value explosion, but the scaled cosine similarity attention works well under Lipschitz guarantee. Meanwhile, SCSA works better than the L2 distance attention.

Original: Impact of the Residual Weight α .
Revised: Impact of the Residual Weight α .

Original: As detailed in 4.1.3, the weight of residual path α has a substantial influence on the upper bound of LipsFormer’s Lipschitz constant.
Revised: As detailed in 4.1.3, the weight of residual path α has a substantial influence on the upper bound of LipsFormer’s Lipschitz constant.

Original: We evaluate different choices of α , and find that with a large α initialization value, network either does not converge or diverges quickly.
Revised: We evaluate different choices of α , and find that with a large α initialization value, network either does not converge or diverges quickly.

Original: This validates that deeper networks need a smaller α .
Revised: This validates that deeper networks need a smaller α .

Original: Convolution Blocks.
Revised: Convolution Blocks.

Original: In LipsFormer, we use two depth-wise convolutions (dwc) and one point-wise convolution (pwc).
Revised: In LipsFormer-CSwin, we use two depth-wise convolutions (dwc) and one pointwise convolution (pwc).

Original: We evaluate four different convolution configurations: A) no convolution; B) one dwc; C) dwc + pwc; and D) dwc + pwc + dwc.
Revised: We evaluate four different convolution configurations: A) no convolution; B) one dwc; C) dwc + pwc; and D) dwc + pwc + dwc.

Original: Table 3 shows that one dwc increases LipsFormer’s accuracy by 0.5%, one dwc + one pwc further improves its performance by 0.4%, adding more convolutions saturates performance gains.
Revised: Table 3 shows that one dwc increases LipsFormer’s accuracy by 0.5%, one dwc + one pwc further improves its performance by 0.4%, adding more convolutions saturates performance gains.

Original: DropPath Ratio.
Revised: DropPath Ratio.

Original: In Appendix F, we show that DropPath effectively decreases the upper bound of a network’s Lipschitz constant, making training process more stable.
Revised: In Appendix J, we show that DropPath effectively decreases the upper bound of a network’s Lipschitz constant, making training process more stable.

Original: The results in Table 3 show that reasonable DropPath can effectively improve training performance.
Revised: The results in Table 3 show that reasonable DropPath can effectively improve training performance.

Original: To summarize, CenterNorm, scaled cosine similarity attention, and convolution blocks all contribute positively to LipsFormer’s superior performance.
Revised: To summarize, CenterNorm, scaled cosine similarity attention, and convolution blocks all contribute positively to LipsFormer-CSwin’s superior performance.

Original: Weighted residual shortcut with small α , reasonable DropPath ratio p and spectral initialization are effective in stabilizing LipsFormer by constraining its Lipschitz constant.
Revised: Weighted residual shortcut with small α , reasonable DropPath ratio p and spectral initialization are effective in stabilizing LipsFormer-CSwin by constraining its Lipschitz constant.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we present a Lipschitz continuous Transformer, called LipsFormer, to pursue a more stable training process by enforcing the Lipschitz continuity of the whole network.
Revised: In this paper, we present a Lipschitz continuous Transformer, called LipsFormer, to pursue a more stable training process by enforcing the Lipschitz continuity of the whole network.

Original: We analyze key components of Transformer and replace the ones violating Lipschitz continuity by introducing
Revised: We analyze key components of Transformer and replace the ones violating Lipschitz continuity by introducing

Original: CenterNorm, scaled cosine similarity attention, and spectral initialization.
Revised: CenterNorm, scaled cosine similarity attention, and spectral initialization.

Original: LipsFormer also uses weighted residual shortcut and DropPath to further decrease the upper bound of its Lipschitz constant.
Revised: LipsFormer also uses weighted residual shortcut and DropPath to further decrease the upper bound of its Lipschitz constant.

Original: Finally, we derive an upper bound of the Lipschitz constant of a LipsFormer network architecture.
Revised: Finally, we derive an upper bound of the Lipschitz constant of a LipsFormer network architecture.

Original: We empirically validate the effectiveness of LipsFormer on ImageNet 1K classification with state-ofthe-art performance for model variants of different parameter sizes.
Revised: We empirically validate the effectiveness of LipsFormer-Swin and LipsFormer-CSwin, based on Swin Transformer and CSwin individually, on ImageNet 1K classification with state-of-the-art performance for model variants of different parameter sizes.

Original: The analysis of the Lipschitz continuity of a network is general.
Revised: The analysis of the Lipschitz continuity of a network is general.

Original: We look forward to extending it to a broader class of models and application areas, including multi-modal model and natural language processing.
Revised: We look forward to extending it to a broader class of models and application areas, including multi-modal model and natural language processing.

Original: We also hope future works will discuss the Lipschitz continuity of LipsFormer in the backward process in depth. 
Revised: We also hope future works will discuss the Lipschitz continuity of LipsFormer in the backward process in depth. A A PPENDIX

Original: A. L IPSCHITZ C ONSTANT OF C OMMON A CTIVATION F UNCTIONS
Revised: A. L IPSCHITZ C ONSTANT OF C OMMON A CTIVATION F UNCTIONS

Original: In Figure 2 we plot common non-linear activation functions in neural networks:
Revised: In Figure 2 we plot common non-linear activation functions in neural networks:

Original: Sigmoid, Tanh,
Revised: Sigmoid, Tanh,

Original: ReLU and GELU.
Revised: ReLU and GELU.

Original: According to [22], GeLU can be approximated by GeLU( x ) ≈ x sigmoid(1 .
Revised: According to [25], GeLU can be approximated by GeLU( x ) ≈ x sigmoid(1 .

Original: 702 x ) .
Revised: 702 x ) .

Original: According to Lemma 1, the Lipschitz constants of Sigmoid, Tanh, ReLU and GELU are 14 , 1, 1, 1.0998 respectively.
Revised: According to Lemma 1, the Lipschitz constants of Sigmoid, Tanh, ReLU and GELU are 14 , 1, 1, 1.0998 respectively.

Original: A. M ULTI - HEAD A TTENTION
Revised: A. M ULTI - HEAD A TTENTION

Original: For a K -head attention, we have the i -th attention, i ∈ { 1 , ..., K } defined as,
Revised: For a K -head attention, we have the i -th attention, i ∈ { 1 , ..., K } defined as,

Original: [Equation]
Revised: [Equation]

Original: where W i is the set of projection weight matrices ( W Qi , W Ki , W Vi ) .
Revised: where W i is the set of projection weight matrices ( W Qi , W Ki , W Vi ) .

Original: Multi-head attention simply concatenates different attention results,
Revised: Multi-head attention simply concatenates different attention results,

Original: [Equation]
Revised: [Equation]

Original: h [Equation] ( x
Revised: h 2 ( x

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: h K ( x , W K )] .
Revised: h K ( x , W K )] .

Original: According to the Lipschitz definition, we have, ∥ h ( x 1 , W ) − h ( x 2 , W ) ∥
Revised: According to the Lipschitz definition, we have, ∥ h ( x 1 , W ) − h ( x 2 , W ) ∥

Original: ≤ (Lip( h 1 ( W 1 ))
Revised: ≤ (Lip( h 1 ( W 1 ))

Original: + Lip( h 2 ( W 2 ))
Revised: + Lip( h 2 ( W 2 ))

Original: + ...
Revised: + ...

Original: + Lip( h K ( W K )))
Revised: + Lip( h K ( W K )))

Original: ∥ x 1 − x 2 ∥ .
Revised: ∥ x 1 − x 2 ∥ .

Original: B N ETWORK A RCHITECTURE AND C ONFIGURATIONS
Revised: B N ETWORK A RCHITECTURE AND C ONFIGURATIONS

Original: The overall architecture of LipsFormer is shown in Figure 3.
Revised: The overall architecture of LipsFormer-CSwin is shown in Figure 3.

Original: For patch embedding and patch merging, we use non-overlapped convolution as in Swin Transformer.
Revised: For patch embedding and patch merging, we use non-overlapped convolution as in Swin Transformer.

Original: Following CSwin Transformer, we use the same cross-shaped window when computing attention results and also the same Locally enhanced Positional Encoding (LePE).
Revised: Following CSwin Transformer, we use the same cross-shaped window when computing attention results and also the same Locally enhanced Positional Encoding (LePE).

Original: We base Lipsformer configurations  on CSwin Transformer and summarize the three variants in Table 4.
Revised: The configurations of Lipsformer-CSwin are based on CSwin Transformer and Table 4 summarizes three variants of Lipsformer-CSwin.

Revised: LipsFormer-CSwin-T and LipsFormer-CSwin-S only varies in the number of LipsFormer-CSwin blocks.

Original: LipsFormer-T and LipsFormer 
Revised: LipsFormer-CSwin-S/B share the same depth configuration but varies in hidden layer channel size. Similar to LipsFormer-CSwin, we also build LipsFormer based on Swin Transformer [35]. Here, we term it as LipsFormer-Swin. We create five versions of LipsFormer-Swin, and the detailed configurations are shown in Table 5.

Original: 4 ×
Revised: 4 ×

Original: P a t c h
Revised: P a t c h

Original: M e r g i ng
Revised: M e r g i ng

Original: L i p s F o r m e r
Revised: L i p s F o r m e r

Original: B l o ck 2 ×
Revised: B l o ck 2 ×

Original: P a t c h
Revised: P a t c h

Original: M e r g i ng
Revised: M e r g i ng

Original: L i p s F o r m e r
Revised: L i p s F o r m e r

Original: B l o ck 2 ×
Revised: B l o ck 2 ×

Original: P a t c h
Revised: P a t c h

Original: M e r g i ng
Revised: M e r g i ng

Original: L i p s F o r m e r
Revised: L i p s F o r m e r

Original: B l o ck 2 ×
Revised: B l o ck 2 ×

Original: P a t c h
Revised: P a t c h

Original: M e r g i ng
Revised: M e r g i ng

Original: L i p s F o r m e r
Revised: L i p s F o r m e r

Original: B l o ck
Revised: B l o ck

Original: P oo li ng & F C
Revised: P oo li [Table] F C

Original: [Table]
Revised: [Table]

Original: 𝐻4×𝑊 𝐻8×𝑊 𝐻16×𝑊 𝐻32×𝑊
Revised: 𝐻4×𝑊 𝐻8×𝑊 𝐻16×𝑊 𝐻32×𝑊

Original: [Table]
Revised: [Table]

Original: H×𝑊
Revised: H×𝑊

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C T RAINING D ETAILS
Revised: C T RAINING D ETAILS

Original: In Table 5 we provide the ImageNet 1K training details used for producing the main results in Table 2.
Revised: In Table 6 we provide the ImageNet 1K training details used for producing the main results in Table 2.

Original: All LipsFormer variants use the same training hyperparameters, except for DropPath ratio, weight decay, learning rate and EMA.
Revised: All LipsFormer variants use the same training hyperparameters, except for DropPath ratio, weight decay, learning rate and EMA.

Original: All the models are implemented with PyTorch, and trained on NVIDIA
Revised: All the models are implemented with PyTorch, and trained on NVIDIA

Original: Tesla A100 GPUs.
Revised: Tesla A100 GPUs.

Original: We do not use learning rate warmup in all experiments.
Revised: We do not use learning rate warmup in all experiments.

Original: Hyperparameters
Revised: D E XPERIMENTS OF L IPS F ORMER -S WIN

Original: Tiny
Revised: We evaluate the Tiny, Small, Base and Large versions of LipsFormer-Swin on the ImageNet-1K, and compare our results with their corresponding counterpart Swin Transformer. The results are shown in

Original: [Table]
Revised: Table 7.

Original: Optimizer
Revised: We have the following two findings from Table 7, • the proposed LipsFormer-Swin consistently outperforms its counterpart Swin Transformer. Specifically, LipsFormer-Swin-T improves Swin-T by 1.5%.

Original: AdamW
Revised: • LipsFormer-Swin-L shows obvious overfitting on ImageNet-1K, and performs worst than LipsFormer-Swin-B.

Original: AdamW
Revised: According to our observation in the training process, the training loss (around 2.2) of LipsFormer-Swin-L is much smaller than that (around 2.5) of LipsFormer-Swin-B, but the test accuracy is lower.) We also observe that in some github discussion issues, some people also find that the original Swin L cannot outperform Swin

Original: AdamW
Revised: B if

Original: [Table]
Revised: only training on ImageNet-1K. Since LipsFormer-Swin-L has shown overfitting on ImageNet-1K, we do not report the performance of LipsFormer-Swin-L++ on the table.

Original: [Table]
Revised: In the future, we will train it on a larger scale of data to test its fitting ability.

Original: residual
Revised: On a single A100-40GB GPU, with a batch size fixed to 256 and a mixed precision,

Original: [Table]
Revised: [Table]

Original: in

Original: [Table] 
Revised: [Table] [Table]

Revised: s

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: the throughput and peak memory are 963 im/s and 5483 MB for LipsFormer-Swin-T, and 433 im/s and 7415 MB for LipsFormer-Swin-B.

Revised: E O VERFITTING E VALUATION

Revised: Following DeiT III [47], we also evaluate our method on ImageNet-v2 [43] and ImageNet-real [4] data sets.

Revised: As pointed out by [48], to test how the method performs in a nearby setting without any finetuning is a good way to assess overfitting.

Revised: We directly apply the obtained models trained on the original ImageNet data set onto these two data sets.

Revised: The results are shown in Table 8.

Revised: We can see that from Table 8, our method that works well on the original ImageNet data set consistently performs well on the ImageNet-v2 and ImageNet-real data sets.

Revised: This observation fully validates the generalization ability of the proposed method.

Original: D P ROOF OF T HEOREM
Revised: F T RAINING C URVES

Revised: In Figure 4, we show the training curves of the training losses and the classification accuracies of

Revised: LipsFormer-Swin-T and LipsFormer-Swin-B.

Revised: We can find LipsFormer-Swin-B can fit the training data better than LipsFormer-Swin-T because the loss of LipsFormer-Swin-B is much lower than that of LipsFormer-Swin-T.

Revised: [Table]

Revised: [Figure]

Revised: G P ARAMETER V ARIATIONS ALONG WITH T RAINING E POCHS

Original: In this subsection
Revised: In Figure 5, we show the variations of the α along with the training epochs. Our statistic is based on LipsFormer-Swin-T model. We statistic the mean and standard variance of the absolute value of the α . We select one set of α from each stage.

Revised: We find that from Figure 5, the mean value of the absolute value of the α first grows and then tend to stabilize at a value.

Original: [Table]
Revised: [Figure]

Revised: H P ROOF OF T HEOREM

Original:  we derive the Lipschitz constant upper bound for the scaled cosine similarity attention (SCSA) [Table]
Revised: In this subsection, we derive the Lipschitz constant upper bound for the scaled cosine similarity attention (SCSA).

Original: First [Table] we list some useful notations and identities for deriving the Jacobians of attention computation.
Revised: First, we list some useful notations and identities for deriving the Jacobians of attention computation.

Original: [Equation]
Revised: [Equation]

Original: For column vectors u , z ∈ R N the chain rule has:
Revised: For column vectors u , z ∈ R N the chain rule has:

Original: [Equation]
Revised: [Equation]

Original: The standard dot-product attention is defined as,
Revised: The standard dot-product attention is defined as,

Original: [Equation]
Revised: [Equation]

Original: In [25], Kim et al.
Revised: In [28], Kim et al.

Original: proved that the standard dot-product attention is not Lipschitz continuous, and proposed L2-distance attention which is Lipschitz continuous conditioning on W Q = W K .
Revised: proved that the standard dot-product attention is not Lipschitz continuous, and proposed L2-distance attention which is Lipschitz continuous conditioning on W Q = W K .

Original: But enforcing the equality of W Q and W K limits the expressiveness of the Transformer and degrades training performance empirically.
Revised: But enforcing the equality of W Q and W K limits the expressiveness of the Transformer and degrades training performance empirically.

Original: Our scaled cosine similarity attention is defined as,
Revised: Our scaled cosine similarity attention is defined as,

Original: [Equation]
Revised: [Equation]

Original: where P = softmax (cid:0) τ QK ⊤ (cid:1) ,
Revised: where P = softmax [Equation]

Original: [Equation]
Revised: [Equation]

Original: 6) ν

Original:  and τ are predefined or learnable scalars.
Revised: ν and τ are predefined or learnable scalars.

Original: The definitions of Q , K , V are as follows,
Revised: The definitions of Q , K , V are as follows,

Original: [Equation]
Revised: [Equation]

Revised:   − q ⊤− ...

Revised: − q ⊤ N − 

Revised:  ∈ R N × D ,

Revised: K =   − k ⊤− ...

Revised: − k ⊤ N − 

Revised:  ∈ R N × D ,

Revised: V =   − v ⊤− ...

Revised: − v ⊤ N − 

Revised:  ∈ R N × D .

Original: For each input x i , the projected q i , k i , v i are defined as,
Revised: For each input x i , the projected q i , k i , v i are defined as,

Original: [Equation]
Revised: [Equation]

Original: where ϵ is a small smoothing factor to guarantee that the definition of cosine similarity is valid everywhere.
Revised: where ϵ is a small smoothing factor to guarantee that the definition of cosine similarity is valid everywhere.

Original: By taking partial derivatives
Revised: By taking partial derivatives

Original: [Equation]
Revised: [Equation]

Original: we have the following Jacobian matrices,
Revised: we have the following Jacobian matrices,

Original: [Equation]
Revised: [Equation]

Original: The attention matrix P is defined as,
Revised: The attention matrix P is defined as,

Original: [Equation]
Revised: [Equation]

Original: We can rewrite our SCSA attention in Eq.
Revised: We can rewrite our SCSA attention in Eq.

Original: 6 as,
Revised: 6 as,

Original: [Equation]
Revised: [Equation]

Original: For simplification we focus on derivations for single-head attention, mutli-head attention requires only minor modifications for concatenating attention results for each head as discussed in A.2 .
Revised: For simplification we focus on derivations for single-head attention, mutli-head attention requires only minor modifications for concatenating attention results for each head as discussed in A.2 .

Original: The
Revised: The

Original: Jacobian matrix for SCSA can be written as,
Revised: Jacobian matrix for SCSA can be written as,

Original: [Equation]
Revised: [Equation]

Original: whereJ ij =
Revised: where

Original: ∂f i ( X ) ∂ x j ∈ R D × D .
Revised: [Equation]

Original: By taking partial derivatives we can show that,
Revised: By taking partial derivatives we can show that,

Original: [Equation]
Revised: [Equation]

Original: where E ij ∈ R N × N is a binary matrix with zeros everywhere except the (i, j)-th entry, δ ij is the
Revised: where E ij ∈ R N × N is a binary matrix with zeros everywhere except the (i, j)-th entry, δ ij is the

Original: Kronecker delta, and the Jacobian of the softmax is well-known as below,
Revised: Kronecker delta, and the Jacobian of the softmax is well-known as below,

Original: [Equation]
Revised: [Equation]

Original: When i =
Revised: When i =

Original: j , we have,
Revised: j , we have,

Original: [Equation]
Revised: [Equation]

Original: When i ̸
Revised: When i ̸

Original: = j , we have,
Revised: = j , we have,

Original: [Equation]
Revised: [Equation]

Original: Lemma 3.
Revised: Lemma 3.

Original: The scaled cosine similarity attention (SCSA) is Lipschitz continuous
Revised: The scaled cosine similarity attention (SCSA) is Lipschitz continuous

Original: if and only if
Revised: if and only if

Original: W Q , W K , W V have bounded norm.
Revised: W Q , W K , W V have bounded norm.

Original: Sketch Proof.
Revised: Sketch Proof.

Original: Our key observation is that most of the terms in J ii and J ij have bounded norm: ν and τ are scalars; Q , K , V are normalized so all elements are less than or equal to 1; E ij has zeros everywhere except the (i,j)-th entry; P is an attention matrix with all elements within [0, 1] so all elements in P ( i ) are bounded by [-0.25, 0.25].
Revised: Our key observation is that most of the terms in J ii and J ij have bounded norm: ν and τ are scalars; Q , K , V are normalized so all elements are less than or equal to 1; E ij has zeros everywhere except the (i,j)-th entry; P is an attention matrix with all elements within [0, 1] so all elements in P ( i ) are bounded by [-0.25, 0.25].

Original: Taking a closer look at (cid:102) Q i , (cid:102) K i , (cid:102)
Revised: Taking a closer look at (cid:102) Q i , (cid:102) K i , (cid:102)

Original: V i as shown in Eq.
Revised: V i as shown in Eq.

Original: Eq. 8 and Eq. 9, they are bounded as long as W Q , W K , W V are bounded.
Revised: Eq. 8 and Eq. 9, they are bounded as long as W Q , W K , W V are bounded.

Original: Consequently the final product of J ii and J ij have bounded norm if W Q , W K , W V have bounded norm.
Revised: Consequently the final product of J ii and J ij have bounded norm if W Q , W K , W V have bounded norm.

Original: D. U PPER B OUND ON L IP ∞ F OR SCSA 
Revised: H.

Revised: U PPER B OUND ON L IP ∞ F OR SCSA

Original: Let us review some basic definitions for matrix norm.
Revised: Let us review some basic definitions for matrix norm.

Original: Suppose we have matrices A ∈ R N × D , and
Revised: Suppose we have matrices A ∈ R N × D , and

Original: B ∈ R N × D .
Revised: B ∈ R N × D .

Original: Then, we have:
Revised: Then, we have:

Original: [Equation]
Revised: [Equation]

Original: We also have the following inequalities,
Revised: We also have the following inequalities,

Original: [Equation]
Revised: [Equation]

Original: where ∥ · ∥ F is the Frobenius norm.
Revised: where ∥ · ∥ F is the Frobenius norm.

Original: Equality holds if and only if matrix A is a rank-one matrix or a zero matrix.
Revised: Equality holds if and only if matrix A is a rank-one matrix or a zero matrix.

Original: According to the above inequalities
Revised: According to the above inequalities

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]

Original: [Equation]

Original: J

Original: [Equation]
Revised: [Equation]

Original: We can compute the L 2 norm Lipschitz constant by replacing the L ∞ norm in the above equation with L 2 norm.
Revised: We can compute the L 2 norm Lipschitz constant by replacing the L ∞ norm in the above equation with L 2 norm.

Original: With simple derivations we list ∥ · ∥ ∞ for each term in 16:
Revised: With simple derivations we list ∥ · ∥ ∞ for each term in 16:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: for Equation
Revised: for Equation

Revised: [Equation]

Revised: ∥

Original: [Equation]
Revised: [Equation]

Original:  to Equation 
Revised: ∥ [Equation] similar to Equation [Equation]

Revised: ∥

Original: [Equation]
Revised: [Equation]

Original: According to
Revised: ∥

Original: [Equation]
Revised: [Equation]

Revised: ∥ (cid:102) V j ∥ ∞ ≤ 2 × ϵ − 1 2 × ∥

Revised: W V ⊤ ∥ ∞ , similar to Equation 17,

Original:  the Lip ∞ constant of the scaled cosine similarity attention (SCSA) is:
Revised: According to 16, the Lip ∞ constant of the scaled cosine similarity attention (SCSA) is:

Original: [Equation]
Revised: [Equation]

Original: After merging and rearranging the terms,
Revised: After merging and rearranging the terms

Original: [Equation]
Revised: [Equation]

Original: D. U PPER B OUND ON L IP 2 F OR SCSA
Revised: H. U PPER B OUND ON L IP 2 F OR SCSA

Original: Correspondingly, we list ∥ · ∥ 2 for each term in 16:
Revised: Correspondingly, we list ∥ · ∥ 2 for each term in 16:

Original: [Equation]
Revised: [Equation]

Original: Proof of Equation
Revised: Proof of Equation

Original: According to Eq 12, P ( i ) is a semi-definite matrix, thus its ordered eigenvalues λ 1 ≥ λ 2 ≥ , . .
Revised: According to Eq 12, P ( i ) is a semi-definite matrix, thus its ordered eigenvalues λ 1 ≥ λ 2 ≥ , . .

Original: , ≥ λ N ≥ 0 , and (cid:80) Ni
Revised: , ≥ λ N ≥ 0 , and (cid:80) Ni

Original: =

Revised: =

Original: λ i = tr( P ( i ) )
Revised: λ i = tr( P ( i ) )

Original: = (cid:80) Nj P
Revised: = (cid:80) Nj P

Original: ( i ) jj ≤ ( (cid:80) Nj =1 1 N N − 1 N ) =
Revised: ( i ) jj ≤ ( (cid:80) Nj =1 1 N N − 1 N ) =

Original: N − 1 N .
Revised: N − 1 N .

Original: According to 15, ∥ P ( i ) ∥ 2 = ( (cid:80) Ni =1 λ 2 i ) 12 ≤ ( (cid:80)
Revised: According to 15, ∥ P ( i ) ∥ 2 = ( (cid:80) Ni =1 λ 2 i ) 12 ≤ ( (cid:80)

Original: Ni =
Revised: Ni =

Original: 1 λ i )
Revised: 1 λ i )

Original: 2 × 12 ≤ N − 1 N∥ E ii ∥ 2 =
Revised: 2 × 12 ≤ N − 1 N

Original: [Equation]
Revised: [Equation]

Original: Substituting the above results into Eq.
Revised: Substituting the above results into Eq.

Original: 16 and changing L ∞ norm to L 2 norm, we have
Revised: 16 and changing L ∞ norm to L 2 norm, we have

Original: [Equation] From the upper bound above
Revised: Lip

Original: [Equation]
Revised: [Equation]

Original:  we highlight the following observations: 1)In contrast to L2 attention detailed in [25] where W Q and W K are constrained to be the same, we do not limit expressiveness of the attention computation; 2) ϵ is to guarantee validity of cosine similarity computation when any participating vector is equal to zero; 3) In Lip( SCSA ) 2 , the scale factor for the first term is 2 N ( N − 1) ντϵ − 12 , which multiplies with an extra ∼ N when compared to the other terms, meaning that ∥ W K ∥ 2 plays a more significant role in the Lipschitz constant of Lip( SCSA ) 2 . 
Revised: From the upper bound above, we highlight the following observations: 1) ϵ is to guarantee validity of cosine similarity computation when any participating vector is equal to zero; 2) In Lip( SCSA ) 2 , the scale factor for the first term is 2 N ( N − 1) ντϵ − 12 , which multiplies with an extra ∼ N when compared to the other terms, meaning that ∥ W K ∥ 2 plays a more significant role in the Lipschitz constant of Lip( SCSA ) 2 . Different from the L2 distance attention [28], to promise the module is Lipschitz continuous, the scaled cosine similarity attention has no requirement for the weight matrices, but the L2 distance attention detailed in [28] requires that W Q and W K should be the same.

Original: D. C OMPARISON WITH D OT -P RODUCT A TTENTION [43] AND L2-A TTENTION [25]
Revised: H. C OMPARISON WITH D OT -P RODUCT A TTENTION [49] AND L2-A TTENTION [28]

Original: As proved in [25], the standard dot-product attention is not Lipschitz continuous.
Revised: As proved in [28], the standard dot-product attention is not Lipschitz continuous.

Original: The proposed
Revised: The proposed

Original: L2-attention is also not Lipschitz continuous for general W Q and W K , but only Lipschitz continuous when W Q = W K .
Revised: L2-attention is also not Lipschitz continuous for general W Q and W K , but only Lipschitz continuous when W Q = W K .

Original: However, enforcing W Q = W K degrades model performance as shown in [25].
Revised: However, enforcing W Q = W K degrades model performance as shown in [28].

Original: As proved above, the scaled cosine similarity attention (SCSA) is in general Lipschitz continuous, only requiring that W Q , W K , W V have bounded norm and that the computation of cosine similarity is valid.
Revised: As proved above, the scaled cosine similarity attention (SCSA) is in general Lipschitz continuous, only requiring that W Q , W K , W V have bounded norm and that the computation of cosine similarity is valid.

Original: We can easily guarantee that our computation of similarity is valid by introducing a small smoothing factor ϵ .
Revised: We can easily guarantee that our computation of similarity is valid by introducing a small smoothing factor ϵ .

Original: E P ROOF OF T HEOREM
Revised: I P ROOF OF T HEOREM

Original: In this section, we give the upper bound on LipsFormer’s Lipschitz constant.
Revised: In this section, we give the upper bound on LipsFormer’s Lipschitz constant.

Original: For a LipsFormer with S stages where the s -th stage has M s residual blocks, its Lipschitz constant is upper bounded by the inequality below,
Revised: For a LipsFormer with S stages where the s -th stage has M s residual blocks, its Lipschitz constant is upper bounded by the inequality below,

Original: [Equation]
Revised: [Equation]

Original: Here, we define κ = max( { Lips( f i ) : i = 1 , . . .
Revised: Here, we define κ = max( { Lips( f i ) : i = 1 , . . .

Original: , (cid:80) Ss =1 M s } ) .
Revised: , (cid:80) Ss =1 M s } ) .

Original: When α is set to
Revised: When α is set to

Original: [Equation]
Revised: [Equation]

Original:  , the above inequality can be rewritten as,
Revised: Ss =1 M s , the above inequality can be rewritten as,

Original: [Equation]
Revised: [Equation]

Original: F D ROPPATH IS AN EFFICIENT WAY TO CONSTRAINT THE L IPSCHITZ CONSTANT
Revised: J D ROPPATH IS AN EFFICIENT WAY TO CONSTRAINT THE L IPSCHITZ CONSTANT

Original: DropPath [26] is another effective technique for training deep transformers, where [Equation] residual path is dropped [Equation]
Revised: DropPath [29] is another effective technique for training deep transformers, where y = (cid:26) x , if residual path is dropped x + α · f ( x ) otherwise

Original: otherwise When using DropPath with drop probability p within each residual block, the Lipschitz constant of
Revised: When using DropPath with drop probability p within each residual block, the Lipschitz constant of

Original: LipsFormer is refined as,
Revised: LipsFormer is refined as,

Original: [Equation]
Revised: [Equation]

Original: where droppath( α s,m , p ) = (cid:26) 0 , with probability p α s,m Lip( f s,m
Revised: where droppath( α s,m , p ) = (cid:26) 0 , with probability p α s

Original: [Equation]
Revised: [Equation]

Original: ) with probability 1 − p .
Revised: m Lip( f s,m )) with probability 1 − p .

Original: We can see that DropPath effectively decreases the upper bound of a network’s Lipschitz constant by randomly dropping the contributions of residual paths.
Revised: We can see that DropPath effectively decreases the upper bound of a network’s Lipschitz constant by randomly dropping the contributions of residual paths.

Original: G C OMPARISON WITH E XISTING W ORKS
Revised: K C OMPARISON WITH E XISTING W ORKS

Original: In this section,  we provide a detailed comparison  with existing works, including Admin [29], ReZero [3], Swin-V2 [30], DeepNorm [45], L2 self-attention [25] and Spectral Normalization [50].
Revised: In this section, to clarify our contribution more clearly, we provide a detailed comparison of our method with existing works, including Admin [33], ReZero [3], Swin-V2 [34], DeepNorm [51], L self-attention [28] and Spectral Normalization [56].

Original: Admin [29] identifies that within a residual block, the residual branch amplifies network output and the amplification effect makes training unstable.
Revised: Admin [33] identifies that within a residual block, the residual branch amplifies network output and the amplification effect makes training unstable.

Original: They propose to initialize the weight contributions of a residual branch according to the variance of its previous layer.
Revised: They propose to initialize the weight contributions of a residual branch according to the variance of its previous layer.

Original: ReZero [3] introduces an effective strategy to improve training stability.
Revised: ReZero [3] introduces an effective strategy to improve training stability.

Original: They notice that initializing the residual branch with 0 satisfies initial dynamical isometry, thus stabilizes model training.
Revised: They notice that initializing the residual branch with 0 satisfies initial dynamical isometry, thus stabilizes model training.

Original: ReZero demonstrates that they can train very deep transformer without warmup but it requires removing
Revised: ReZero demonstrates that they can train very deep transformer without warmup but it requires removing

Original: Layer Normalization.
Revised: Layer Normalization.

Original: According to Equation 19, initializing the residual contribution to 0 trivially constraints network Lipschitz constant.
Revised: According to Equation 19, initializing the residual contribution to 0 trivially constraints network Lipschitz constant.

Original: However, with Layer Normalization back into the network,
Revised: However, with Layer Normalization back into the network,

Original: ReZero is likely to encounter training instability again.
Revised: ReZero is likely to encounter training instability again.

Original: DeepNorm [45] shares similar motivation with Admin [29] and analyzes the influence of the residual block and initialization.
Revised: DeepNorm [51] shares similar motivation with Admin [33] and analyzes the influence of the residual block and initialization.

Original: They introduce a new normalization function to modify the residual connection in Transformer and propose a new initialization method.
Revised: They introduce a new normalization function to modify the residual connection in Transformer and propose a new initialization method.

Original: However, we observe that learning rate warmup is still necessary in DeepNorm [45].
Revised: However, we observe that learning rate warmup is still necessary in DeepNorm [51].

Original: Training of Admin [29] and DeepNorm [45] requires learning rate warmup, ReZero [3] could train without learning rate warmup but requires that LayerNorm is not present in the network.
Revised: Training of Admin [33] and DeepNorm [51] requires learning rate warmup, ReZero [3] could train without learning rate warmup but requires that LayerNorm is not present in the network.

Original: The analyses of Admin, ReZero, and DeepNorm are not from the perspective of Lipschitz continuity.
Revised: The analyses of Admin, ReZero, and DeepNorm are not from the perspective of Lipschitz continuity.

Original: In [50], Yuichi et al.
Revised: In [56], Yuichi et al.

Original: introduce a simple and effective spectral norm regularization, which penalizes high spectral norm of weight matrices in neural networks.
Revised: introduce a simple and effective spectral norm regularization, which penalizes high spectral norm of weight matrices in neural networks.

Original: This work focuses on regularization without considering residual block and self-attention block.
Revised: This work focuses on regularization without considering residual block and self-attention block.

Original: In [25], Kim et al.
Revised: In [28], Kim et al.

Original: prove that the standard dot-product self-attention is not Lipschitz continuous.
Revised: prove that the standard dot-product self-attention is not Lipschitz continuous.

Original: They introduce an alternative L2 self-attention that is Lipschitz continuous under the constraint that
Revised: They introduce an alternative L2 self-attention that is Lipschitz continuous under the constraint that

Original: W Q = W K .
Revised: W Q = W K .

Original: Such constraint limits expressiveness of the attention block and empirically degrades training performance.
Revised: Such constraint limits expressiveness of the attention block and empirically degrades training performance.

Original: Also, L2 self-attention focuses only on the Lipschitz continuity of self-attention block.
Revised: Also, L2 self-attention focuses only on the Lipschitz continuity of self-attention block.

Original: Swin-V2 [30] introduces two strategies to improve training stability of transformer model, including replacing post-norm with pre-norm and a scaled cosine attention replacing the original dot product attention.
Revised: Swin-V2 [34] introduces two strategies to improve training stability of transformer model, including replacing post-norm with pre-norm and a scaled cosine attention replacing the original dot product attention.

Original: The introduced scale cosine attention is defined as,
Revised: The introduced scale cosine attention is defined as,

Original: [Equation]
Revised: [Equation]

Original: It should be noted that there’s a difference between cos ( q i , k j ) /τ and τ cos ( q i , k j ) , the former is not a Lipschitz continuous function with respect to variable τ but the latter is.
Revised: It should be noted that there’s a difference between cos ( q i , k j ) /τ and τ cos ( q i , k j ) , the former is not a Lipschitz continuous function with respect to variable τ but the latter is.

Original: According to our derivation, self-attention based on the scaled cosine attention defined as in Swin-V2 is not Lipschitz continuous if V is not normalized.
Revised: According to our derivation, self-attention based on the scaled cosine attention defined as in Swin-V2 is not Lipschitz continuous if V is not normalized.

Original: The above-mentioned works only deal with one or several standard neural computation modules.
Revised: The above-mentioned works only deal with one or several standard neural computation modules.

Original: Our LipsFormer gives a holistic Lipschitz analysis of a typical transformer network instead of focusing exclusively on a single or few constituent modules.
Revised: Our LipsFormer gives a holistic Lipschitz analysis of a typical transformer network instead of focusing exclusively on a single or few constituent modules.

Original: To derive the Lipschitz constant of
Revised: To derive the Lipschitz constant of

Original: LipsFormer, we analyze each constituent module of a standard transformer, including convolutions, fully-connected layer, self-attention, normalization and residual block.
Revised: LipsFormer, we analyze each constituent module of a standard transformer, including convolutions, fully-connected layer, self-attention, normalization and residual block.

Original: In this work we propose a
Revised: In this work we propose a

Original: Lipschitz continuous self-attention and construct a Lipschitz continuous transformer network by bounding each constituent computation layer.
Revised: Lipschitz continuous self-attention and construct a Lipschitz continuous transformer network by bounding each constituent computation layer.

Original: The resultant LipsFormer induces stable training and does not require learning rate warmup.
Revised: The resultant LipsFormer induces stable training and does not require learning rate warmup.

Revised: We summarize our contributions from both theoretical and empirical perspectives as follows,

Revised: Theoretically, • We derive a theoretical Lipschitz constant upper bound for scaled cosine similarity attention.

Revised: Meanwhile, we give a thorough analysis of key Transformer components: LayerNorm, selfattention, residual shortcut, and weight initialization.

Revised: • We propose a Lipschitz continuous Transformer (LipsFormer), and derive a theoretical Lipschitz constant upper bound for LipsFormer.

Revised: The derivation provides a principled guidance for designing LipsFormer networks.

Revised: Empirically, • We make an assumption about the Lipschitz continuity of the network, and experimentally validate this assumption.

Revised: • We build LipsFormer on CSwin and Swin-Transformer.

Revised: We validate the efficacy of the different versions (Tiny, Small, Base, Large and Large++) of LipsFormer on ImageNet, ImageNet-v2 and ImageNet-Real data sets.