Original: C ONTRA N ORM : A C ONTRASTIVE L EARNING P ERSPECTIVE ON O VERSMOOTHING AND B EYOND
Revised: C ONTRA N ORM : A C ONTRASTIVE L EARNING P ERSPECTIVE ON O VERSMOOTHING AND B EYOND

Original: A BSTRACT
Revised: A BSTRACT

Original: Oversmoothing is a common phenomenon in a wide range of Graph Neural Networks (GNNs) and Transformers, where performance worsens as the number of layers increases.
Revised: Oversmoothing is a common phenomenon in a wide range of Graph Neural Networks (GNNs) and Transformers, where performance worsens as the number of layers increases.

Original: Instead of characterizing oversmoothing from the view of complete collapse in which representations converge to a single point, we dive into a more general perspective of dimensional collapse in which representations lie in a narrow cone.
Revised: Instead of characterizing oversmoothing from the view of complete collapse in which representations converge to a single point, we dive into a more general perspective of dimensional collapse in which representations lie in a narrow cone.

Original: Accordingly, inspired by the effectiveness of contrastive learning in preventing dimensional collapse, we propose a novel normalization layer called
Revised: Accordingly, inspired by the effectiveness of contrastive learning in preventing dimensional collapse, we propose a novel normalization layer called

Original: ContraNorm .
Revised: ContraNorm .

Original: Intuitively, ContraNorm implicitly shatters representations in the embedding space, leading to a more uniform distribution and a slighter dimensional collapse.
Revised: Intuitively, ContraNorm implicitly shatters representations in the embedding space, leading to a more uniform distribution and a slighter dimensional collapse.

Original: On the theoretical analysis, we prove that ContraNorm can alleviate both complete collapse and dimensional collapse under certain conditions.
Revised: On the theoretical analysis, we prove that ContraNorm can alleviate both complete collapse and dimensional collapse under certain conditions.

Original: Our proposed normalization layer can be easily integrated into GNNs and Transformers with negligible parameter overhead.
Revised: Our proposed normalization layer can be easily integrated into GNNs and Transformers with negligible parameter overhead.

Original: Experiments on various real-world datasets demonstrate the effectiveness of our proposed ContraNorm.
Revised: Experiments on various real-world datasets demonstrate the effectiveness of our proposed ContraNorm.

Original: Our implementation is available at https://github.com/PKU-ML/ContraNorm .
Revised: Our implementation is available at https://github.com/PKU-ML/ContraNorm .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recently, the rise of Graph Neural Networks (GNNs) has enabled important breakthroughs in various fields of graph learning (Ying et al., 2018; Senior et al., 2020).
Revised: Recently, the rise of Graph Neural Networks (GNNs) has enabled important breakthroughs in various fields of graph learning (Ying et al., 2018; Senior et al., 2020).

Original: Along the other avenue, although getting rid of bespoke convolution operators, Transformers (Vaswani et al., 2017) also achieve phenomenal success in multiple natural language processing (NLP) tasks (Lan et al., 2020; Liu et al., 2019; Rajpurkar et al., 2018) and have been transferred successfully to computer vision (CV) field (Dosovitskiy et al., 2021;
Revised: Along the other avenue, although getting rid of bespoke convolution operators, Transformers (Vaswani et al., 2017) also achieve phenomenal success in multiple natural language processing (NLP) tasks (Lan et al., 2020; Liu et al., 2019; Rajpurkar et al., 2018) and have been transferred successfully to computer vision (CV) field (Dosovitskiy et al., 2021;

Original: Liu et al., 2021; Strudel et al., 2021).
Revised: Liu et al., 2021; Strudel et al., 2021).

Original: Despite their different model architectures, GNNs and Transformers are both hindered by the oversmoothing problem (Li et al., 2018;
Revised: Despite their different model architectures, GNNs and Transformers are both hindered by the oversmoothing problem (Li et al., 2018;

Original: Tang et al., 2021), where deeply stacking layers give rise to indistinguishable representations and significant performance deterioration.
Revised: Tang et al., 2021), where deeply stacking layers give rise to indistinguishable representations and significant performance deterioration.

Original: In order to get rid of oversmoothing, we need to dive into the modules inside and understand how oversmoothing happens on the first hand.
Revised: In order to get rid of oversmoothing, we need to dive into the modules inside and understand how oversmoothing happens on the first hand.

Original: However, we notice that existing oversmoothing analysis fails to fully characterize the behavior of learned features.
Revised: However, we notice that existing oversmoothing analysis fails to fully characterize the behavior of learned features.

Original: A canonical metric for oversmoothing is the average similarity (Zhou et al., 2021;
Revised: A canonical metric for oversmoothing is the average similarity (Zhou et al., 2021;

Original: Gong et al., 2021; Wang et al., 2022).
Revised: Gong et al., 2021; Wang et al., 2022).

Original: The tendency of similarity converging to 1 indicates representations shrink to a single point (complete collapse).
Revised: The tendency of similarity converging to 1 indicates representations shrink to a single point (complete collapse).

Original: However, this metric can not depict a more general collapse case, where representations span a lowdimensional manifold in the embedding space and also sacrifice expressive power, which is called dimensional collapse (left figure in Figure 1).
Revised: However, this metric can not depict a more general collapse case, where representations span a lowdimensional manifold in the embedding space and also sacrifice expressive power, which is called dimensional collapse (left figure in Figure 1).

Original: In such cases, the similarity metric fails to quantify the collapse level.
Revised: In such cases, the similarity metric fails to quantify the collapse level.

Original: Therefore, we need to go beyond existing measures and take this so-called dimensional collapse into consideration.
Revised: Therefore, we need to go beyond existing measures and take this so-called dimensional collapse into consideration.

Original: Actually, this dimensional collapse behavior is widely discussed in the contrastive learning literature (Jing et al., 2022; Hua et al., 2021; Chen & He, 2021;
Revised: Actually, this dimensional collapse behavior is widely discussed in the contrastive learning literature (Jing et al., 2022; Hua et al., 2021; Chen & He, 2021;

Original: Grill et al., 2020), which may hopefully help us characterize the oversmoothing problem of GNNs and
Revised: Grill et al., 2020), which may hopefully help us characterize the oversmoothing problem of GNNs and

Original: Transformers.
Revised: Transformers.

Original: The main idea of contrastive learning is maximizing agreement between different augmented views of the same data example (i.e. positive pairs) via a contrastive loss.
Revised: The main idea of contrastive learning is maximizing agreement between different augmented views of the same data example (i.e. positive pairs) via a contrastive loss.

Original: Common contrastive loss can be decoupled into alignment loss and uniformity loss (Wang & Isola, 2020).
Revised: Common contrastive loss can be decoupled into alignment loss and uniformity loss (Wang & Isola, 2020).

Original: The two ingredients correspond to different objectives: alignment loss expects the distance between positive pairs to be closer, while uniformity loss measures how well the embeddings are uniformly distributed.
Revised: The two ingredients correspond to different objectives: alignment loss expects the distance between positive pairs to be closer, while uniformity loss measures how well the embeddings are uniformly distributed.

Original: Pure training with only alignment loss may lead to a trivial solution where all representations shrink to one single point.
Revised: Pure training with only alignment loss may lead to a trivial solution where all representations shrink to one single point.

Original: Fortunately, the existence of uniformity loss naturally helps solve this problem by drawing representations evenly distributed in the embedding space.
Revised: Fortunately, the existence of uniformity loss naturally helps solve this problem by drawing representations evenly distributed in the embedding space.

Original: Given the similarities between the oversmoothing problem and the representation collapse issue, we establish a connection between them.
Revised: Given the similarities between the oversmoothing problem and the representation collapse issue, we establish a connection between them.

Original: Instead of directly adding uniformity loss into model training, we design a normalization layer that can be easily used out of the box with almost no parameter overhead.
Revised: Instead of directly adding uniformity loss into model training, we design a normalization layer that can be easily used out of the box with almost no parameter overhead.

Original: To achieve this, we first transfer the uniformity loss used for training to a loss defined over graph node representations, thus it can optimize representations themselves rather than model parameters.
Revised: To achieve this, we first transfer the uniformity loss used for training to a loss defined over graph node representations, thus it can optimize representations themselves rather than model parameters.

Original: Intuitively, the loss meets the need of drawing a uniform node distribution.
Revised: Intuitively, the loss meets the need of drawing a uniform node distribution.

Original: Following the recent research in combining optimization scheme and model architecture (Yang et al., 2021; Zhu et al., 2021; Xie et al., 2021; Chen et al., 2022), we use the transferred uniformity loss as an energy function underlying our proposed normalization layer, such that descent steps along it corresponds with the forward pass.
Revised: Following the recent research in combining optimization scheme and model architecture (Yang et al., 2021; Zhu et al., 2021; Xie et al., 2021; Chen et al., 2022), we use the transferred uniformity loss as an energy function underlying our proposed normalization layer, such that descent steps along it corresponds with the forward pass.

Original: By analyzing the unfolding iterations of the principled uniformity loss, we design
Revised: By analyzing the unfolding iterations of the principled uniformity loss, we design

Original: As a proof of concept, Figure 1 demonstrates that ContraNorm makes the features away from each other, which eases the dimensional collapse.
Revised: As a proof of concept, Figure 1 demonstrates that ContraNorm makes the features away from each other, which eases the dimensional collapse.

Original: Theoretically, we prove that ContraNorm increases both average variance and effective rank of representations, thus solving complete collapse and dimensional collapse effectively.
Revised: Theoretically, we prove that ContraNorm increases both average variance and effective rank of representations, thus solving complete collapse and dimensional collapse effectively.

Original: We also conduct a comprehensive evaluation of ContraNorm on various tasks.
Revised: We also conduct a comprehensive evaluation of ContraNorm on various tasks.

Original: Specifically,
Revised: Specifically,

Original: ContraNorm boosts the average performance of BERT (Devlin et al., 2018) from 82.59% to 83.54% on the validation set of General
Revised: ContraNorm boosts the average performance of BERT (Devlin et al., 2018) from 82.59% to 83.54% on the validation set of General

Original: Language Understanding Evaluation (GLUE) datasets (Wang et al., 2019), and raises the test accuracy of DeiT (Touvron et al., 2021) withblocks from 77.69% to 78.67% on ImageNet1K (Russakovsky et al., 2015) dataset.
Revised: Language Understanding Evaluation (GLUE) datasets (Wang et al., 2019), and raises the test accuracy of DeiT (Touvron et al., 2021) withblocks from 77.69% to 78.67% on ImageNet1K (Russakovsky et al., 2015) dataset.

Original: For GNNs, experiments are conducted on fully supervised graph node classification tasks, and our proposed model outperforms vanilla Graph Convolution Network (GCN) (Kipf & Welling, 2017) on all depth settings.
Revised: For GNNs, experiments are conducted on fully supervised graph node classification tasks, and our proposed model outperforms vanilla Graph Convolution Network (GCN) (Kipf & Welling, 2017) on all depth settings.

Original: Our contributions are summarized as:
Revised: Our contributions are summarized as:

Original: • We dissect the limitations of existing oversmoothing analysis, and highlight the importance of incorporating the dimensional collapse issue into consideration.
Revised: • We dissect the limitations of existing oversmoothing analysis, and highlight the importance of incorporating the dimensional collapse issue into consideration.

Original: • Inspired by the techniques from contrastive learning to measure and resolve oversmoothing, we propose ContraNorm as an optimization-induced normalization layer to prevent dimensional collapse.
Revised: • Inspired by the techniques from contrastive learning to measure and resolve oversmoothing, we propose ContraNorm as an optimization-induced normalization layer to prevent dimensional collapse.

Original: • Experiments on a wide range of tasks show that ContraNorm can effectively mitigate dimensional collapse in various model variants, and demonstrate clear benefits across three different scenarios: ViT for image classification, BERT for natural language understanding, and GNNs for node classifications.
Revised: • Experiments on a wide range of tasks show that ContraNorm can effectively mitigate dimensional collapse in various model variants, and demonstrate clear benefits across three different scenarios: ViT for image classification, BERT for natural language understanding, and GNNs for node classifications.

Original:  B ACKGROUND & R ELATED W ORK
Revised:  B ACKGROUND & R ELATED W ORK

Original: Message Passing in Graph Neural Networks.
Revised: Message Passing in Graph Neural Networks.

Original: In the literature of GNNs, message-passing graph neural networks (MP-GNNs) are intensively studied.
Revised: In the literature of GNNs, message-passing graph neural networks (MP-GNNs) are intensively studied.

Original: It progressively updates representations by exchanging information with neighbors.
Revised: It progressively updates representations by exchanging information with neighbors.

Original: The update of node i ’s representation in l -th layer is formalized as h ( l ) i = UPDATE (cid:0) h ( l − 1) , AGGREGATE ( h ( l − 1) i , { h ( l − 1)
Revised: The update of node i ’s representation in l -th layer is formalized as h ( l ) i = UPDATE (cid:0) h ( l − 1) , AGGREGATE ( h ( l − 1) i , { h ( l − 1)

Original: j | j ∈ N ( i ) } ) (cid:1) , where N ( i ) denotes the neighborhood set of node i , AGGREGATE ( · ) is the procedure where nodes exchange message, and UPDATE ( · ) is often a multi-layer perceptron (MLP).
Revised: j | j ∈ N ( i ) } ) (cid:1) , where N ( i ) denotes the neighborhood set of node i , AGGREGATE ( · ) is the procedure where nodes exchange message, and UPDATE ( · ) is often a multi-layer perceptron (MLP).

Original: A classical MP-GNNs model is GCN (Kipf & Welling, 2017), which propagates messages between 1-hop neighbors using an adjacency matrix.
Revised: A classical MP-GNNs model is GCN (Kipf & Welling, 2017), which propagates messages between 1-hop neighbors using an adjacency matrix.

Original: Self-Attention in Transformers.
Revised: Self-Attention in Transformers.

Original: Transformers encode information in a global scheme with selfattention as the key ingredient (Vaswani et al diate representations by aggregating semantically near neighbors.
Revised: Transformers encode information in a global scheme with selfattention as the key ingredient (Vaswani et al diate representations by aggregating semantically near neighbors.

Original: Formally, it estimates similarities between key and query, namely self-attention matrix, as ¯ A = softmax ( QK ⊤ ) , Q = XW Q , and K = XW K where X , W Q , and W K are the input, weight matrices for query and key, respectively.
Revised: Formally, it estimates similarities between key and query, namely self-attention matrix, as ¯ A = softmax ( QK ⊤ ) , Q = XW Q , and K = XW K where X , W Q , and W K are the input, weight matrices for query and key, respectively.

Original: A multi-head self-attention module with a residual connection can be formularized as attn ( X ) =
Revised: A multi-head self-attention module with a residual connection can be formularized as attn ( X ) =

Original: X + (cid:80) hk
Revised: X + (cid:80) hk

Original: =1 ¯ A k XV k W ⊤ k , where h is the number of heads, and V , W are weights for value and final output.
Revised: =1 ¯ A k XV k W ⊤ k , where h is the number of heads, and V , W are weights for value and final output.

Original: Connections between Message Passing and Self-Attention.
Revised: Connections between Message Passing and Self-Attention.

Original: Note that the self-attention matrix can be seen as a normalized adjacent matrix of a corresponding graph (Shi et al., 2022).
Revised: Note that the self-attention matrix can be seen as a normalized adjacent matrix of a corresponding graph (Shi et al., 2022).

Original: Considering a weighted fully connected graph G with adjacency matrix denoted as ˆ A , we map nodes to token representations and set weights of the edge between node i and node j to exp( Q ⊤ i K j ) .
Revised: Considering a weighted fully connected graph G with adjacency matrix denoted as ˆ A , we map nodes to token representations and set weights of the edge between node i and node j to exp( Q ⊤ i K j ) .

Original: Then ( i, j ) entry of a normalized adjacency matrix is ˜
Revised: Then ( i, j ) entry of a normalized adjacency matrix is ˜

Original: A ij =
Revised: A ij =

Original: ˆ A ij / ˆ D ij = exp( Q ⊤ i K j ) / (cid:80) k exp( Q ⊤ i K k ) , where diagonal matrix ˆ D ii =
Revised: ˆ A ij / ˆ D ij = exp( Q ⊤ i K j ) / (cid:80) k exp( Q ⊤ i K k ) , where diagonal matrix ˆ D ii =

Original: (cid:80)
Revised: (cid:80)

Original: j ˆ A ij .
Revised: j ˆ A ij .

Original: Apparently, ˜ A is equal to the form of self-attention matrix defined in Transformers.
Revised: Apparently, ˜ A is equal to the form of self-attention matrix defined in Transformers.

Original: Simultaneously, ˜ A plays a major part in the message-passing scheme by deciding which nodes to exchange information with.
Revised: Simultaneously, ˜ A plays a major part in the message-passing scheme by deciding which nodes to exchange information with.

Original: Oversmoothing in GNNs and Transformers.
Revised: Oversmoothing in GNNs and Transformers.

Original: The term oversmoothing is firstly proposed by Li et al.
Revised: The term oversmoothing is firstly proposed by Li et al.

Original: (2018) in research of GNNs.
Revised: (2018) in research of GNNs.

Original: Intuitively, representations converge to a constant after repeatedly exchanging messages with neighbors when the layer goes to infinity.
Revised: Intuitively, representations converge to a constant after repeatedly exchanging messages with neighbors when the layer goes to infinity.

Original: Zhou et al.
Revised: Zhou et al.

Original: (2020) mathematically proves that, under some conditions, the convergence point carries only information of the graph topology.
Revised: (2020) mathematically proves that, under some conditions, the convergence point carries only information of the graph topology.

Original: Coincidentally, an oversmoothing-like phenomenon is also observed in Transformers.
Revised: Coincidentally, an oversmoothing-like phenomenon is also observed in Transformers.

Original: Unlike CNNs, Transformers can not benefit from simply deepening layers, and even saturates with depth increasing.
Revised: Unlike CNNs, Transformers can not benefit from simply deepening layers, and even saturates with depth increasing.

Original: Early works empirically ascribe it to attention/feature collapse or patch/token uniformity (Tang et al., 2021;
Revised: Early works empirically ascribe it to attention/feature collapse or patch/token uniformity (Tang et al., 2021;

Original: Zhou et al., 2021;
Revised: Zhou et al., 2021;

Original: Gong et al., 2021; Yan et al., 2022).
Revised: Gong et al., 2021; Yan et al., 2022).

Original: To be specific, the attention maps tend to be overly similar in later layers, whereby features insufficiently exchange information and lose diversity in the end.
Revised: To be specific, the attention maps tend to be overly similar in later layers, whereby features insufficiently exchange information and lose diversity in the end.

Original: Outputs of pure transformers, i.e., attention without skip connections or MLPs, are also observed to converge to a rank-1 matrix (Dong et al., 2021).
Revised: Outputs of pure transformers, i.e., attention without skip connections or MLPs, are also observed to converge to a rank-1 matrix (Dong et al., 2021).

Original: For illustration proposes, we also refer to the degeneration issue in Transformers as oversmoothing.
Revised: For illustration proposes, we also refer to the degeneration issue in Transformers as oversmoothing.

Original: Whitening Methods for Dimensional Collapse.
Revised: Whitening Methods for Dimensional Collapse.

Original: Whitening methods ensure that the covariance matrix of normalized outputs is diagonal, making dimensions mutually independent and implicitly solving dimensional collapse.
Revised: Whitening methods ensure that the covariance matrix of normalized outputs is diagonal, making dimensions mutually independent and implicitly solving dimensional collapse.

Original: Huang et al.
Revised: Huang et al.

Original: (2018; 2019); Siarohin et al.
Revised: (2018; 2019); Siarohin et al.

Original: ; Ermolov et al.
Revised: ; Ermolov et al.

Original: (2021) all adopt the idea of whitening, but differ in calculating details of whitening matrix and application domain.
Revised: (2021) all adopt the idea of whitening, but differ in calculating details of whitening matrix and application domain.

Original: Compared with them, we avoid complicated calculations on the squared root of the inverse covariance matrix and delicate design of backward pass for differentiability.
Revised: Compared with them, we avoid complicated calculations on the squared root of the inverse covariance matrix and delicate design of backward pass for differentiability.

Original: Moreover,
Revised: Moreover,

Original: Huang et al.
Revised: Huang et al.

Original: (2018; 2019) are proposed for convolution operations, Siarohin et al.
Revised: (2018; 2019) are proposed for convolution operations, Siarohin et al.

Original: (2018) is for GAN, and Ermolov et al.
Revised: (2018) is for GAN, and Ermolov et al.

Original: (2021) works in self-supervised learning.
Revised: (2021) works in self-supervised learning.

Original: In contrast, we borrow the idea from contrastive learning and solve oversmoothing in completely different fields like Transformers and
Revised: In contrast, we borrow the idea from contrastive learning and solve oversmoothing in completely different fields like Transformers and

Original: GNNs.
Revised: GNNs.

Original:  M ITIGATING O VERSMOOTHING FROM THE P ERSPECTIVE OF C ONTRASTIVE L EARNING
Revised:  M ITIGATING O VERSMOOTHING FROM THE P ERSPECTIVE OF C ONTRASTIVE L EARNING

Original: In this section, we first empirically demonstrate that current similarity metric fails to characterize dimensional collapse, thus overlooking a crucial part of oversmoothing.
Revised: In this section, we first empirically demonstrate that current similarity metric fails to characterize dimensional collapse, thus overlooking a crucial part of oversmoothing.

Original: To address this problem, we draw inspiration from contrastive learning whose uniformity property naturally rules out dimensional collapse.
Revised: To address this problem, we draw inspiration from contrastive learning whose uniformity property naturally rules out dimensional collapse.

Original: Specifically, we transfer the uniformity loss to a loss that directly acts on representations.
Revised: Specifically, we transfer the uniformity loss to a loss that directly acts on representations.

Original: By unfolding the optimization steps along this loss, we induce a normalization layer called
Revised: By unfolding the optimization steps along this loss, we induce a normalization layer called

Original: ContraNorm.
Revised: ContraNorm.

Original: Theoretically, we prove our proposed layer helps mitigate dimensional collapse.
Revised: Theoretically, we prove our proposed layer helps mitigate dimensional collapse.

Original: 3. T HE C HARACTERIZATION OF O VERSMOOTHING In
Revised: 3. T HE C HARACTERIZATION OF O VERSMOOTHING In

Original: this part, we begin by highlighting the limitations of existing metrics in characterizing oversmoothing.
Revised: this part, we begin by highlighting the limitations of existing metrics in characterizing oversmoothing.

Original: These limitations motivate us to adopt the effective rank metric, which has been demonstrated to be effective in capturing the degree of dimensional collapse in contrastive learning.
Revised: These limitations motivate us to adopt the effective rank metric, which has been demonstrated to be effective in capturing the degree of dimensional collapse in contrastive learning.

Original: Taking the oversmoothing problem of Transformers as an example without loss of generality, a prevailing metric is evaluating attention map similarity (Wang et al., 2022;
Revised: Taking the oversmoothing problem of Transformers as an example without loss of generality, a prevailing metric is evaluating attention map similarity (Wang et al., 2022;

Original: Gong et al., 2021; Shi et al., 2022).
Revised: Gong et al., 2021; Shi et al., 2022).

Original: The intuition is that as attention map similarity converges to one, feature similarity increases, which can result in a loss of representation expressiveness and decreased performance.
Revised: The intuition is that as attention map similarity converges to one, feature similarity increases, which can result in a loss of representation expressiveness and decreased performance.

Original: However, by conducting experiments on transformer structured models like ViT and BERT, we find
Revised: However, by conducting experiments on transformer structured models like ViT and BERT, we find

Original: 8 10 layer index 0.
Revised: 8 10 layer index 0.

Original: c o s i n e s i m il a r i t y featureattention map (a) Similarity on BERT    8 10 layer index 0.
Revised: c o s i n e s i m il a r i t y featureattention map (a) Similarity on BERT    8 10 layer index 0.

Original: c o s i n e s i m il a r i t y featureattention map
Revised: c o s i n e s i m il a r i t y featureattention map

Original: [Figure]
Revised: [Figure]

Original: 0 20 40 60 80 100 120 140 singular value103050 nu m b e r
Revised: 0 20 40 60 80 100 120 140 singular value103050 nu m b e r

Original: BERT+ ContraNorm
Revised: BERT+ ContraNorm

Original: [Figure]
Revised: [Figure]

Original: Distribution    8 10 layer index    e ff e c t
Revised: Distribution    8 10 layer index    e ff e c t

Original: i v e r a n k
Revised: i v e r a n k

Original: No norm ContraNorm
Revised: No norm ContraNorm

Original: [Figure]
Revised: [Figure]

Original: that high attention map similarity does not necessarily correspond to high feature similarity.
Revised: that high attention map similarity does not necessarily correspond to high feature similarity.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2a and Fig.
Revised: 2a and Fig.

Original: 2b, although attention map similarity is close to 0.8 or even higher, the feature similarity remains below 0.5 in most cases.
Revised: 2b, although attention map similarity is close to 0.8 or even higher, the feature similarity remains below 0.5 in most cases.

Original: It means the oversmoothing problem still occurs even with a low feature similarity.
Revised: It means the oversmoothing problem still occurs even with a low feature similarity.

Original: This finding suggests that similarity can not fully depict the quality of representations and the oversmoothing problem.
Revised: This finding suggests that similarity can not fully depict the quality of representations and the oversmoothing problem.

Original: Intuitively, we can consider a case that representations in the latent embedding space do not shrink to one single point but span a low dimensional space.
Revised: Intuitively, we can consider a case that representations in the latent embedding space do not shrink to one single point but span a low dimensional space.

Original: In such cases, feature similarity may be relatively low, but representations still lose expressive power.
Revised: In such cases, feature similarity may be relatively low, but representations still lose expressive power.

Original: Such representation degeneration problem is known as dimensional collapse , and it is widely discussed in the literature of contrastive learning.
Revised: Such representation degeneration problem is known as dimensional collapse , and it is widely discussed in the literature of contrastive learning.

Original: In contrastive learning, common practice to describe dimensional collapse is the vanishing distribution of singular values (Gao et al., 2019; Ethayarajh, 2019;
Revised: In contrastive learning, common practice to describe dimensional collapse is the vanishing distribution of singular values (Gao et al., 2019; Ethayarajh, 2019;

Original: Jing et al., 2022).
Revised: Jing et al., 2022).

Original: To investigate whether dimensional collapse occurs in Transformers, we draw the singular value distribution of features in the last block of 12-layer BERT.
Revised: To investigate whether dimensional collapse occurs in Transformers, we draw the singular value distribution of features in the last block of 12-layer BERT.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2c, the insignificant (nearly zero) values dominate singular value distribution in the deep layer of vanilla BERT, indicating that representations reside on a low-dimensional manifold and dimensional collapse happens.
Revised: 2c, the insignificant (nearly zero) values dominate singular value distribution in the deep layer of vanilla BERT, indicating that representations reside on a low-dimensional manifold and dimensional collapse happens.

Original: To show the collapse tendency along layer index, we simplify the singular value distribution to a concise scalar effective rank (erank) (Roy & Vetterli, 2007), which covers the full singular value spectrum.
Revised: To show the collapse tendency along layer index, we simplify the singular value distribution to a concise scalar effective rank (erank) (Roy & Vetterli, 2007), which covers the full singular value spectrum.

Original: Definition 3.1 (Effective Rank) .
Revised: Definition 3.1 (Effective Rank) .

Original: Considering matrix X ∈ R m × n whose singular value decomposition is given by X = U Σ V , where Σ is a diagonal matrix with singular values σ 1 ≥ σ 2 ≥ · · · ≥ σ Q ≥ 0 with Q = min { m, n } .
Revised: Considering matrix X ∈ R m × n whose singular value decomposition is given by X = U Σ V , where Σ is a diagonal matrix with singular values σ 1 ≥ σ 2 ≥ · · · ≥ σ Q ≥ 0 with Q = min { m, n } .

Original: The distribution of singular values is defined as L 1 -normalized form p i
Revised: The distribution of singular values is defined as L 1 -normalized form p i

Original: =
Revised: =

Original: σ i / (cid:80)
Revised: σ i / (cid:80)

Original: Qk =1 | σ k | .
Revised: Qk =1 | σ k | .

Original: The effective rank of the matrix X , denoted as erank ( X ) , is defined as erank ( X ) =
Revised: The effective rank of the matrix X , denoted as erank ( X ) , is defined as erank ( X ) =

Original: exp( H { p 1 , p 2 , · · · , p Q } ) , where H ( p 1 , p 2 , · · · , p Q ) is the Shannon entropy given by
Revised: exp( H { p 1 , p 2 , · · · , p Q } ) , where H ( p 1 , p 2 , · · · , p Q ) is the Shannon entropy given by

Original: H ( p 1 , p 2 , · · · , p Q ) = − (cid:80)
Revised: H ( p 1 , p 2 , · · · , p Q ) = − (cid:80)

Original: Qk =1 p k log p k .
Revised: Qk =1 p k log p k .

Original: Based on erank, we revisit the oversmoothing issue of Transformers in Fig.
Revised: Based on erank, we revisit the oversmoothing issue of Transformers in Fig.

Original: 2d.
Revised: 2d.

Original: We can see that the effective rank descends along with layer index, indicating an increasingly imbalanced singular value distribution in deeper layers.
Revised: We can see that the effective rank descends along with layer index, indicating an increasingly imbalanced singular value distribution in deeper layers.

Original: This finding not only verifies dimensional collapse does occur in Transformers, but also indicates the effectiveness of using effective rank as a metric to detect this issue.
Revised: This finding not only verifies dimensional collapse does occur in Transformers, but also indicates the effectiveness of using effective rank as a metric to detect this issue.

Original: 3. I NSPIRATIONS FROM THE U NIFORMITY L OSS IN C ONTRASTIVE L EARNING
Revised: 3. I NSPIRATIONS FROM THE U NIFORMITY L OSS IN C ONTRASTIVE L EARNING

Original: The core idea of contrastive learning is maximizing agreement between augmented views of the same example (i.e., positive pairs) and disagreement of views from different samples (i.e. negative pairs).
Revised: The core idea of contrastive learning is maximizing agreement between augmented views of the same example (i.e., positive pairs) and disagreement of views from different samples (i.e. negative pairs).

Original: A popular form of contrastive learning optimizes feature representations using a loss function with limited negative samples (Chen et al., 2020).
Revised: A popular form of contrastive learning optimizes feature representations using a loss function with limited negative samples (Chen et al., 2020).

Original: Concretely, given a batch of randomly sampled examples, for each example we generate its augmented positive views and finally get a total of N samples.
Revised: Concretely, given a batch of randomly sampled examples, for each example we generate its augmented positive views and finally get a total of N samples.

Original: Considering an encoder function f , the contrastive loss for the positive pair ( i, i + ) is
Revised: Considering an encoder function f , the contrastive loss for the positive pair ( i, i + ) is

Original: [Equation]
Revised: [Equation]

Original: where τ denotes a temperature factor.
Revised: where τ denotes a temperature factor.

Original: The loss can be decoupled into two parts named alignment loss and uniformity loss :
Revised: The loss can be decoupled into two parts named alignment loss and uniformity loss :

Original: [Equation]
Revised: [Equation]

Original: The alignment loss encourages feature representations for positive pairs to be similar, thus being invariant to unnecessary noises.
Revised: The alignment loss encourages feature representations for positive pairs to be similar, thus being invariant to unnecessary noises.

Original: However, training with only the alignment loss will result in a trivial solution where all representations are identical.
Revised: However, training with only the alignment loss will result in a trivial solution where all representations are identical.

Original: In other words, complete collapse happens.
Revised: In other words, complete collapse happens.

Original: While batch normalization (Ioffe & Szegedy, 2015) can help avoid this issue, it cannot fully prevent the problem of dimensional collapse, which still negatively impacts learning (Hua et al., 2021).
Revised: While batch normalization (Ioffe & Szegedy, 2015) can help avoid this issue, it cannot fully prevent the problem of dimensional collapse, which still negatively impacts learning (Hua et al., 2021).

Original: Thanks to the property of uniformity loss, dimensional collapse can be solved effectively.
Revised: Thanks to the property of uniformity loss, dimensional collapse can be solved effectively.

Original: Reviewing the form of uniformity loss in Eq.(2), it maximizes average distances between all samples, resulting in embeddings that are roughly uniformly distributed in the latent space, and thus more information is preserved.
Revised: Reviewing the form of uniformity loss in Eq.(2), it maximizes average distances between all samples, resulting in embeddings that are roughly uniformly distributed in the latent space, and thus more information is preserved.

Original: The inclusion of the uniformity loss in the training process helps to alleviate dimensional collapse.
Revised: The inclusion of the uniformity loss in the training process helps to alleviate dimensional collapse.

Original: Intuitively, it can also serve as a sharp tool for alleviating oversmoothing in models such as GNNs and Transformers.
Revised: Intuitively, it can also serve as a sharp tool for alleviating oversmoothing in models such as GNNs and Transformers.

Original: An alternative approach to address oversmoothing is to directly incorporate the uniformity loss into the training objective.
Revised: An alternative approach to address oversmoothing is to directly incorporate the uniformity loss into the training objective.

Original: However, our experiments reveal that this method has limited effectiveness (see Appendix D for more details).
Revised: However, our experiments reveal that this method has limited effectiveness (see Appendix D for more details).

Original: Instead, we propose a normalization layer that can be easily integrated into various models.
Revised: Instead, we propose a normalization layer that can be easily integrated into various models.

Original: Our approach utilizes the uniformity loss as an underlying energy function of the proposed layer, such that a descent step along the energy function corresponds to a forward pass of the layer.
Revised: Our approach utilizes the uniformity loss as an underlying energy function of the proposed layer, such that a descent step along the energy function corresponds to a forward pass of the layer.

Original: Alternatively, we can view the layer as the unfolded iterations of an optimization function.
Revised: Alternatively, we can view the layer as the unfolded iterations of an optimization function.

Original: This perspective is adopted to elucidate GNNs (Yang et al., 2021; Zhu et al., 2021), Transformers (Yang et al., 2022), and classic MLPs (Xie et al., 2021).
Revised: This perspective is adopted to elucidate GNNs (Yang et al., 2021; Zhu et al., 2021), Transformers (Yang et al., 2022), and classic MLPs (Xie et al., 2021).

Original: Note that the uniformity loss works by optimizing model parameters while what the normalization layer directly updates is representation itself.
Revised: Note that the uniformity loss works by optimizing model parameters while what the normalization layer directly updates is representation itself.

Original: So we first transfer the uniformity loss, which serves as a training loss, to a kind of architecture loss.
Revised: So we first transfer the uniformity loss, which serves as a training loss, to a kind of architecture loss.

Original: Consider a fully connected graph with restricted number of nodes, where node h i is viewed as representation of a random sample x i .
Revised: Consider a fully connected graph with restricted number of nodes, where node h i is viewed as representation of a random sample x i .

Original: Reminiscent of L uniform , we define ˆ L uniform over all nodes in the graph as
Revised: Reminiscent of L uniform , we define ˆ L uniform over all nodes in the graph as

Original: [Equation]
Revised: [Equation]

Original: This form of uniformity loss is defined directly on representations, and we later use it as the underlying energy function for representation update.
Revised: This form of uniformity loss is defined directly on representations, and we later use it as the underlying energy function for representation update.

Original: 3. T HE P ROPOSED C ONTRA N ORM
Revised: 3. T HE P ROPOSED C ONTRA N ORM

Original: Till now, we are able to build a connection between layer design and the unfolded iterations of descent steps used to minimize uniformity loss.
Revised: Till now, we are able to build a connection between layer design and the unfolded iterations of descent steps used to minimize uniformity loss.

Original: Specifically, we take the derivative of ˆ L uniform on node representations:
Revised: Specifically, we take the derivative of ˆ L uniform on node representations:

Original: [Equation]
Revised: [Equation]

Original: by denoting feature matrix as H with h ⊤ i as the i -th row, we rewrite Eq.(4) into a matrix form for simplicity:
Revised: by denoting feature matrix as H with h ⊤ i as the i -th row, we rewrite Eq.(4) into a matrix form for simplicity:

Original: [Equation]
Revised: [Equation]

Original: where A = exp( HH ⊤ /τ ) and D = deg ( A ) 1 .
Revised: where A = exp( HH ⊤ /τ ) and D = deg ( A ) 1 .

Original: In order to reduce the uniformity loss ˆ L uniform , a natural way is to take a single step performing gradient descent, which is to update representations
Revised: In order to reduce the uniformity loss ˆ L uniform , a natural way is to take a single step performing gradient descent, which is to update representations

Original: H in the way of
Revised: H in the way of

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: where H b and H t denote the representations before and after the update, respectively, and s is the step size of the gradient descent.
Revised: where H b and H t denote the representations before and after the update, respectively, and s is the step size of the gradient descent.

Original: By taking this update after a certain representation layer, we can reduce the uniformity loss of the representations and thus help ease the dimensional collapse.
Revised: By taking this update after a certain representation layer, we can reduce the uniformity loss of the representations and thus help ease the dimensional collapse.

Original: In Eq.(6), there exist two terms D − 1 A and AD − 1 multiplied with H b .
Revised: In Eq.(6), there exist two terms D − 1 A and AD − 1 multiplied with H b .

Original: Empirically, the two terms play a similar role in our method.
Revised: Empirically, the two terms play a similar role in our method.

Original: Note that the first term is related to self-attention matrix in
Revised: Note that the first term is related to self-attention matrix in

Original: Transformers, so we only preserve it and discard the second one.
Revised: Transformers, so we only preserve it and discard the second one.

Original: Then Eq.(6) becomes
Revised: Then Eq.(6) becomes

Original: [Equation]
Revised: [Equation]

Original: In fact, the operation corresponds to the stop-gradient technique, which is widely used in contrastive learning methods (He et al., 2020; Grill et al., 2020; Tao et al., 2022).
Revised: In fact, the operation corresponds to the stop-gradient technique, which is widely used in contrastive learning methods (He et al., 2020; Grill et al., 2020; Tao et al., 2022).

Original: By throwing away some terms in the gradient, stop-gradient makes the training process asymmetric and thus avoids representation collapse with less computational overhead, which is discussed in detail in Appendix A.
Revised: By throwing away some terms in the gradient, stop-gradient makes the training process asymmetric and thus avoids representation collapse with less computational overhead, which is discussed in detail in Appendix A.

Original: However, the layer induced by Eq.(7) still can not ensure uniformity on representations.
Revised: However, the layer induced by Eq.(7) still can not ensure uniformity on representations.

Original: Consider an extreme case where softmax( H b H ⊤ b ) is equal to identity matrix I . Eq.(7) becomes H t =
Revised: Consider an extreme case where softmax( H b H ⊤ b ) is equal to identity matrix I . Eq.(7) becomes H t =

Original: H b − s/
Revised: H b − s/

Original: τ × H b
Revised: τ × H b

Original: =
Revised: =

Original: (1 − s/τ ) H b , which just makes the scale of H smaller and does not help alleviate the complete collapse.
Revised: (1 − s/τ ) H b , which just makes the scale of H smaller and does not help alleviate the complete collapse.

Original: To keep the representation stable, we propose two methods:
Revised: To keep the representation stable, we propose two methods:

Original: Regularization.
Revised: Regularization.

Original: We add a regularization term − 12 (cid:80) i ∥ h i ∥ 2 2 to the uniformity loss.
Revised: We add a regularization term − 12 (cid:80) i ∥ h i ∥ 2 2 to the uniformity loss.

Original: When the regularization term becomes smaller, the norm of h i becomes larger.
Revised: When the regularization term becomes smaller, the norm of h i becomes larger.

Original: Therefore, adding this term can help prevent the norm of representation h i from becoming smaller.
Revised: Therefore, adding this term can help prevent the norm of representation h i from becoming smaller.

Original: In this way, the update form becomes
Revised: In this way, the update form becomes

Original: [Equation]
Revised: [Equation]

Original: Proposition 1.
Revised: Proposition 1.

Original: Let e =
Revised: Let e =

Original: . , 1) ⊤ / √ n .
Revised: . , 1) ⊤ / √ n .

Original: For attention matrix ¯ A = softmax( H b H ⊤ b ) , let σ min be the smallest eigenvalue of P =
Revised: For attention matrix ¯ A = softmax( H b H ⊤ b ) , let σ min be the smallest eigenvalue of P =

Original: ( I − ee ⊤ )( I − ¯ A )
Revised: ( I − ee ⊤ )( I − ¯ A )

Original: +( I − ¯ A ) ⊤ ( I − ee ⊤ ) .
Revised: +( I − ¯ A ) ⊤ ( I − ee ⊤ ) .

Original: For the update in Eq.8, i.e. H t =
Revised: For the update in Eq.8, i.e. H t =

Original: ((1 + s )
Revised: ((1 + s )

Original: I − s ¯ A ) H b , s > 0 , we have V ar ( H t ) ≥ (1 + sσ min ) · V ar
Revised: I − s ¯ A ) H b , s > 0 , we have V ar ( H t ) ≥ (1 + sσ min ) · V ar

Original: H b ) .
Revised: H b ) .

Original: Especially, if σ min ≥ 0 , we have V ar ( H t ) ≥
Revised: Especially, if σ min ≥ 0 , we have V ar ( H t ) ≥

Original: V ar ( H b ) .
Revised: V ar ( H b ) .

Original: Proposition 1 gives a bound for the ratio of the variance after and before the update.
Revised: Proposition 1 gives a bound for the ratio of the variance after and before the update.

Original: It shows that the change of the variance is influenced by the symmetric matrix P
Revised: It shows that the change of the variance is influenced by the symmetric matrix P

Original: = ( I − ee ⊤ )( I − ¯ A )
Revised: = ( I − ee ⊤ )( I − ¯ A )

Original: +( I − ¯ A ) ⊤ ( I − ee ⊤ ) .
Revised: +( I − ¯ A ) ⊤ ( I − ee ⊤ ) .

Original: If P is a semi-positive definite matrix, we will get the result that V ar ( H t ) ≥
Revised: If P is a semi-positive definite matrix, we will get the result that V ar ( H t ) ≥

Original: V ar ( H b ) , which indicates that the representations become more uniform.
Revised: V ar ( H b ) , which indicates that the representations become more uniform.

Original: In Appendix B, we will give out some sufficient conditions for that P is semi-positive definite.
Revised: In Appendix B, we will give out some sufficient conditions for that P is semi-positive definite.

Original: LayerNorm.
Revised: LayerNorm.

Original: We leverage layer normalization (Ba et al., 2016), which adjusts the representation according to its mean and variance.
Revised: We leverage layer normalization (Ba et al., 2016), which adjusts the representation according to its mean and variance.

Original: The update form of the layer normalization on representation h
Revised: The update form of the layer normalization on representation h

Original: i is LayerNorm ( h i )
Revised: i is LayerNorm ( h i )

Original: = γ · (( h i − mean ( h i )) / (cid:112) Var ( h i ) + ε ) + β , where γ and β are learnable parameters and ε = 10 − 5 prevents the denominator from becoming zero.
Revised: = γ · (( h i − mean ( h i )) / (cid:112) Var ( h i ) + ε ) + β , where γ and β are learnable parameters and ε = 10 − 5 prevents the denominator from becoming zero.

Original: The learnable parameters γ and β can rescale the representation h i to help ease the problem.
Revised: The learnable parameters γ and β can rescale the representation h i to help ease the problem.

Original: We append the layer normalization to Eq.(7) as
Revised: We append the layer normalization to Eq.(7) as

Original: [Equation]
Revised: [Equation]

Original: where applying the layer normalization to a representation matrix H means applying the layer normalization to all its components h 1 , . .
Revised: where applying the layer normalization to a representation matrix H means applying the layer normalization to all its components h 1 , . .

Original: , h n .
Revised: , h n .

Original: We empirically compare the two proposed methods and find their performance comparable, while the second one performs slightly better.
Revised: We empirically compare the two proposed methods and find their performance comparable, while the second one performs slightly better.

Original: Therefore, we adopt the second update form and name it Contrastive Normalization ( ContraNorm ).
Revised: Therefore, we adopt the second update form and name it Contrastive Normalization ( ContraNorm ).

Original: The ContraNorm layer can be added after any representation layer to reduce the uniformity loss and help relieve the dimensional collapse.
Revised: The ContraNorm layer can be added after any representation layer to reduce the uniformity loss and help relieve the dimensional collapse.

Original: We discuss the best place to plug our ContraNorm in Appendix E.
Revised: We discuss the best place to plug our ContraNorm in Appendix E.

Original: Given H b ∈ R n × d where n and d is the number of samples in the batch and hidden dimension, the time complexity of ContraNorm is O ( n 2 d ) , which is the same order as the self-attention operation in Transformer.
Revised: Given H b ∈ R n × d where n and d is the number of samples in the batch and hidden dimension, the time complexity of ContraNorm is O ( n 2 d ) , which is the same order as the self-attention operation in Transformer.

Original: For enhancing scalability, we propose a modified version of ContraNorm with a linear complexity in the number of samples.
Revised: For enhancing scalability, we propose a modified version of ContraNorm with a linear complexity in the number of samples.

Original: See Appendix F for more details.
Revised: See Appendix F for more details.

Original: 3. T HEORETICAL A NALYSIS
Revised: 3. T HEORETICAL A NALYSIS

Original: In this part, we will give a theoretical result on our proposed ContraNorm method.
Revised: In this part, we will give a theoretical result on our proposed ContraNorm method.

Original: We will show that with a different form of the uniformity loss, the ContraNorm update can help alleviate the dimensional collapse.
Revised: We will show that with a different form of the uniformity loss, the ContraNorm update can help alleviate the dimensional collapse.

Original: Proposition 2.
Revised: Proposition 2.

Original: Consider the update form
Revised: Consider the update form

Original: [Equation]
Revised: [Equation]

Original: let σ max be the largest singular value of H b .
Revised: let σ max be the largest singular value of H b .

Original: For s > 0 satisfying 1 + (1 − σ 2max ) s > 0 , we have erank( H t ) >
Revised: For s > 0 satisfying 1 + (1 − σ 2max ) s > 0 , we have erank( H t ) >

Original: erank( H b ) .
Revised: erank( H b ) .

Original: Proposition 2 gives a promise of alleviating dimensional collapse under a special update form.
Revised: Proposition 2 gives a promise of alleviating dimensional collapse under a special update form.

Original: Denote ˆ L dim = tr (( I − HH ⊤ ) 2 ) / 4 , then ˆ L dim shares a similar form with Barlow Twins (Zbontar et al., 2021).
Revised: Denote ˆ L dim = tr (( I − HH ⊤ ) 2 ) / 4 , then ˆ L dim shares a similar form with Barlow Twins (Zbontar et al., 2021).

Original: This loss tries to equate the diagonal elements of the similarity matrix HH ⊤ to 1 and equate the off-diagonal elements of the matrix to 0, which drives different features becoming orthogonal, thus helping the features become more uniform in the space.
Revised: This loss tries to equate the diagonal elements of the similarity matrix HH ⊤ to 1 and equate the off-diagonal elements of the matrix to 0, which drives different features becoming orthogonal, thus helping the features become more uniform in the space.

Original: Note that ∂ ˆ L dim ∂ H =
Revised: Note that ∂ ˆ L dim ∂ H =

Original: ( I − HH ⊤ ) H , therefore, Eq.(10) can be rewritten as H t
Revised: ( I − HH ⊤ ) H , therefore, Eq.(10) can be rewritten as H t

Original: = H b
Revised: = H b

Original: + s ∂ ˆ L dim ( H b ) ∂ H b , which implies that this update form is in fact a ContraNorm with a different uniformity loss.
Revised: + s ∂ ˆ L dim ( H b ) ∂ H b , which implies that this update form is in fact a ContraNorm with a different uniformity loss.

Original: Propositionclaims that this update can increase the effective rank of the representation matrix, when s satisfies 1+(1 − σ max ) s > 0 .
Revised: Propositionclaims that this update can increase the effective rank of the representation matrix, when s satisfies 1+(1 − σ max ) s > 0 .

Original: Note that no matter whether σ max > 0 or not, if s is sufficiently close to 0 , the condition will be satisfied.
Revised: Note that no matter whether σ max > 0 or not, if s is sufficiently close to 0 , the condition will be satisfied.

Original: Under this situation, the update will alleviate the dimensional collapse.
Revised: Under this situation, the update will alleviate the dimensional collapse.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we demonstrate the effectiveness of ContraNorm by experiments including 1) language understanding tasks on GLUE datasets with BERT and ALBERT (Lan et al., 2020) as the backbones; 2) image classification task on ImageNet100 and ImageNet1k datasets with DeiT as the base model; 3) fully supervised graph node classification task on popular graph datasets with GCN as the backbone.
Revised: In this section, we demonstrate the effectiveness of ContraNorm by experiments including 1) language understanding tasks on GLUE datasets with BERT and ALBERT (Lan et al., 2020) as the backbones; 2) image classification task on ImageNet100 and ImageNet1k datasets with DeiT as the base model; 3) fully supervised graph node classification task on popular graph datasets with GCN as the backbone.

Original: Moreover, we conduct ablative studies on ContraNorm variants comparison and hyperparameters sensitivity analysis.
Revised: Moreover, we conduct ablative studies on ContraNorm variants comparison and hyperparameters sensitivity analysis.

Original: 4. E XPERIMENTS ON L ANGUAGE M ODELS
Revised: 4. E XPERIMENTS ON L ANGUAGE M ODELS

Original: To corroborate the potential of ContraNorm, we integrate it into two transformer structured models:
Revised: To corroborate the potential of ContraNorm, we integrate it into two transformer structured models:

Original: BERT and ALBERT, and evaluate it on GLUE datasets.
Revised: BERT and ALBERT, and evaluate it on GLUE datasets.

Original: GLUE includes three categories of tasks: (i) single-sentence tasks CoLA and SST-2; (ii) similarity and paraphrase tasks MRPC, QQP, and
Revised: GLUE includes three categories of tasks: (i) single-sentence tasks CoLA and SST-2; (ii) similarity and paraphrase tasks MRPC, QQP, and

Original: STS-B; (iii) inference tasks MNLI, QNLI, and RTE.
Revised: STS-B; (iii) inference tasks MNLI, QNLI, and RTE.

Original: For MNLI task, we experiment on both the matched (MNLI-m) and mismatched (MNLI-mm) versions.
Revised: For MNLI task, we experiment on both the matched (MNLI-m) and mismatched (MNLI-mm) versions.

Original: Setup.
Revised: Setup.

Original: We default plug ContraNorm after the self-attention module and residual connection (more position choices are in Appendix E).
Revised: We default plug ContraNorm after the self-attention module and residual connection (more position choices are in Appendix E).

Original: We use a batch size of 32 and fine-tune for 5 epochs over the data for all GLUE tasks.
Revised: We use a batch size of 32 and fine-tune for 5 epochs over the data for all GLUE tasks.

Original: For each task, we select the best scale factor s in Eq.(6) among (0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2) .
Revised: For each task, we select the best scale factor s in Eq.(6) among (0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2) .

Original: We use base models (BERT-base and ALBERT-base) of 12 stacked blocks with hyperparameters fixed for all tasks: number of hidden size 128, number of attention heads 12, maximum sequence length 384.
Revised: We use base models (BERT-base and ALBERT-base) of 12 stacked blocks with hyperparameters fixed for all tasks: number of hidden size 128, number of attention heads 12, maximum sequence length 384.

Original: We use Adam (Kingma & Ba, 2014) optimizer with the learning rate of 2 e − 5 .
Revised: We use Adam (Kingma & Ba, 2014) optimizer with the learning rate of 2 e − 5 .

Original: Results.
Revised: Results.

Original: As shown in Table 1, ContraNorm substantially improves results on all datasets compared with vanilla BERT.
Revised: As shown in Table 1, ContraNorm substantially improves results on all datasets compared with vanilla BERT.

Original: Specifically, our ContraNorm improves the previous average performance from 82.59% to 83.39% for BERT backbone and from 83.74% to 84.67% for ALBERT backbone.
Revised: Specifically, our ContraNorm improves the previous average performance from 82.59% to 83.39% for BERT backbone and from 83.74% to 84.67% for ALBERT backbone.

Original: We also submit our trained model to GLUE benchmark leaderboard and the results can be found in Appendix
Revised: We also submit our trained model to GLUE benchmark leaderboard and the results can be found in Appendix

Original: G. It is observed that BERT with ContraNorm also outperforms vanilla model across all datasets.
Revised: G. It is observed that BERT with ContraNorm also outperforms vanilla model across all datasets.

Original: To verify the de-oversmoothing effect of ContraNorm.
Revised: To verify the de-oversmoothing effect of ContraNorm.

Original: We build models with/without ContraNorm on various layer depth settings.
Revised: We build models with/without ContraNorm on various layer depth settings.

Original: The performance comparison is shown in Fig. 3a.
Revised: The performance comparison is shown in Fig. 3a.

Original: Constant stack of blocks causes obvious deterioration in vanilla BERT, while BERT with ContraNorm maintains competitive advantage.
Revised: Constant stack of blocks causes obvious deterioration in vanilla BERT, while BERT with ContraNorm maintains competitive advantage.

Original: Moreover, for deep models, we also show the tendency of variance and effective rank in Fig.
Revised: Moreover, for deep models, we also show the tendency of variance and effective rank in Fig.

Original: 3b and Fig.
Revised: 3b and Fig.

Original: 3c, which verifies the power of ContraNorm in alleviating complete collapse and dimensional collapse, respectively.
Revised: 3c, which verifies the power of ContraNorm in alleviating complete collapse and dimensional collapse, respectively.

Original: 4. E XPERIMENTS ON V ISION T RANSFORMERS
Revised: 4. E XPERIMENTS ON V ISION T RANSFORMERS

Original: We also validate effectiveness of ContraNorm in computer vision field.
Revised: We also validate effectiveness of ContraNorm in computer vision field.

Original: We choose DeiT as backbone and models are trained from scratch.
Revised: We choose DeiT as backbone and models are trained from scratch.

Original: Experiments with different depth settings are evaluated on ImageNet100 and ImageNet1k datasets.
Revised: Experiments with different depth settings are evaluated on ImageNet100 and ImageNet1k datasets.

Original: Based on the Timm (Wightman, 2019) and DeiT repositories, we insert ContraNorm into base model intermediately following self-attention module.
Revised: Based on the Timm (Wightman, 2019) and DeiT repositories, we insert ContraNorm into base model intermediately following self-attention module.

Original: Setup.
Revised: Setup.

Original: We follow the training recipe of Touvron et al.
Revised: We follow the training recipe of Touvron et al.

Original: (2021), and make minimal changes to hyperparameters.
Revised: (2021), and make minimal changes to hyperparameters.

Original: Specifically, we use AdamW (Loshchilov & Hutter, 2019) optimizer with cosine learning rate decay.
Revised: Specifically, we use AdamW (Loshchilov & Hutter, 2019) optimizer with cosine learning rate decay.

Original: We train each model for 150 epochs and the batch size is set to 1024.
Revised: We train each model for 150 epochs and the batch size is set to 1024.

Original: Aug-
Revised: Aug-

Original: [Figure]
Revised: [Figure]

Original: Fig.(3a) presents performances of models with block number=12, 24, 36, 48, 60, 72, 84. Fig.
Revised: Fig.(3a) presents performances of models with block number=12, 24, 36, 48, 60, 72. Fig.(3b) and

Original: 3b and Fig.(3c) show the tendency of features variance and effective rank for models with 72 blocks
Revised: Fig.(3c) show the tendency of features variance and effective rank for models with 72 blocks.

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: mentation techniques are used to boost the performance.
Revised: mentation techniques are used to boost the performance.

Original: For all experiments, the image size is set to be 224 × 224 .
Revised: For all experiments, the image size is set to be 224 × 224 .

Original: For Imagenet100, we set the scale factor s to 0.2 for all layer depth settings.
Revised: For Imagenet100, we set the scale factor s to 0.2 for all layer depth settings.

Original: For
Revised: For

Original: Imagenet1k, we set s to 0.2 for models with 12 and 16 blocks
Revised: Imagenet1k, we set s to 0.2 for models with 12 and 16 blocks

Original: [Table]
Revised: [Table]

Original: and raise it to 0.3 for models withblocks.
Revised: and raise it to 0.3 for models withblocks.

Original: Results.
Revised: Results.

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: DeiT with ContraNorm outperforms vanilla DeiT with all layer settings on both datasets.
Revised: DeiT with ContraNorm outperforms vanilla DeiT with all layer settings on both datasets.

Original: Typically
Revised: Typically

Original: [Table]
Revised: [Table]

Original: our method shows a gain of nearly 5% on test accuracy for ImageNet100.
Revised: our method shows a gain of nearly 5% on test accuracy for ImageNet100.

Original: For
Revised: For

Original: ImageNet1k, we boost the performance of DeiT with 24 blocks from 77.69% to 78.67%.
Revised: ImageNet1k, we boost the performance of DeiT with 24 blocks from 77.69% to 78.67%.

Original: 4. E XPERIMENTS ON G RAPH N EURAL N ETWORKS
Revised: 4. E XPERIMENTS ON G RAPH N EURAL N ETWORKS

Original: We implement ContraNorm as a simple normalization layer after each graph convolution of GCN, and evaluate it on fully supervised graph node classification task.
Revised: We implement ContraNorm as a simple normalization layer after each graph convolution of GCN, and evaluate it on fully supervised graph node classification task.

Original: For datasets, we choose two popular citation networks Cora (McCallum et al., 2000) and Citeseer (Giles et al., 1998), and
Revised: For datasets, we choose two popular citation networks Cora (McCallum et al., 2000) and Citeseer (Giles et al., 1998), and

Original: Wikipedia article networks Chameleon and Squirrel (Rozemberczki et al., 2021).
Revised: Wikipedia article networks Chameleon and Squirrel (Rozemberczki et al., 2021).

Original: More information of datasets is deferred to Appendix H. We compare ContraNorm against a popular normalization layer PairNorm (Zhao & Akoglu, 2020) designed for preventing oversmoothing.
Revised: More information of datasets is deferred to Appendix H. We compare ContraNorm against a popular normalization layer PairNorm (Zhao & Akoglu, 2020) designed for preventing oversmoothing.

Original: We also take
Revised: We also take

Original: LayerNorm as a baseline by setting the scale s = 0 .
Revised: LayerNorm as a baseline by setting the scale s = 0 .

Original: Setup.
Revised: Setup.

Original: We follow data split setting in Kipf & Welling (2017) with train/validation/test splits of 60%, 20%, 20%, respectively.
Revised: We follow data split setting in Kipf & Welling (2017) with train/validation/test splits of 60%, 20%, 20%, respectively.

Original: To keep fair in comparison, we fix the hidden dimension to 32, and dropout rate to 0.6 as reported in Zhao & Akoglu (2020).
Revised: To keep fair in comparison, we fix the hidden dimension to 32, and dropout rate to 0.6 as reported in Zhao & Akoglu (2020).

Original: We choose the best of scale controller s in range of { 0 . 2 , 0 . 5 , 0 . 8 , 1 . 0 } for both PairNorm and ContraNorm.
Revised: We choose the best of scale controller s in range of { 0 . 2 , 0 . 5 , 0 . 8 , 1 . 0 } for both PairNorm and ContraNorm.

Original: For PairNorm, we choose the best variant presented by Zhao & Akoglu (2020).
Revised: For PairNorm, we choose the best variant presented by Zhao & Akoglu (2020).

Original: Results.
Revised: Results.

Original: As shown in Table 3, in shallow layers (e.g., two layer)
Revised: As shown in Table 3, in shallow layers (e.g., two layer)

Original: , the addition of ContraNorm reduces the accuracy of vanilla GCN by a small margin, while it helps prevent the performance from sharply deteriorating as the layer goes deeper.
Revised: , the addition of ContraNorm reduces the accuracy of vanilla GCN by a small margin, while it helps prevent the performance from sharply deteriorating as the layer goes deeper.

Original: ContraNorm outperforms PairNorm and Lay-
Revised: ContraNorm outperforms PairNorm and Lay-

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: erNorm in the power of de-oversmoothing.
Revised: erNorm in the power of de-oversmoothing.

Original: Here, we show the staple in diluting oversmoothing is
Revised: Here, we show the staple in diluting oversmoothing is

Original: ContraNorm, and LayerNorm alone fails to prevent GCN from oversmoothing, even amplifying the negative effect on Cora with more than 16 layers.
Revised: ContraNorm, and LayerNorm alone fails to prevent GCN from oversmoothing, even amplifying the negative effect on Cora with more than 16 layers.

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: Comparison of different ContraNorm variants.
Revised: Comparison of different ContraNorm variants.

Original: Recalling Section 3.3, we improve ContraNorm with stop-gradient technique (SG) by masking the second term.
Revised: Recalling Section 3.3, we improve ContraNorm with stop-gradient technique (SG) by masking the second term.

Original: For solving data representation instability, we apply layer normalization (LN) to the original version, while for the convenience of theoretical analysis, layer normalization is replaced by L 2 normalization ( L 2 N).
Revised: For solving data representation instability, we apply layer normalization (LN) to the original version, while for the convenience of theoretical analysis, layer normalization is replaced by L 2 normalization ( L 2 N).

Original: Here, we investigate the effect of these tricks and the results are shown in Table 4.
Revised: Here, we investigate the effect of these tricks and the results are shown in Table 4.

Original: Compared with the variant with only LayerNorm, ContraNorm with both stop gradient and layer normalization presents better performance.
Revised: Compared with the variant with only LayerNorm, ContraNorm with both stop gradient and layer normalization presents better performance.

Original: As for the two normalization methods, they are almost comparable to our methods, which verifies the applicability of our theoretical analysis.
Revised: As for the two normalization methods, they are almost comparable to our methods, which verifies the applicability of our theoretical analysis.

Original: Hyperparameters sensitivity analysis.
Revised: Hyperparameters sensitivity analysis.

Original: We conduct an ablation study regarding the gains of ContraNorm while using different values of scale factor s .
Revised: We conduct an ablation study regarding the gains of ContraNorm while using different values of scale factor s .

Original: In Appendix I, we show that ContraNorm is robust in an appropriate range of s .
Revised: In Appendix I, we show that ContraNorm is robust in an appropriate range of s .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we point out the deficiencies of current metrics in characterizing over-smoothing, and highlight the importance of taking dimensional collapse as the entry point for oversmoothing analysis.
Revised: In this paper, we point out the deficiencies of current metrics in characterizing over-smoothing, and highlight the importance of taking dimensional collapse as the entry point for oversmoothing analysis.

Original: Inspired by the uniformity loss in contrastive learning, we design an optimization-induced normalization layer ContraNorm.
Revised: Inspired by the uniformity loss in contrastive learning, we design an optimization-induced normalization layer ContraNorm.

Original: Our theoretical findings indicate ContraNorm mitigates dimensional collapse successfully by reducing variance and effective rank of representations.
Revised: Our theoretical findings indicate ContraNorm mitigates dimensional collapse successfully by reducing variance and effective rank of representations.

Original: Experiments show that ContraNorm boosts the performance of Transformers and GNNs on various tasks.
Revised: Experiments show that ContraNorm boosts the performance of Transformers and GNNs on various tasks.

Original: Our work provides a new perspective from contrastive learning on solving the oversmoothing problem, which helps motivate the following extensive research.
Revised: Our work provides a new perspective from contrastive learning on solving the oversmoothing problem, which helps motivate the following extensive research.

Original: A CKNOWLEDGEMENT
Revised: A CKNOWLEDGEMENT

Original: Yisen Wang is partially supported by the National Key R&D Program of China (2022ZD0160304), the National Natural Science Foundation of China (62006153), Open Research Projects of Zhejiang
Revised: Yisen Wang is partially supported by the National Key R&D Program of China (2022ZD0160304), the National Natural Science Foundation of China (62006153), Open Research Projects of Zhejiang

Original: Lab (No. 2022RC0AB05) and Huawei Technologies Inc.
Revised: Lab (No. 2022RC0AB05) and Huawei Technologies Inc.

Original: Xiaojun Guo thanks for Hanqi Yan for her kind guidance on the implementation of BERT models.
Revised: Xiaojun Guo thanks for Hanqi Yan for her kind guidance on the implementation of BERT models.

Original: We thank anonymous reviewers for their constructive advice on this work
Revised: We thank anonymous reviewers for their constructive advice on this work

Original: A I LLUSTRATION ON THE S TOP -G RADIENT O PERATION IN E Q .(7)
Revised: A I LLUSTRATION ON THE S TOP -G RADIENT O PERATION IN E Q .(7)

Original: In this section, we will illustrate the stop-gradient operation in Eq.(7) by using the framework proposed by (Tao et al., 2022).
Revised: In this section, we will illustrate the stop-gradient operation in Eq.(7) by using the framework proposed by (Tao et al., 2022).

Original: The original update form should be Eq.(6):
Revised: The original update form should be Eq.(6):

Original: [Equation]
Revised: [Equation]

Original: We take SimCLR(Chen et al., 2020) as the contrastive learning framework.
Revised: We take SimCLR(Chen et al., 2020) as the contrastive learning framework.

Original: Tao et al.
Revised: Tao et al.

Original: (2022) have studied the stop-gradient form of SimCLR and illustrated that the stop-gradient operation will make a similar performance with the original one.
Revised: (2022) have studied the stop-gradient form of SimCLR and illustrated that the stop-gradient operation will make a similar performance with the original one.

Original: Based on this, we will elaborate on how AD − 1 is removed in the following part.
Revised: Based on this, we will elaborate on how AD − 1 is removed in the following part.

Original: In fact, we can directly illustrate how the second term in Eq(4) can be omitted.
Revised: In fact, we can directly illustrate how the second term in Eq(4) can be omitted.

Original: [Equation]
Revised: [Equation]

Original: In SimCLR, by denoting the normalized features from the online branch as u oi , i = 1 , 2 , . . . , n , the normalized features from the target branch (although the two branches have no differences in
Revised: In SimCLR, by denoting the normalized features from the online branch as u oi , i = 1 , 2 , . . . , n , the normalized features from the target branch (although the two branches have no differences in

Original: SimCLR) are u ti , i = 1 , 2 , . .
Revised: SimCLR) are u ti , i = 1 , 2 , . .

Original: . , n and V = {
Revised: . , n and V = {

Original: u o 1 , u o 2 , . .
Revised: u o 1 , u o 2 , . .

Original: , u on , u t 1 , u t 2 , . . .
Revised: , u on , u t 1 , u t 2 , . . .

Original: , u tn } , the SimCLR loss can be represented as
Revised: , u tn } , the SimCLR loss can be represented as

Original: [Equation]
Revised: [Equation]

Original: where u 1 and u ′ 1 are positive pairs and τ is the temperature.
Revised: where u 1 and u ′ 1 are positive pairs and τ is the temperature.

Original: Then the gradient of L on u o 1 can be calculated as
Revised: Then the gradient of L on u o 1 can be calculated as

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: is the softmax results over similarities between u o 1 and other samples, and
Revised: is the softmax results over similarities between u o 1 and other samples, and

Original: [Equation]
Revised: [Equation]

Original: is computed over similarities between sample v and its contrastive samples V /v .
Revised: is computed over similarities between sample v and its contrastive samples V /v .

Original: We can see that the first term of ∂L/∂u o 1 comes from the part which takes u o 1 as the anchor, and the second term comes from the part which takes the other feature as the anchor.
Revised: We can see that the first term of ∂L/∂u o 1 comes from the part which takes u o 1 as the anchor, and the second term comes from the part which takes the other feature as the anchor.

Original: Tao et al.
Revised: Tao et al.

Original: (2022) proposes to stop the second term and verifies that stopping the second gradient term will not affect the performance empirically.
Revised: (2022) proposes to stop the second term and verifies that stopping the second gradient term will not affect the performance empirically.

Original: Note that the u t 1 term in the gradient is from the alignment loss.
Revised: Note that the u t 1 term in the gradient is from the alignment loss.

Original: So the gradient of the uniformity loss on u o 1 can be written as ∂L uniform
Revised: So the gradient of the uniformity loss on u o 1 can be written as ∂L uniform

Original: [Equation]
Revised: [Equation]

Original: It is noteworthy that by writing V
Revised: It is noteworthy that by writing V

Original: = { h 1 , h 2 , . . .
Revised: = { h 1 , h 2 , . . .

Original: , h N } , Eq(4) shares the same form as Eq(11).
Revised: , h N } , Eq(4) shares the same form as Eq(11).

Original: By adopting the stop-gradient method just as (Tao et al., 2022) takes, we remove the second term in
Revised: By adopting the stop-gradient method just as (Tao et al., 2022) takes, we remove the second term in

Original: Eq(4), which is just the AD − 1 term in Eq(6).
Revised: Eq(4), which is just the AD − 1 term in Eq(6).

Original: Empirically, we draw the singular value distribution of embeddings for vanilla BERT and +ContraNorm with only AD − 1 term or D − 1 A term on RTE task.
Revised: Empirically, we draw the singular value distribution of embeddings for vanilla BERT and +ContraNorm with only AD − 1 term or D − 1 A term on RTE task.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4, compared with vanilla BERT with a long-tail distribution (dimensional collapse), adding ContraNorm with D − 1 A and AD − 1 both reduce the number of insignificant (nearly zero) singular values and make a more balanced distribution.
Revised: 4, compared with vanilla BERT with a long-tail distribution (dimensional collapse), adding ContraNorm with D − 1 A and AD − 1 both reduce the number of insignificant (nearly zero) singular values and make a more balanced distribution.

Original: The similar singular value distributions mean that they play a similar role in alleviating dimensional collapse.
Revised: The similar singular value distributions mean that they play a similar role in alleviating dimensional collapse.

Original: [Figure]
Revised: [Figure]

Original: BERT
Revised: BERT

Original: [Figure]
Revised: [Figure]

Original: ContraNorm with D 1 A + ContraNorm
Revised: ContraNorm with D 1 A + ContraNorm

Original: [Figure]
Revised: [Figure]

Original: B P ROOFS FOR P ROPOSITIONS IN S ECTION 4. 
Revised: B P ROOFS FOR P ROPOSITIONS IN S ECTION 4.

Original: We give out two lemmas.
Revised: We give out two lemmas.

Original: Lemma B.1.
Revised: Lemma B.1.

Original: Denote e = [1 , 1 , . .
Revised: Denote e = [1 , 1 , . .

Original: , 1] ⊤ / √ n .
Revised: , 1] ⊤ / √ n .

Original: For any update form X 1 = P X 0 and λ > 0 , if the eigenvalues of ( I − ee ⊤ ) − λ P ⊤ ( I − ee ⊤ ) P are all not larger than zero, then we have
Revised: For any update form X 1 = P X 0 and λ > 0 , if the eigenvalues of ( I − ee ⊤ ) − λ P ⊤ ( I − ee ⊤ ) P are all not larger than zero, then we have

Original: Var( X 1 ) ≥ λ − 1 · Var( X 0 ) .
Revised: Var( X 1 ) ≥ λ − 1 · Var( X 0 ) .

Original: Proof.
Revised: Proof.

Original: We denote X 0 X ⊤= Y diag ( ω 1 , . .
Revised: We denote X 0 X ⊤= Y diag ( ω 1 , . .

Original: . , ω n ) Y ⊤ for the eigen-decomposition of X 0 X ⊤ 0 , where Y =
Revised: . , ω n ) Y ⊤ for the eigen-decomposition of X 0 X ⊤ 0 , where Y =

Original: [ y 1 , y 2 , . . .
Revised: [ y 1 , y 2 , . . .

Original: , y n ] is the orthogonal basis and all ω i ≥ 0 .
Revised: , y n ] is the orthogonal basis and all ω i ≥ 0 .

Original: Note that ( I − ee ⊤ ) 

Revised: Note that ( I − ee ⊤ )

Original: =
Revised: =

Original: ( I − ee ⊤ ) .
Revised: ( I − ee ⊤ ) .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Similarly, we have
Revised: Similarly, we have

Original: Var( X 1 ) =
Revised: Var( X 1 ) =

Original: n (cid:88) i =
Revised: n (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: i y ⊤
Revised: i y ⊤

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: Thus, if the eigenvalues of ( I − ee ⊤ ) − λ P ⊤ ( I − ee ⊤ ) P := Σ are all not larger than zero, Σ is semi-negative definite, then we have
Revised: Thus, if the eigenvalues of ( I − ee ⊤ ) − λ P ⊤ ( I − ee ⊤ ) P := Σ are all not larger than zero, Σ is semi-negative definite, then we have

Original: [Equation]
Revised: [Equation]

Original: which implies that Var( X 0 ) − λ Var( X 1 ) ≤ 0 .
Revised: which implies that Var( X 0 ) − λ Var( X 1 ) ≤ 0 .

Original: Therefore, Var( X 1 ) ≥ λ − 1 · Var( X 0 ) .
Revised: Therefore, Var( X 1 ) ≥ λ − 1 · Var( X 0 ) .

Original: The second lemma is from the Eq.(13) in (Fulton, 2000).
Revised: The second lemma is from the Eq.(13) in (Fulton, 2000).

Original: Lemma B.2.
Revised: Lemma B.2.

Original: Let A, B, C be symmetric matrices and C
Revised: Let A, B, C be symmetric matrices and C

Original: =
Revised: =

Original: A + B. Suppose the eigenvalues of A are α 1 ≥ α 2 ≥ · · · ≥ α n , the eigenvalues of B are β 1 ≥ β 2 ≥ · · · ≥ β n , and the eigenvalues of C are γ 1 ≥ γ 2 ≥ · · · ≥ γ n .
Revised: A + B. Suppose the eigenvalues of A are α 1 ≥ α 2 ≥ · · · ≥ α n , the eigenvalues of B are β 1 ≥ β 2 ≥ · · · ≥ β n , and the eigenvalues of C are γ 1 ≥ γ 2 ≥ · · · ≥ γ n .

Original: Then we have the inequality
Revised: Then we have the inequality

Original: [Equation]
Revised: [Equation]

Original: We can now start to prove Proposition 1.
Revised: We can now start to prove Proposition 1.

Original: Proposition 1.
Revised: Proposition 1.

Original: Let e =
Revised: Let e =

Original: . , 1) ⊤ / √ n .
Revised: . , 1) ⊤ / √ n .

Original: For attention matrix ¯ A = softmax( H b H ⊤ b ) , let σ min be the smallest eigenvalue of P =
Revised: For attention matrix ¯ A = softmax( H b H ⊤ b ) , let σ min be the smallest eigenvalue of P =

Original: ( I − ee ⊤ )( I − ¯ A )
Revised: ( I − ee ⊤ )( I − ¯ A )

Original: + ( I − ¯ A ) ⊤ ( I − ee ⊤ ) .
Revised: + ( I − ¯ A ) ⊤ ( I − ee ⊤ ) .

Original: For the ContraNorm update H t =
Revised: For the ContraNorm update H t =

Original: ((1 + s )
Revised: ((1 + s )

Original: I − s ¯ A ) H b , s > 0 , we have V ar ( H t ) ≥ (1 − sσ min ) − 1 · V ar ( H b ) .
Revised: I − s ¯ A ) H b , s > 0 , we have V ar ( H t ) ≥ (1 − sσ min ) − 1 · V ar ( H b ) .

Original: Especially, if σ min ≥ 0 , we have V ar ( H t ) ≥
Revised: Especially, if σ min ≥ 0 , we have V ar ( H t ) ≥

Original: V ar ( H b ) .
Revised: V ar ( H b ) .

Original: Proof.
Revised: Proof.

Original: We denote Σ
Revised: We denote Σ

Original: = ( I − ee ⊤ ) − λ ((1 + s )
Revised: = ( I − ee ⊤ ) − λ ((1 + s )

Original: I − s
Revised: I − s

Original: ¯ A ) ⊤ ( I − ee ⊤ )((1 + s )
Revised: ¯ A ) ⊤ ( I − ee ⊤ )((1 + s )

Original: I − s ¯ A ) .
Revised: I − s ¯ A ) .

Original: Then, Σ = (1 − λ )
Revised: Then, Σ = (1 − λ )

Original: I − λ
Revised: I − λ

Original: ( s ( I − ¯ A
Revised: ( s ( I − ¯ A

Original: ) ⊤ ( I − ee ⊤ ) − s ( I − ee ⊤ )( I − ¯ A ) − s 2 ( I − ¯ A ) ⊤ ( I − ee ⊤ )( I − ¯ A )) =
Revised: ) ⊤ ( I − ee ⊤ ) − s ( I − ee ⊤ )( I − ¯ A ) − s 2 ( I − ¯ A ) ⊤ ( I − ee ⊤ )( I − ¯ A )) =

Original: (1 − λ )
Revised: (1 − λ )

Original: I − λs P − λs
Revised: I − λs P − λs

Original: ( I − ¯ A ) ⊤ ( I − ee ⊤ )( I − ¯ A ) .
Revised: ( I − ¯ A ) ⊤ ( I − ee ⊤ )( I − ¯ A ) .

Original: Let α 1 ≥ α 2 ≥ · · · ≥ α n be the eigenvalues of Σ .
Revised: Let α 1 ≥ α 2 ≥ · · · ≥ α n be the eigenvalues of Σ .

Original: Since I − ee ⊤ has a eigenvalue of 0 and n −eigenvalues of 1, I − ee ⊤ is a semi-definite positive matrix.
Revised: Since I − ee ⊤ has a eigenvalue of 0 and n −eigenvalues of 1, I − ee ⊤ is a semi-definite positive matrix.

Original: Thus, s 2 ( I − ¯ A ) ⊤ ( I − ee ⊤ )( I − ¯ A ) is also a semi-definite positive matrix.
Revised: Thus, s 2 ( I − ¯ A ) ⊤ ( I − ee ⊤ )( I − ¯ A ) is also a semi-definite positive matrix.

Original: Notice that the largest eigenvalue of (1 − λ )
Revised: Notice that the largest eigenvalue of (1 − λ )

Original: I − s P is (1 − λ ) − sσ min and the largest eigenvalue of − s 2 ( I − ¯ A ) ⊤ ( I − ee ⊤ )
Revised: I − s P is (1 − λ ) − sσ min and the largest eigenvalue of − s 2 ( I − ¯ A ) ⊤ ( I − ee ⊤ )

Original: ( I − ¯ A ) is 0 .
Revised: ( I − ¯ A ) is 0 .

Original: Therefore, by Lemma B.2, the largest eigenvalue of Σ is less or equal to (1 − λ ) − sσ min .
Revised: Therefore, by Lemma B.2, the largest eigenvalue of Σ is less or equal to (1 − λ ) − sσ min .

Original: Let λ = 1 − sσ min , then the largest eigenvalue of Σ is less or equal to 0 .
Revised: Let λ = 1 − sσ min , then the largest eigenvalue of Σ is less or equal to 0 .

Original: By Lemma B.1, we have V ar ( H t ) ≥ (1 − sσ min ) − 1 · V ar ( H b ) .
Revised: By Lemma B.1, we have V ar ( H t ) ≥ (1 − sσ min ) − 1 · V ar ( H b ) .

Original: Moreover, if σ min ≥ 0 , then (1 − sσ min ) − 1 ≥ 1 , leading to V ar ( H t ) ≥ V ar ( H b ) .
Revised: Moreover, if σ min ≥ 0 , then (1 − sσ min ) − 1 ≥ 1 , leading to V ar ( H t ) ≥ V ar ( H b ) .

Original: Remark.
Revised: Remark.

Original: Now we discuss some sufficient conditions for σ min ≥ 0 .
Revised: Now we discuss some sufficient conditions for σ min ≥ 0 .

Original: If P is a diagonally dominant matrix, then we will have the result σ min ≥ 0 .
Revised: If P is a diagonally dominant matrix, then we will have the result σ min ≥ 0 .

Original: Denote a ij = ¯
Revised: Denote a ij = ¯

Original: A ij , b ij = σ ij − a ij and Q =
Revised: A ij , b ij = σ ij − a ij and Q =

Original: ( I − ee ⊤ )( I − ¯ A ) , where σ ij = 1 if i = j and σ ij = 

Revised: ( I − ee ⊤ )( I − ¯ A ) , where σ ij = 1 if i = j and σ ij =

Original: if i ̸
Revised: if i ̸

Original: = j , then we have
Revised: = j , then we have

Original: [Equation]
Revised: [Equation]

Original: If we have (cid:80) nk =1 a kj ≤ 1 + na ij for any i, j , then we will have
Revised: If we have (cid:80) nk =1 a kj ≤ 1 + na ij for any i, j , then we will have

Original: [Equation]
Revised: [Equation]

Original: Notice that (cid:80) nk =1 b kj = 0 , we have
Revised: Notice that (cid:80) nk =1 b kj = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Since ¯ A is an attention matrix, we also have
Revised: Since ¯ A is an attention matrix, we also have

Original: [Equation]
Revised: [Equation]

Original: Q jk | .
Revised: Q jk | .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: which indicates that P is a diagonally dominated matrix, thus σ min ≥ 0 .
Revised: which indicates that P is a diagonally dominated matrix, thus σ min ≥ 0 .

Original: Therefore, (cid:80) nk
Revised: Therefore, (cid:80) nk

Original: =
Revised: =

Original: a kj ≤ 1 + na ij for any i, j is just a sufficient condition.
Revised: a kj ≤ 1 + na ij for any i, j is just a sufficient condition.

Original: A special case for this is (cid:80) nk =1 a kj = 1 , ∀ k .
Revised: A special case for this is (cid:80) nk =1 a kj = 1 , ∀ k .

Original: We now move on to the proof of Proposition 2.
Revised: We now move on to the proof of Proposition 2.

Original: We first give a lemma on the property of effective rank.
Revised: We first give a lemma on the property of effective rank.

Original: Lemma B.3.
Revised: Lemma B.3.

Original: Let the eigenvalues of AA ⊤ be λ 1 ≥ λ 2 ≥ · · · ≥ λ n and the eigenvalues of BB ⊤ be σ 1 ≥ σ 2 ≥ . . .
Revised: Let the eigenvalues of AA ⊤ be λ 1 ≥ λ 2 ≥ · · · ≥ λ n and the eigenvalues of BB ⊤ be σ 1 ≥ σ 2 ≥ . . .

Original: σ n .
Revised: σ n .

Original: If σ i /λ i is increasing as i increases, then we have erank( B ) ≥ erank( A ) .
Revised: If σ i /λ i is increasing as i increases, then we have erank( B ) ≥ erank( A ) .

Original: This lemma can be proved just by using the definition of the effective rank.
Revised: This lemma can be proved just by using the definition of the effective rank.

Original: Proposition 2.
Revised: Proposition 2.

Original: Consider the update form
Revised: Consider the update form

Original: [Equation]
Revised: [Equation]

Original: let σ max be the largest singular value of H b .
Revised: let σ max be the largest singular value of H b .

Original: For s > 0 satisfying 1 + (1 − σ 2max ) s > 0 , we have erank( H t ) >
Revised: For s > 0 satisfying 1 + (1 − σ 2max ) s > 0 , we have erank( H t ) >

Original: erank( H b ) .
Revised: erank( H b ) .

Original: Proof.
Revised: Proof.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Suppose λ 1 ≥ λ 2 ≥ · · · ≥ λ n are the eigenvalues of H b H ⊤ b , then H t H Tt has the same eigenvectors as H b H ⊤ b , and its eigenvalues are ( λ i s − (1 + s )) 2 λ i .
Revised: Suppose λ 1 ≥ λ 2 ≥ · · · ≥ λ n are the eigenvalues of H b H ⊤ b , then H t H Tt has the same eigenvectors as H b H ⊤ b , and its eigenvalues are ( λ i s − (1 + s )) 2 λ i .

Original: Since s satisfies 1 + (1 − σ 2max s ) > 0 , we have 1 + s >
Revised: Since s satisfies 1 + (1 − σ 2max s ) > 0 , we have 1 + s >

Original: λ i s .
Revised: λ i s .

Original: Therefore, ( λ i s − (1 + s )) 2 is increasing as i increases, resulting the fact that erank( H t ) >
Revised: Therefore, ( λ i s − (1 + s )) 2 is increasing as i increases, resulting the fact that erank( H t ) >

Original: erank( H b ) by using Lemma B.3.
Revised: erank( H b ) by using Lemma B.3.

Original: C M ETRICS CALCULATING FEATURE AND ATTENTION MAP COSINE SIMILARITY
Revised: C M ETRICS CALCULATING FEATURE AND ATTENTION MAP COSINE SIMILARITY

Original: Following Wang et al.
Revised: Following Wang et al.

Original: , given feature map H ∈ R N × D and attention map of the h -th head
Revised: , given feature map H ∈ R N × D and attention map of the h -th head

Original: A ( h ) ∈ R N × N with batch size N and hidden embedding size D , the feature cosine similarity and the attention cosine similarity is computed as
Revised: A ( h ) ∈ R N × N with batch size N and hidden embedding size D , the feature cosine similarity and the attention cosine similarity is computed as

Original: [Equation]
Revised: [Equation]

Original: where h i denotes the i -th row of H , a i is the i -th column of A , and H is the number of attention heads.
Revised: where h i denotes the i -th row of H , a i is the i -th column of A , and H is the number of attention heads.

Original: D A PPLYING THE U NIFORMITY L OSS D IRECTLY
Revised: D A PPLYING THE U NIFORMITY L OSS D IRECTLY

Original: We conduct a comparative experiment on BERT model with straightforwardly applied uniformity loss and our proposed ContraNorm.
Revised: We conduct a comparative experiment on BERT model with straightforwardly applied uniformity loss and our proposed ContraNorm.

Original: Specifically, we add the uniformity loss ( loss uni ) to the classification loss (MSELoss, CrossEntropyLoss or BCELoss depending on the task type, denoted by loss cls ).
Revised: Specifically, we add the uniformity loss ( loss uni ) to the classification loss (MSELoss, CrossEntropyLoss or BCELoss depending on the task type, denoted by loss cls ).

Original: Formally, the final loss is written as
Revised: Formally, the final loss is written as

Original: [Equation]
Revised: [Equation]

Original: where loss uni = (cid:80) Ni =1 log (cid:80) Nj =

Revised: where loss uni = (cid:80) Ni =1 log (cid:80) Nj =

Original: exp( f ( x i ) T f ( x j ) /τ ) and N is the number of samples in the batch.
Revised: exp( f ( x i ) T f ( x j ) /τ ) and N is the number of samples in the batch.

Original: We tune τ in the range of [0 . 5 , 0 . 8 , 1 . 0 , 1 . 2 , 1 . 5] and choose the best one in terms of average performance.
Revised: We tune τ in the range of [0 . 5 , 0 . 8 , 1 . 0 , 1 . 2 , 1 . 5] and choose the best one in terms of average performance.

Original: Other hyperparameters are kept the same as settings of ContraNorm.
Revised: Other hyperparameters are kept the same as settings of ContraNorm.

Original: The results are shown in the Table.
Revised: The results are shown in the Table.

Original: We can see that +ContraNorm gets the best score in 8 / 9 tasks, while +Uniformity loss reaches the best only in 1 / 9 tasks.
Revised: We can see that +ContraNorm gets the best score in 8 / 9 tasks, while +Uniformity loss reaches the best only in 1 / 9 tasks.

Original: ContraNorm also has the highest average score among all tasks.
Revised: ContraNorm also has the highest average score among all tasks.

Original: The reason is that updating the total loss is a combined process for objectives of correct classification and uniform distribution.
Revised: The reason is that updating the total loss is a combined process for objectives of correct classification and uniform distribution.

Original: Thus, a lower loss total may be only caused by a lower classification loss while uniformity loss is kept the same, which cannot ensure a more uniform distribution of representations.
Revised: Thus, a lower loss total may be only caused by a lower classification loss while uniformity loss is kept the same, which cannot ensure a more uniform distribution of representations.

Original: In contrast, ContraNorm acts directly on representations in each layer and enforces the uniform property.
Revised: In contrast, ContraNorm acts directly on representations in each layer and enforces the uniform property.

Original: In fact, there are many methods in GNNs such as Yang et al.
Revised: In fact, there are many methods in GNNs such as Yang et al.

Original: (2021) and Zhu et al.
Revised: (2021) and Zhu et al.

Original: (2021), which design the propagation mechanism under the guidance of the corresponding objective.
Revised: (2021), which design the propagation mechanism under the guidance of the corresponding objective.

Original: The welldesigned propagation mechanism is shown to be the most fundamental part of GNNs (Zhu et al.,
Revised: The welldesigned propagation mechanism is shown to be the most fundamental part of GNNs (Zhu et al.,

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Instead of directly using the loss function, these methods transfer the loss function into a specific propagation method and achieve superior performance
Revised: Instead of directly using the loss function, these methods transfer the loss function into a specific propagation method and achieve superior performance

Original: [Table]
Revised: [Table]

Original: which indicates that changing the network may be more effective than directly adding the objective to the loss function.
Revised: which indicates that changing the network may be more effective than directly adding the objective to the loss function.

Original: E C HOICES OF C ONTRA N ORM PLUGGING POSITION
Revised: E C HOICES OF C ONTRA N ORM PLUGGING POSITION

Original: We explore two ways to integrate ContraNorm into BERT and ALBERT.
Revised: We explore two ways to integrate ContraNorm into BERT and ALBERT.

Original: Concretely, consider the update of H ( l ) in l -th block of Transformers
Revised: Concretely, consider the update of H ( l ) in l -th block of Transformers

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where
Revised: where

Original: X = H b is the input tensor.
Revised: X = H b is the input tensor.

Original: We choose positions 1) between Eq.(26) and Eq.(27), named as before-residual ; 2) between Eq.(27) and Eq.(28), named as after-residual .
Revised: We choose positions 1) between Eq.(26) and Eq.(27), named as before-residual ; 2) between Eq.(27) and Eq.(28), named as after-residual .

Original: The performance comparison between the two positions on
Revised: The performance comparison between the two positions on

Original: GLUE datasets is shown in Table.
Revised: GLUE datasets is shown in Table.

Original: It is observed that putting ContraNorm after the residual connection slightly outperforms that before residual.
Revised: It is observed that putting ContraNorm after the residual connection slightly outperforms that before residual.

Original: Therefore, we choose the after-residual variant as our basic ContraNorm.
Revised: Therefore, we choose the after-residual variant as our basic ContraNorm.

Original: F T IME C OMPLEXITY A NALYSIS AND A M ODIFIED C ONTRA N ORM
Revised: F T IME C OMPLEXITY A NALYSIS AND A M ODIFIED C ONTRA N ORM

Original: The most time-consuming operation of ContraNorm is the matrix multiplication.
Revised: The most time-consuming operation of ContraNorm is the matrix multiplication.

Original: Given H ∈ R n × d , where n and d denote the number of samples in a batch and feature size respectively, the time complexity of ContraNorm is O ( n 2 d ) , which is the same order as the self-attention operation in
Revised: Given H ∈ R n × d , where n and d denote the number of samples in a batch and feature size respectively, the time complexity of ContraNorm is O ( n 2 d ) , which is the same order as the self-attention operation in

Original: Transformer.
Revised: Transformer.

Original: Empirically, we report the training time of BERT with or without ContraNorm on
Revised: Empirically, we report the training time of BERT with or without ContraNorm on

Original: GLUE tasks in Table 7.
Revised: GLUE tasks in Table 7.

Original: All experiments are conducted on a single NVIDIA GeForce RTX 3090.
Revised: All experiments are conducted on a single NVIDIA GeForce RTX 3090.

Original: On average, we raise the performance of BERT on GLUE tasks from 82.59% to 83.54% (see Table 1) with less than 4 minutes overhead.
Revised: On average, we raise the performance of BERT on GLUE tasks from 82.59% to 83.54% (see Table 1) with less than 4 minutes overhead.

Original: We think the time overhead is acceptable considering the benefits it brings.
Revised: We think the time overhead is acceptable considering the benefits it brings.

Original: Moreover, to enhance scalability we propose a modified version of ContraNorm inspired by Ali et al.
Revised: Moreover, to enhance scalability we propose a modified version of ContraNorm inspired by Ali et al.

Original: As already alluded to, ContraNorm resolves dimensional collapse by pushing samples away from each other with the uniformity loss as the energy function.
Revised: As already alluded to, ContraNorm resolves dimensional collapse by pushing samples away from each other with the uniformity loss as the energy function.

Original: A different choice of the underlying function generates a new form of normalization layer.
Revised: A different choice of the underlying function generates a new form of normalization layer.

Original: Here, to maintain the same goal of settling dimensional collapse, we choose another contrastive loss named Barlow Twins (Zbontar et al., 2021)
Revised: Here, to maintain the same goal of settling dimensional collapse, we choose another contrastive loss named Barlow Twins (Zbontar et al., 2021)

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: with C ij = (cid:80) b f ( x Ab,i ) f
Revised: with C ij = (cid:80) b f ( x Ab,i ) f

Original: ( x Bb,j ) / (cid:113)(cid:80) b ( f ( x Ab,i )) 2 (cid:113)(cid:80) b ( f ( x Bb,j )) 2 , where λ is a trading-off constant, b is the batch index, i, j index the vector dimension, and x A ,
Revised: ( x Bb,j ) / (cid:113)(cid:80) b ( f ( x Ab,i )) 2 (cid:113)(cid:80) b ( f ( x Bb,j )) 2 , where λ is a trading-off constant, b is the batch index, i, j index the vector dimension, and x A ,

Original: x B denotes different augmented views of x .
Revised: x B denotes different augmented views of x .

Original: Dislike the sample-wise loss InfoNCE, Barlow Twins decorrelates the dimensions and can be seen as contrastive between the dimensions of the embeddings (Garrido et al., 2022).
Revised: Dislike the sample-wise loss InfoNCE, Barlow Twins decorrelates the dimensions and can be seen as contrastive between the dimensions of the embeddings (Garrido et al., 2022).

Original: Despite seeming different, recent works empirically show strong similarities between the two loss functions, and theoretically prove the equivalence between them under limited assumptions (Garrido et al., 2022;
Revised: Despite seeming different, recent works empirically show strong similarities between the two loss functions, and theoretically prove the equivalence between them under limited assumptions (Garrido et al., 2022;

Original: Tao et al., 2022; Balestriero & LeCun, 2022; Huang et al., 2021).
Revised: Tao et al., 2022; Balestriero & LeCun, 2022; Huang et al., 2021).

Original: Therefore, the layer generated by unfolding iterations of Barlow Twins is believed to work equally well as ContraNorm.
Revised: Therefore, the layer generated by unfolding iterations of Barlow Twins is believed to work equally well as ContraNorm.

Original: Specifically, we reorder the matrix multiplication from HH T ∈ R n × n to H T
Revised: Specifically, we reorder the matrix multiplication from HH T ∈ R n × n to H T

Original: H ∈ R d × d to enforce the channels of embeddings different from each other.
Revised: H ∈ R d × d to enforce the channels of embeddings different from each other.

Original: Then the modified version of ContraNorm becomes
Revised: Then the modified version of ContraNorm becomes

Original: [Equation]
Revised: [Equation]

Original: The transposed alternative has a linear complexity in the number of samples,
Revised: The transposed alternative has a linear complexity in the number of samples,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e. O ( nd 2 ) .
Revised: e. O ( nd 2 ) .

Original: In the case of n ≫ d , the modified version greatly alleviates the computation burden.
Revised: In the case of n ≫ d , the modified version greatly alleviates the computation burden.

Original: To verify whether it performs equally well as ContraNorm
Revised: To verify whether it performs equally well as ContraNorm

Original: [Table]
Revised: [Table]

Original: we conduct experiments on the validation set of GLUE tasks.
Revised: we conduct experiments on the validation set of GLUE tasks.

Original: The learning rate of BERT with ContraNorm-v2 is set to 4 e − 5 and other experiment setups are the same in Section 4.1.
Revised: The learning rate of BERT with ContraNorm-v2 is set to 4 e − 5 and other experiment setups are the same in Section 4.1.

Original: As shown from Table 8, for each task the performance of BERT with ContraNorm-v2 surpasses the vanilla BERT, and the average performance is raised from 82 .
Revised: As shown from Table 8, for each task the performance of BERT with ContraNorm-v2 surpasses the vanilla BERT, and the average performance is raised from 82 .

Original: 59% to 84 .
Revised: 59% to 84 .

Original: The results imply effectiveness of this modified version of ContraNorm, which can also be explained with the relationship between Gram matrix ( G = HH T ∈ R n × n ) and covariance matrix (
Revised: The results imply effectiveness of this modified version of ContraNorm, which can also be explained with the relationship between Gram matrix ( G = HH T ∈ R n × n ) and covariance matrix (

Original: C = H T
Revised: C = H T

Original: H ∈ R d × d ) (Ali et al., 2021).
Revised: H ∈ R d × d ) (Ali et al., 2021).

Original: [Table]
Revised: [Table]

Original: G R ESULTS ON TEST SET OF GLUE DATASETS
Revised: G R ESULTS ON TEST SET OF GLUE DATASETS

Original: We submit our trained model to GLUE benchmark leaderboard and the resultant feedback of performance is shown in Table (9).
Revised: We submit our trained model to GLUE benchmark leaderboard and the resultant feedback of performance is shown in Table (9).

Original: [Table]
Revised: [Table]

Original: H I NTRODUCTION OF GRAPH DATASETS
Revised: H I NTRODUCTION OF GRAPH DATASETS

Original: Citation Network.
Revised: Citation Network.

Original: Cora, CiteSeer are three popular citation graph datasets.
Revised: Cora, CiteSeer are three popular citation graph datasets.

Original: In these graphs, nodes represent papers and edges correspond to the citation relationship between two papers.
Revised: In these graphs, nodes represent papers and edges correspond to the citation relationship between two papers.

Original: Nodes are classified according to academic topics.
Revised: Nodes are classified according to academic topics.

Original: Wikipedia Network.
Revised: Wikipedia Network.

Original: Chameleon and Squirrel are Wikipedia page networks on specific topics, where nodes represent web pages and edges are the mutual links between them.
Revised: Chameleon and Squirrel are Wikipedia page networks on specific topics, where nodes represent web pages and edges are the mutual links between them.

Original: Node features are the bag-of-words representation of informative nouns.
Revised: Node features are the bag-of-words representation of informative nouns.

Original: The nodes are classified into four categories according to the number of the average monthly traffic of the page.
Revised: The nodes are classified into four categories according to the number of the average monthly traffic of the page.

Original: Statics of datasets are shown in Table.
Revised: Statics of datasets are shown in Table.

Original: [Table]
Revised: [Table]

Original: I A BLATION STUDY FULL RESULTS
Revised: I A BLATION STUDY FULL RESULTS

Original: We conduct experiments using BERT+ContraNorm with varying scaling factor s on GLUE datasets.
Revised: We conduct experiments using BERT+ContraNorm with varying scaling factor s on GLUE datasets.

Original: For each dataset, we vary the normalization scale around the best choice, and other parameters remain consistent.
Revised: For each dataset, we vary the normalization scale around the best choice, and other parameters remain consistent.

Original: As shown in Fig.(5), the results illustrate that model with ContraNorm is robust in an appropriate range of normalization scaling factor.
Revised: As shown in Fig.(5), the results illustrate that model with ContraNorm is robust in an appropriate range of normalization scaling factor.

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: F
Revised: F

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: r a c y
Revised: r a c y

Original: [Figure]
Revised: [Figure]