Original: C ONTRA N ORM : A C ONTRASTIVE L EARNING P ERSPECTIVE ON O VERSMOOTHING AND B EYONDAnonymous
Revised: C ONTRA N ORM : A C ONTRASTIVE L EARNING P ERSPECTIVE ON O VERSMOOTHING AND B EYOND

Original: authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Oversmoothing is a common phenomenon in a wide range of Graph Neural Networks (GNNs) and Transformers, where performance degenerates as the layer goes deeper.
Revised: Oversmoothing is a common phenomenon in a wide range of Graph Neural Networks (GNNs) and Transformers, where performance degenerates as the layer goes deeper.

Original: Instead of characterizing oversmoothing from the view of complete collapse in which representations converge to a single point, we dive into a more general perspective dimensional collapse in which representations lie in a narrow cone.
Revised: Instead of characterizing oversmoothing from the view of complete collapse in which representations converge to a single point, we dive into a more general perspective dimensional collapse in which representations lie in a narrow cone.

Original: Accordingly, inspired by the power of contrastive learning in preventing dimensional collapse, we propose a novel normalization layer ContraNorm .
Revised: Accordingly, inspired by the power of contrastive learning in preventing dimensional collapse, we propose a novel normalization layer ContraNorm .

Original: Intuitively, ContraNorm implicitly shatters representations in the embedding space, leading to a more uniform distribution and slighter dimensional collapse.
Revised: Intuitively, ContraNorm implicitly shatters representations in the embedding space, leading to a more uniform distribution and slighter dimensional collapse.

Original: On the theoretical analysis, we prove that ContraNorm can alleviate both complete collapse and dimensional collapse under some conditions.
Revised: On the theoretical analysis, we prove that ContraNorm can alleviate both complete collapse and dimensional collapse under some conditions.

Original: Our proposed normalization layer can be easily inserted into GNNs and Transformers with negligible parameter overhead.
Revised: Our proposed normalization layer can be easily inserted into GNNs and Transformers with negligible parameter overhead.

Original: Experiments on various real-world datasets verify the effectiveness of our method.
Revised: Experiments on various real-world datasets verify the effectiveness of our method.

Revised: Our implementation is available at https://github.com/PKU-ML/ContraNorm .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recently, the rise of Graph Neural Networks (GNNs) has enabled important breakthroughs in various fields of graph learning (Ying et al., 2018; Senior et al., 2020).
Revised: Recently, the rise of Graph Neural Networks (GNNs) has enabled important breakthroughs in various fields of graph learning (Ying et al., 2018; Senior et al., 2020).

Original: Along the other avenue, although getting rid of bespoke convolution operators, Transformers (Vaswani et al., 2017) also achieve phenomenal success in multiple natural language processing (NLP) tasks (Lan et al., 2020; Liu et al., 2020; Rajpurkar et al., 2018) and be transferred successfully to computer vision (CV) field (Dosovitskiy et al., 2021; Liu et al., 2021; Strudel et al., 2021).
Revised: Along the other avenue, although getting rid of bespoke convolution operators, Transformers (Vaswani et al., 2017) also achieve phenomenal success in multiple natural language processing (NLP) tasks (Lan et al., 2020; Liu et al., 2020; Rajpurkar et al., 2018) and be transferred successfully to computer vision (CV) field (Dosovitskiy et al., 2021; Liu et al., 2021; Strudel et al., 2021).

Original: Despite sharing different model architectures,
Revised: Despite sharing different model architectures,

Original: GNNs and Transformers are both hindered by oversmoothing problem (Li et al., 2018; Tang et al., 2021), where deeply stacking layers gives rise to indistinguishable representations and significant performance deterioration.
Revised: GNNs and Transformers are both hindered by oversmoothing problem (Li et al., 2018; Tang et al., 2021), where deeply stacking layers gives rise to indistinguishable representations and significant performance deterioration.

Original: In order to get rid of oversmoothing, we need to dive into the modules inside and understand how oversmoothing happens on the first hand.
Revised: In order to get rid of oversmoothing, we need to dive into the modules inside and understand how oversmoothing happens on the first hand.

Original: However, we notice that existing oversmoothing analysis fails to fully characterize the behavior of learned features.
Revised: However, we notice that existing oversmoothing analysis fails to fully characterize the behavior of learned features.

Original: A canonical metric for oversmoothing is the average similarity (Zhou et al., 2021;
Revised: A canonical metric for oversmoothing is the average similarity (Zhou et al., 2021;

Original: Gong et al., 2021; Wang et al., 2022).
Revised: Gong et al., 2021; Wang et al., 2022).

Original: The tendency of similarity converging to 1 indicates representations shrink to a single point (complete collapse).
Revised: The tendency of similarity converging to 1 indicates representations shrink to a single point (complete collapse).

Original: However, this metric can not depict a more general collapse case, where representations span a lowdimensional manifold in the embedding space and also sacrifice expressive power, which is called dimensional collapse (left figure in Figure 1).
Revised: However, this metric can not depict a more general collapse case, where representations span a lowdimensional manifold in the embedding space and also sacrifice expressive power, which is called dimensional collapse (left figure in Figure 1).

Original: In such cases, similarity fails to quantify the collapse level.
Revised: In such cases, similarity fails to quantify the collapse level.

Original: Therefore, we need to go beyond existing measures and take this so-called dimensional collapse into consideration.
Revised: Therefore, we need to go beyond existing measures and take this so-called dimensional collapse into consideration.

Original: Actually, this dimensional collapse behavior is widely discussed in the contrastive learning literature (Jing et al., 2022; Hua et al., 2021; Chen & He, 2021;
Revised: Actually, this dimensional collapse behavior is widely discussed in the contrastive learning literature (Jing et al., 2022; Hua et al., 2021; Chen & He, 2021;

Original: Grill et al., 2020), which will hopefully help us characterize the oversmoothing of GNNs and Transformers.
Revised: Grill et al., 2020), which will hopefully help us characterize the oversmoothing of GNNs and Transformers.

Original: The main idea of contrastive learning is maximizing agreement between different augmented views of the same data example (i.e. positive pairs) via a contrastive loss.
Revised: The main idea of contrastive learning is maximizing agreement between different augmented views of the same data example (i.e. positive pairs) via a contrastive loss.

Original: Common contrastive loss can be decoupled into alignment loss and uniformity loss (Wang & Isola, 2020).
Revised: Common contrastive loss can be decoupled into alignment loss and uniformity loss (Wang & Isola, 2020).

Original: The two ingredients correspond to different objectives: alignment loss expects distance between positive pairs to be closer, while uniformity loss measures how well the embeddings are uniformly distributed.
Revised: The two ingredients correspond to different objectives: alignment loss expects distance between positive pairs to be closer, while uniformity loss measures how well the embeddings are uniformly distributed.

Original: Pure training with only alignment loss may lead to a trivial solution where all representations shrink to one single point.
Revised: Pure training with only alignment loss may lead to a trivial solution where all representations shrink to one single point.

Original: Fortunately, the existence of uniformity loss naturally helps solve this problem by drawing representations evenly distributed in the embedding space.
Revised: Fortunately, the existence of uniformity loss naturally helps solve this problem by drawing representations evenly distributed in the embedding space.

Original: In view of the inherent similarities of the two domains, we build a connection between the oversmoothing problem and the representation collapse issue.
Revised: In view of the inherent similarities of the two domains, we build a connection between the oversmoothing problem and the representation collapse issue.

Original: Instead of directly adding uniformity loss into model training, we design a normalization layer that can be used out of the box with almost no parameter overhead.
Revised: Instead of directly adding uniformity loss into model training, we design a normalization layer that can be used out of the box with almost no parameter overhead.

Original: For this end, we first transfer the uniformity loss used for training to a loss defined over graph node representations, thus it can optimize representation itself rather than model parameters.
Revised: For this end, we first transfer the uniformity loss used for training to a loss defined over graph node representations, thus it can optimize representation itself rather than model parameters.

Original: Intuitively, the loss meets the need of drawing a uniform node distribution.
Revised: Intuitively, the loss meets the need of drawing a uniform node distribution.

Original: Following research in combining optimization scheme and model architecture (Yang et al., 2021;
Revised: Following research in combining optimization scheme and model architecture (Yang et al., 2021;

Original: Zhu et al., 2021; Xie et al., 2021), we use the transferred uniformity loss as an energy function underlying our proposed layer, such that descent steps along it corresponds with the forward pass.
Revised: Zhu et al., 2021; Xie et al., 2021), we use the transferred uniformity loss as an energy function underlying our proposed layer, such that descent steps along it corresponds with the forward pass.

Original: By analyzing the unfolding iterations of the principled uniformity loss, we design a new normalization layer ContraNorm .
Revised: By analyzing the unfolding iterations of the principled uniformity loss, we design a new normalization layer ContraNorm .

Original: As a proof of concept, Figure 1 demonstrates that ContraNorm makes the features away from each other, which eases the dimensional collapse.
Revised: As a proof of concept, Figure 1 demonstrates that ContraNorm makes the features away from each other, which eases the dimensional collapse.

Original: Theoretically, we prove ContraNorm increases both average variance and effective rank of representations, thus solving complete collapse and dimensional collapse effectively.
Revised: Theoretically, we prove ContraNorm increases both average variance and effective rank of representations, thus solving complete collapse and dimensional collapse effectively.

Original: We also conduct a comprehensive evaluation of ContraNorm on various tasks.
Revised: We also conduct a comprehensive evaluation of ContraNorm on various tasks.

Original: Specifically, we boost the average performance of BERT (Devlin et al., 2018) from 82.59% to 83.54% on the validation set of General Language Understanding Evaluation (GLUE) datasets (Wang et al., 2019), and raise the test accuracy of DeiT (Touvron et al., 2021) with 24 blocks from 77.69% to 78.67% on ImageNet1k (Russakovsky et al., 2015) dataset.
Revised: Specifically, we boost the average performance of BERT (Devlin et al., 2018) from 82.59% to 83.54% on the validation set of General Language Understanding Evaluation (GLUE) datasets (Wang et al., 2019), and raise the test accuracy of DeiT (Touvron et al., 2021) with 24 blocks from 77.69% to 78.67% on ImageNet1k (Russakovsky et al., 2015) dataset.

Original: For GNNs, experiments are conducted on fully supervised graph node classification tasks, and our proposed model outperforms vanilla Graph Convolution Network (GCN)
Revised: For GNNs, experiments are conducted on fully supervised graph node classification tasks, and our proposed model outperforms vanilla Graph Convolution Network (GCN)

Original: (Kipf &
Revised: (Kipf &

Original: Welling, 2017) on all depth settings.
Revised: Welling, 2017) on all depth settings.

Original: Our contributions are summarized as:
Revised: Our contributions are summarized as:

Original: • We dissect the limitations of existing oversmoothing analysis, and highlight the importance of incorporating the dimensional collapse issue into consideration.
Revised: • We dissect the limitations of existing oversmoothing analysis, and highlight the importance of incorporating the dimensional collapse issue into consideration.

Original: • Inspired by the techniques from contrastive learning to measure and resolve oversmoothing, we propose Contrastive Normalization (ContraNorm) as an optimization-induced normalization layer to prevent dimensional collapse.
Revised: • Inspired by the techniques from contrastive learning to measure and resolve oversmoothing, we propose Contrastive Normalization (ContraNorm) as an optimization-induced normalization layer to prevent dimensional collapse.

Original: • Experiments on a wide range of tasks show that ContraNorm can effectively mitigate dimensional collapse in various model variants, and demonstrate clear benefits across three different scenarios: ViT for image classification, BERT for natural language understanding, and GNNs for node classifications.
Revised: • Experiments on a wide range of tasks show that ContraNorm can effectively mitigate dimensional collapse in various model variants, and demonstrate clear benefits across three different scenarios: ViT for image classification, BERT for natural language understanding, and GNNs for node classifications.

Original:  B ACKGROUND & R ELATED W ORK
Revised:  B ACKGROUND & R ELATED W ORK

Original: Message Passing in Graph Neural Networks.
Revised: Message Passing in Graph Neural Networks.

Original: In the literature of GNNs, message-passing graph neural networks (MP-GNNs) are intensively studied.
Revised: In the literature of GNNs, message-passing graph neural networks (MP-GNNs) are intensively studied.

Original: It progressively updates representations by exchanging information with neighbors.
Revised: It progressively updates representations by exchanging information with neighbors.

Original: The update of node i ’s representation in l -th layer is formalized as h ( l )
Revised: The update of node i ’s representation in l -th layer is formalized as h ( l )

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original:  N 
Revised: where N (

Original: [Equation]
Revised: [Equation]

Original: i ) } ) (cid:1) , where N ( i ) denotes the neighborhood set of node i , AGGREGATE ( · ) is the procedure where nodes exchange message, and UPDATE ( · ) is often a multi-layer perceptron (MLP).
Revised: ) denotes the neighborhood set of node i , AGGREGATE ( · ) is the procedure where nodes exchange message, and UPDATE ( · ) is often a multi-layer perceptron (MLP).

Original: A classical MP-GNNs model is GCN (Kipf & Welling, 2017), which propagates messages between 1-hop neighbors using an adjacency matrix.
Revised: A classical MP-GNNs model is GCN (Kipf & Welling, 2017), which propagates messages between 1-hop neighbors using an adjacency matrix.

Original: Self-Attention in Transformers.
Revised: Self-Attention in Transformers.

Original: Transformers encode information in a global scheme with selfattention as the key ingredient.
Revised: Transformers encode information in a global scheme with selfattention as the key ingredient.

Original: Self-attention module re-weights intermediate representations by aggregating semantically near neighbors.
Revised: Self-attention module re-weights intermediate representations by aggregating semantically near neighbors.

Original: Formally, it estimates similarities between key and query, namely self-attention matrix, as ¯ A = softmax ( QK ⊤ ) , Q = XW Q , and K = XW K where X , W Q , and W K are the input, weight matrices for query and key, respectively.
Revised: Formally, it estimates similarities between key and query, namely self-attention matrix, as ¯ A = softmax ( QK ⊤ ) , Q = XW Q , and K = XW K where X , W Q , and W K are the input, weight matrices for query and key, respectively.

Original: A multi-head self-attention module with residual connection can be formularized as attn ( X ) =
Revised: A multi-head self-attention module with residual connection can be formularized as attn ( X ) =

Original: X + (cid:80) hk
Revised: X + (cid:80) hk

Original: =1 ¯ A k XV k W ⊤ k , where h is the number of heads, and V , W are weights for value and final output.
Revised: =1 ¯ A k XV k W ⊤ k , where h is the number of heads, and V , W are weights for value and final output.

Original: Connections between Message Passing and Self-Attention.
Revised: Connections between Message Passing and Self-Attention.

Original: Note that the self-attention matrix can be seen as a normalized adjacent matrix of a corresponding graph (Shi et al., 2022).
Revised: Note that the self-attention matrix can be seen as a normalized adjacent matrix of a corresponding graph (Shi et al., 2022).

Original: Considering a weighted fully connected graph G with adjacency matrix denoted as ˆ A , we map nodes to token representations and set weights of the edge between node i and node j to exp( Q ⊤ i K j ) .
Revised: Considering a weighted fully connected graph G with adjacency matrix denoted as ˆ A , we map nodes to token representations and set weights of the edge between node i and node j to exp( Q ⊤ i K j ) .

Original: Then ( i, j ) entry of a normalized adjacency matrix is ˜
Revised: Then ( i, j ) entry of a normalized adjacency matrix is ˜

Original: A ij =
Revised: A ij =

Original: ˆ A ij / ˆ D ij = exp( Q ⊤ i K j ) / (cid:80) k exp( Q ⊤ i K k ) , where diagonal matrix ˆ D ii =
Revised: ˆ A ij / ˆ D ij = exp( Q ⊤ i K j ) / (cid:80) k exp( Q ⊤ i K k ) , where diagonal matrix ˆ D ii =

Original: (cid:80)
Revised: (cid:80)

Original: j ˆ A ij .
Revised: j ˆ A ij .

Original: Apparently, ˜ A is equal to the form of self-attention matrix defined in Transformers.
Revised: Apparently, ˜ A is equal to the form of self-attention matrix defined in Transformers.

Original: Simultaneously, ˜ A plays a major part in the message-passing scheme by deciding which nodes to exchange information with.
Revised: Simultaneously, ˜ A plays a major part in the message-passing scheme by deciding which nodes to exchange information with.

Original: Oversmoothing in GNNs and Transformers.
Revised: Oversmoothing in GNNs and Transformers.

Original: The term oversmoothing is firstly proposed by Li et al.
Revised: The term oversmoothing is firstly proposed by Li et al.

Original: (2018) in research of GNNs.
Revised: (2018) in research of GNNs.

Original: Intuitively, representations converge to a constant after repeatedly exchanging message with neighbors when layer goes to infinity.
Revised: Intuitively, representations converge to a constant after repeatedly exchanging message with neighbors when layer goes to infinity.

Original: Zhou et al.
Revised: Zhou et al.

Original: (2020) mathematically proves that under some conditions, the convergence point carries only information of graph topology.
Revised: (2020) mathematically proves that under some conditions, the convergence point carries only information of graph topology.

Original: Coincidentally, an oversmoothing-like phenomenon is also observed in Transformers.
Revised: Coincidentally, an oversmoothing-like phenomenon is also observed in Transformers.

Original: Unlike
Revised: Unlike

Original: CNNs, Transformers can not benefit from simply deepening layers, and even saturates with depth increasing.
Revised: CNNs, Transformers can not benefit from simply deepening layers, and even saturates with depth increasing.

Original: Early works empirically ascribe it to attention/feature collapse or patch/token uniformity (Tang et al., 2021;
Revised: Early works empirically ascribe it to attention/feature collapse or patch/token uniformity (Tang et al., 2021;

Original: Zhou et al., 2021;
Revised: Zhou et al., 2021;

Original: Gong et al., 2021).
Revised: Gong et al., 2021; Yan et al., 2022).

Original: To be specific, the attention maps tend to be overly similar in later layers, whereby features insufficiently exchange information and lose diversity in the end.
Revised: To be specific, the attention maps tend to be overly similar in later layers, whereby features insufficiently exchange information and lose diversity in the end.

Original: Outputs of pure transformers, i.e., attention without skip connections or MLPs, are also observed to converge to a rank-1 matrix (Dong et al., 2021).
Revised: Outputs of pure transformers, i.e., attention without skip connections or MLPs, are also observed to converge to a rank-1 matrix (Dong et al., 2021).

Original: For illustration proposes, we also refer to the degeneration issue in Transformers as oversmoothing.
Revised: For illustration proposes, we also refer to the degeneration issue in Transformers as oversmoothing.

Original: Whitening Methods for Dimensional Collapse.
Revised: Whitening Methods for Dimensional Collapse.

Original: Whitening methods ensure the covariance matrix of normalized outputs to be diagonal and make dimensions mutually independent to each other, which implicitly solves dimensional collapse.
Revised: Whitening methods ensure the covariance matrix of normalized outputs to be diagonal and make dimensions mutually independent to each other, which implicitly solves dimensional collapse.

Original: Huang et al.
Revised: Huang et al.

Original: (2018; 2019); Siarohin et al.
Revised: (2018; 2019); Siarohin et al.

Original: Ermolov et al.
Revised: Ermolov et al.

Original: (2021) all adopt the idea of whitening, but differ in calculating details of whitening matrix and application domain.
Revised: (2021) all adopt the idea of whitening, but differ in calculating details of whitening matrix and application domain.

Original: Compared with them, we avoid complicated calculations on the squared root of the inverse covariance matrix and delicate design of backward pass for differentiability.
Revised: Compared with them, we avoid complicated calculations on the squared root of the inverse covariance matrix and delicate design of backward pass for differentiability.

Original: Moreover, Huang et al.
Revised: Moreover, Huang et al.

Original: (2018; 2019) are proposed for convolution operations, Siarohin et al.
Revised: (2018; 2019) are proposed for convolution operations, Siarohin et al.

Original: (2018) is for GAN, and Ermolov et al.
Revised: (2018) is for GAN, and Ermolov et al.

Original: (2021) works in self-supervised learning.
Revised: (2021) works in self-supervised learning.

Original: In contrast, we borrow the idea from contrastive learning and solve oversmoothing in completely different fields like Transformers and GNNs.
Revised: In contrast, we borrow the idea from contrastive learning and solve oversmoothing in completely different fields like Transformers and GNNs.

Original:  M ITIGATING O VERSMOOTHING FROM THE P ERSPECTIVE OF C ONTRASTIVE L EARNING
Revised:  M ITIGATING O VERSMOOTHING FROM THE P ERSPECTIVE OF C ONTRASTIVE L EARNING

Original: In this section, we first empirically illustrate that current similarity metric fails to characterize dimensional collapse, thus overlooking a crucial part of oversmoothing.
Revised: In this section, we first empirically illustrate that current similarity metric fails to characterize dimensional collapse, thus overlooking a crucial part of oversmoothing.

Original: In order to solve such dimensional collapse, inspired by contrastive learning whose property of uniformity naturally rules out dimensional collapse, we transfer the uniformity loss to a loss that directly acts on representations, and then by unfolding the optimization steps along such loss we induce a normalization layer ContraNorm.
Revised: In order to solve such dimensional collapse, inspired by contrastive learning whose property of uniformity naturally rules out dimensional collapse, we transfer the uniformity loss to a loss that directly acts on representations, and then by unfolding the optimization steps along such loss we induce a normalization layer ContraNorm.

Original: Theoretically, we prove our proposed layer help mitigate dimensional collapse.
Revised: Theoretically, we prove our proposed layer help mitigate dimensional collapse.

Original: 3. T HE C HARACTERIZATION OF O VERSMOOTHING
Revised: 3. T HE C HARACTERIZATION OF O VERSMOOTHING

Original: In this part, we first analyze the defects of current metrics to characterize oversmoothing, which then inspires us to adopt the effective rank metric from contrastive learning to do this.
Revised: In this part, we first analyze the defects of current metrics to characterize oversmoothing, which then inspires us to adopt the effective rank metric from contrastive learning to do this.

Original: Taking the oversmoothing problem of Transformers as an example without loss of generality, a prevailing metric is evaluating attention map similarity (Wang et al., 2022;
Revised: Taking the oversmoothing problem of Transformers as an example without loss of generality, a prevailing metric is evaluating attention map similarity (Wang et al., 2022;

Original: Gong et al., 2021; Shi et al., 2022).
Revised: Gong et al., 2021; Shi et al., 2022).

Original: As attention map similarity progressively converges to one, attendant high feature similarity brings about loss of representation expressiveness, thus performance declines in the end.
Revised: As attention map similarity progressively converges to one, attendant high feature similarity brings about loss of representation expressiveness, thus performance declines in the end.

Original: However, by conducting experiments on transformer structured models like ViT and BERT, we find that high attention map similarity does not necessarily result in high feature similarity.
Revised: However, by conducting experiments on transformer structured models like ViT and BERT, we find that high attention map similarity does not necessarily result in high feature similarity.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2a and Fig.
Revised: 2a and Fig.

Original: 2b, although attention map similarity is close to 0.8 or even higher, the feature similarity keeps lower than 0.5 in most cases.
Revised: 2b, although attention map similarity is close to 0.8 or even higher, the feature similarity keeps lower than 0.5 in most cases.

Original: It means the oversmoothing problem still occurs evenwith a low feature similarity. A natural conclusion comes that feature similarity can not fully depict the quality of representations and the oversmoothing problem. Intuitively, considering such a case that representations in the latent embedding space do not shrink to one single point but span a low dimensional space, feature similarity may be relatively low, but representations indeed lose expressive power. Such representation degeneration problem is widely discussed in the literature of contrastive learning and often referred to as dimensional collapse .
Revised: It means the oversmoothing problem still occurs even

Original: In 8 10 layer 
Revised: 8 10 layer index 0.

Original: [Figure]
Revised: c o s i n e s i m il a r i t y featureattention map (a) Similarity on BERT    8 10 layer index 0.

Original: c o s i n e s i m il a r i t y featureattention map 
Revised: c o s i n e s i m il a r i t y featureattention map (

Original: [Figure]
Revised: [Figure]

Original: BERT    8 10 layer index 0. c o s i n e s i m il a r i t y featureattention map
Revised: ViT 0 20 40 60 80 100 120 140 singular value103050 nu m b e r BERT+ ContraNorm (

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] 103050 nu m b e r [Figure]
Revised: Feature Distribution    8 10 layer index

Original: [Figure]

Original:    e ff e c t
Revised:    e ff e c t

Original: i v e r a n k
Revised: i v e r a n k

Revised: No norm ContraNorm

Original: [Figure]
Revised: [Figure]

Revised: with a low feature similarity.

Revised: A natural conclusion comes that feature similarity can not fully depict the quality of representations and the oversmoothing problem.

Revised: Intuitively, considering such a case that representations in the latent embedding space do not shrink to one single point but span a low dimensional space, feature similarity may be relatively low, but representations indeed lose expressive power.

Revised: Such representation degeneration problem is widely discussed in the literature of contrastive learning and often referred to as dimensional collapse .

Original:  contrastive learning, common practice to describe dimensional collapse is the vanishing distribution of singular values (Gao et al., 2019; Ethayarajh, 2019;
Revised: In contrastive learning, common practice to describe dimensional collapse is the vanishing distribution of singular values (Gao et al., 2019; Ethayarajh, 2019;

Original: Jing et al., 2022).
Revised: Jing et al., 2022).

Original: We draw the singular value distribution of features in the last block of 12-layer BERT.
Revised: We draw the singular value distribution of features in the last block of 12-layer BERT.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2c, for vanilla BERT, the insignificant (nearly zero) values dominate singular value distribution in the deep layer, which means representations reside on a low-dimensional manifold and dimensional collapse happens.
Revised: 2c, for vanilla BERT, the insignificant (nearly zero) values dominate singular value distribution in the deep layer, which means representations reside on a low-dimensional manifold and dimensional collapse happens.

Original: Here, to show the collapse tendency along layer index, we simplify the singular value distribution down to a concise scalar, namely effective rank (erank) (Roy & Vetterli, 2007), which is a spectral entropy based method covering the full singular value spectrum.
Revised: Here, to show the collapse tendency along layer index, we simplify the singular value distribution down to a concise scalar, namely effective rank (erank) (Roy & Vetterli, 2007), which is a spectral entropy based method covering the full singular value spectrum.

Original: Definition 3.1 (Effective Rank) .
Revised: Definition 3.1 (Effective Rank) .

Original: Considering matrix X ∈ R m × n whose singular value decomposition is given by X = U Σ V , where Σ is a diagonal matrix with singular values σ 1 ≥ σ 2 ≥ · · · ≥ σ Q ≥ 0 with Q = min { m, n } .
Revised: Considering matrix X ∈ R m × n whose singular value decomposition is given by X = U Σ V , where Σ is a diagonal matrix with singular values σ 1 ≥ σ 2 ≥ · · · ≥ σ Q ≥ 0 with Q = min { m, n } .

Original: The distribution of singular values is defined as L 1 -normalized form p i
Revised: The distribution of singular values is defined as L 1 -normalized form p i

Original: =
Revised: =

Original: σ i / (cid:80)
Revised: σ i / (cid:80)

Original: Qk =1 | σ k | .
Revised: Qk =1 | σ k | .

Original: The effective rank of the matrix X , denoted as erank ( X ) , is defined as erank ( X ) =
Revised: The effective rank of the matrix X , denoted as erank ( X ) , is defined as erank ( X ) =

Original: exp( H { p 1 , p 2 , · · · , p Q } ) , where H ( p 1 , p 2 , · · · , p Q ) is the Shannon entropy given by
Revised: exp( H { p 1 , p 2 , · · · , p Q } ) , where H ( p 1 , p 2 , · · · , p Q ) is the Shannon entropy given by

Original: H ( p 1 , p 2 , · · · , p Q ) = [Equation] (cid:80)
Revised: H ( p 1 , p 2 , · · · , p Q ) = − (cid:80)

Original: [Equation]
Revised: Qk =1 p k log p k .

Original: Based on erank, we revisit the oversmoothing issue of Transformers in Fig.
Revised: Based on erank, we revisit the oversmoothing issue of Transformers in Fig.

Original: 2d.
Revised: 2d.

Original: We can see that the effective rank descends along with layer index, which means a more imbalanced singular value distribution in deeper layers.
Revised: We can see that the effective rank descends along with layer index, which means a more imbalanced singular value distribution in deeper layers.

Original: The tendency not only verifies dimensional collapse does occur in
Revised: The tendency not only verifies dimensional collapse does occur in

Original: Transformers, but also indicates the effectiveness of effective rank.
Revised: Transformers, but also indicates the effectiveness of effective rank.

Original: 3. I NSPIRATIONS FROM THE U NIFORMITY L OSS IN C ONTRASTIVE L EARNING
Revised: 3. I NSPIRATIONS FROM THE U NIFORMITY L OSS IN C ONTRASTIVE L EARNING

Original: The core idea of contrastive learning is maximizing agreement between differently augmented views of the same example (i.e., positive pairs) and disagreement of views from different samples (i.e. negative pairs).
Revised: The core idea of contrastive learning is maximizing agreement between differently augmented views of the same example (i.e., positive pairs) and disagreement of views from different samples (i.e. negative pairs).

Original: A popular form of contrastive learning optimizes feature representations using a loss function with limited negative samples Chen et al.
Revised: A popular form of contrastive learning optimizes feature representations using a loss function with limited negative samples Chen et al.

Original: Concretely, given a batch of randomly sampled examples, for each example we generate its augmented positive views and finally get a total of N samples.
Revised: Concretely, given a batch of randomly sampled examples, for each example we generate its augmented positive views and finally get a total of N samples.

Original: Considering an encoder function f , the contrastive loss for the positive pair ( i, i + ) is
Revised: Considering an encoder function f , the contrastive loss for the positive pair ( i, i + ) is

Original: [Equation]
Revised: [Equation]

Original: where τ denotes a temperature factor.
Revised: where τ denotes a temperature factor.

Original: The loss can be decoupled into two parts named alignment loss and uniformity loss :
Revised: The loss can be decoupled into two parts named alignment loss and uniformity loss :

Original: [Equation]
Revised: [Equation]

Original: The alignment loss encourages feature representations for positive pairs to be similar, thus being invariant to unnecessary noises.
Revised: The alignment loss encourages feature representations for positive pairs to be similar, thus being invariant to unnecessary noises.

Original: However, training with single alignment loss will result in a trivial solution where all representations are identical.
Revised: However, training with single alignment loss will result in a trivial solution where all representations are identical.

Original: In other words, complete collapse happens.
Revised: In other words, complete collapse happens.

Original: With carefully chosen implementation details, e.g., batch normalization, the problem can be ruled out.
Revised: With carefully chosen implementation details, e.g., batch normalization, the problem can be ruled out.

Original: But unfortunately, dimensional collapse still has a negative effect on learning.
Revised: But unfortunately, dimensional collapse still has a negative effect on learning.

Original: Thanks to the property of uniformity loss, dimensional collapse can be solved effectively.
Revised: Thanks to the property of uniformity loss, dimensional collapse can be solved effectively.

Original: Reviewing the form of uniformity loss in Eq.(2), it expects average distances between all samples to be maximal, thus embeddings will be roughly uniformly distributed in the latent space and preserve much more information.
Revised: Reviewing the form of uniformity loss in Eq.(2), it expects average distances between all samples to be maximal, thus embeddings will be roughly uniformly distributed in the latent space and preserve much more information.

Original: The existence of uniformity loss helps contrastive learning ease dimensional collapse.
Revised: The existence of uniformity loss helps contrastive learning ease dimensional collapse.

Original: Intuitively, it can also serve as a sharp tool for alleviating oversmoothing in models like GNNs and
Revised: Intuitively, it can also serve as a sharp tool for alleviating oversmoothing in models like GNNs and

Original: Transformers.
Revised: Transformers.

Original: A straightforward way is to apply the uniformity loss to the training objective, but we find it performs not well enough (see Appendix D for more details).
Revised: A straightforward way is to apply the uniformity loss to the training objective, but we find it performs not well enough (see Appendix D for more details).

Original: Here, we choose to design an easy-to-use normalization layer that can be integrated into various models.
Revised: Here, we choose to design an easy-to-use normalization layer that can be integrated into various models.

Original: Concretely, the uniformity loss is used as an energy function underlying our proposed layer, such that descent steps along it corresponds with the forward pass.
Revised: Concretely, the uniformity loss is used as an energy function underlying our proposed layer, such that descent steps along it corresponds with the forward pass.

Original: Similarly, we can also view the layer as the unfolded iterations of a principled optimization function.
Revised: Similarly, we can also view the layer as the unfolded iterations of a principled optimization function.

Original: This perspective is adopted to elucidate GNNs (Yang et al., 2021;
Revised: This perspective is adopted to elucidate GNNs (Yang et al., 2021;

Original: Zhu et al., 2021),
Revised: Zhu et al., 2021),

Original: Transformers (Yang et al., 2022), and classic MLPs (Xie et al., 2021) and CNNs.
Revised: Transformers (Yang et al., 2022), and classic MLPs (Xie et al., 2021) and CNNs.

Original: Unlike previous energy functions for graph regularization or architecture explanation (Yang et al., 2022), we take the interpretable uniformity loss as energy function, so that node representations are enforced to present a more uniform distribution.
Revised: Unlike previous energy functions for graph regularization or architecture explanation (Yang et al., 2022), we take the interpretable uniformity loss as energy function, so that node representations are enforced to present a more uniform distribution.

Original: Note that the uniformity loss works by optimizing model parameters while what the normalization layer directly updates is representation itself.
Revised: Note that the uniformity loss works by optimizing model parameters while what the normalization layer directly updates is representation itself.

Original: So we first transfer the uniformity loss, which serves as a training loss, to a kind of architecture loss.
Revised: So we first transfer the uniformity loss, which serves as a training loss, to a kind of architecture loss.

Original: Consider a fully connected graph with restricted number of nodes, where node h i is viewed as representation of a random sample x i .
Revised: Consider a fully connected graph with restricted number of nodes, where node h i is viewed as representation of a random sample x i .

Original: Reminiscent of L uniform , we define ˆ L uniform over all nodes in the graph as
Revised: Reminiscent of L uniform , we define ˆ L uniform over all nodes in the graph as

Original: [Equation]
Revised: [Equation]

Original: This form of uniformity loss is defined directly on node representations, and we later use it as the underlying energy function for representation update.
Revised: This form of uniformity loss is defined directly on node representations, and we later use it as the underlying energy function for representation update.

Original: 3. T HE P ROPOSED C ONTRA N ORM
Revised: 3. T HE P ROPOSED C ONTRA N ORM

Original: Till now, we are able to build a connection between layer design and the unfolded iterations of descent steps applied to minimize uniformity loss.
Revised: Till now, we are able to build a connection between layer design and the unfolded iterations of descent steps applied to minimize uniformity loss.

Original: Specifically, we take the derivative of ˆ L uniform on node representations:
Revised: Specifically, we take the derivative of ˆ L uniform on node representations:

Original: [Equation]
Revised: [Equation]

Original: by denoting feature matrix as H with h ⊤ i as the i -th row, we rewrite Eq.(4) into a matrix form for simplicity:
Revised: by denoting feature matrix as H with h ⊤ i as the i -th row, we rewrite Eq.(4) into a matrix form for simplicity:

Original: [Equation]
Revised: [Equation]

Original: where A = exp( HH ⊤ /τ ) and D = deg ( A ) 1 .
Revised: where A = exp( HH ⊤ /τ ) and D = deg ( A ) 1 .

Original: In order to reduce the uniformity loss ˆ L uniform , a natural way is to take a single step performing gradient descent, which is to update representations
Revised: In order to reduce the uniformity loss ˆ L uniform , a natural way is to take a single step performing gradient descent, which is to update representations

Original: H in the way of
Revised: H in the way of

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where H b and H t denote the representations before and after the update, respectively, and s is the step size of the gradient descent.
Revised: where H b and H t denote the representations before and after the update, respectively, and s is the step size of the gradient descent.

Original: By taking this update after a certain representation layer, we can reduce the uniformity loss of the representations and thus help ease the dimensional collapse.
Revised: By taking this update after a certain representation layer, we can reduce the uniformity loss of the representations and thus help ease the dimensional collapse.

Original: In Eq.(6), there exist two terms D − 1 A and AD − 1 multiplied with H b .
Revised: In Eq.(6), there exist two terms D − 1 A and AD − 1 multiplied with H b .

Original: Empirically, the two terms play a similar role in our method.
Revised: Empirically, the two terms play a similar role in our method.

Original: Note that the first term is related to self-attention matrix in 
Revised: Note that the first term is related to self-attention matrix in 1 deg ( A ) is

Original: Transformers, so we only preserve it and discard the second one.
Revised: Transformers, so we only preserve it and discard the second one.

Original: Then Eq.(6) becomes
Revised: Then Eq.(6) becomes

Original: [Equation]
Revised: [Equation]

Original: In fact, the operation corresponds to the stop-gradient technique, which is widely used in contrastive learning methods (He et al., 2020; Grill et al., 2020; Tao et al., 2022).
Revised: In fact, the operation corresponds to the stop-gradient technique, which is widely used in contrastive learning methods (He et al., 2020; Grill et al., 2020; Tao et al., 2022).

Original: By throwing away some terms in the gradient, stop-gradient makes the training process asymmetric and thus avoids representation collapse with less computational overhead, which is discussed in detail in Appendix A.
Revised: By throwing away some terms in the gradient, stop-gradient makes the training process asymmetric and thus avoids representation collapse with less computational overhead, which is discussed in detail in Appendix A.

Original: However, the layer induced by Eq.(7) still can not ensure uniformity on representations.
Revised: However, the layer induced by Eq.(7) still can not ensure uniformity on representations.

Original: Consider an extreme case where softmax( H b H ⊤ b ) is equal to identity matrix I . Eq.(7) becomes H t =
Revised: Consider an extreme case where softmax( H b H ⊤ b ) is equal to identity matrix I . Eq.(7) becomes H t =

Original: H b − s/
Revised: H b − s/

Original: τ × H b
Revised: τ × H b

Original: =
Revised: =

Original: (1 − s/τ ) H b , which just makes the scale of H smaller and does not help alleviate the complete collapse.
Revised: (1 − s/τ ) H b , which just makes the scale of H smaller and does not help alleviate the complete collapse.

Original: To keep the representation stable, we propose two methods:
Revised: To keep the representation stable, we propose two methods:

Original: Regularization.
Revised: Regularization.

Original: We add a regularization term − 12 (cid:80) i ∥
Revised: We add a regularization term − 12 (cid:80) i ∥

Original: h i ∥ 22 to the uniformity loss.
Revised: h i ∥ 22 to the uniformity loss.

Original: When the regularization term becomes smaller, the norm of h i becomes larger.
Revised: When the regularization term becomes smaller, the norm of h i becomes larger.

Original: Therefore, adding this term can help prevent the norm of representation h i from becoming smaller.
Revised: Therefore, adding this term can help prevent the norm of representation h i from becoming smaller.

Original: In this way, the update form becomes
Revised: In this way, the update form becomes

Original: [Equation]
Revised: [Equation]

Original: Proposition 1.
Revised: Proposition 1.

Original: Let e =
Revised: Let e =

Original: . , 1) ⊤ / √ n .
Revised: . , 1) ⊤ / √ n .

Original: For attention matrix ¯ A = softmax( H b H ⊤ b ) , let σ min be the smallest eigenvalue of P =
Revised: For attention matrix ¯ A = softmax( H b H ⊤ b ) , let σ min be the smallest eigenvalue of P =

Original: ( I − ee ⊤ )( I − ¯ A )
Revised: ( I − ee ⊤ )( I − ¯ A )

Original: +( I − ¯ A ) ⊤ ( I − ee ⊤ ) .
Revised: +( I − ¯ A ) ⊤ ( I − ee ⊤ ) .

Original: For the update in Eq.8, i.e. H t =
Revised: For the update in Eq.8, i.e. H t =

Original: ((1 + s )
Revised: ((1 + s )

Original: I − s ¯ A ) H b , s > 0 , we have V ar ( H t ) ≥ (1 + sσ min ) · V ar
Revised: I − s ¯ A ) H b , s > 0 , we have V ar ( H t ) ≥ (1 + sσ min ) · V ar

Original: H b ) .
Revised: H b ) .

Original: Especially, if σ min ≥ 0 , we have V ar ( H t ) ≥
Revised: Especially, if σ min ≥ 0 , we have V ar ( H t ) ≥

Original: V ar ( H b ) .
Revised: V ar ( H b ) .

Original: Proposition 1 gives a bound for the ratio of the variance after and before the update.
Revised: Proposition 1 gives a bound for the ratio of the variance after and before the update.

Original: It shows that the change of the variance is influenced by the symmetric matrix P
Revised: It shows that the change of the variance is influenced by the symmetric matrix P

Original: = ( I − ee ⊤ )( I − ¯ A )
Revised: = ( I − ee ⊤ )( I − ¯ A )

Original: +( I − ¯ A ) ⊤ ( I − ee ⊤ ) .
Revised: +( I − ¯ A ) ⊤ ( I − ee ⊤ ) .

Original: If P is a semi-positive definite matrix, we will get the result that V ar ( H t ) ≥
Revised: If P is a semi-positive definite matrix, we will get the result that V ar ( H t ) ≥

Original: V ar ( H b ) , which indicates that the representations become more uniform.
Revised: V ar ( H b ) , which indicates that the representations become more uniform.

Original: In Appendix B, we will give out some sufficient conditions for that P is semi-positive definite.
Revised: In Appendix B, we will give out some sufficient conditions for that P is semi-positive definite.

Original: LayerNorm.
Revised: LayerNorm.

Original: We leverage layer normalization (Ba et al., 2016), which adjusts the representation according to its mean and variance.
Revised: We leverage layer normalization (Ba et al., 2016), which adjusts the representation according to its mean and variance.

Original: The update form of the layer normalization on representation h
Revised: The update form of the layer normalization on representation h

Original: i is LayerNorm ( h i )
Revised: i is LayerNorm ( h i )

Original: = γ · (( h i − mean ( h i )) / (cid:112) Var ( h i ) + ε ) + β , where γ and β are learnable parameters and ε = 10 − 5 prevents the denominator from becoming zero.
Revised: = γ · (( h i − mean ( h i )) / (cid:112) Var ( h i ) + ε ) + β , where γ and β are learnable parameters and ε = 10 − 5 prevents the denominator from becoming zero.

Original: The learnable parameters γ and β can rescale the representation h i to help ease the problem.
Revised: The learnable parameters γ and β can rescale the representation h i to help ease the problem.

Original: We append the layer normalization to Eq.(7) as
Revised: We append the layer normalization to Eq.(7) as

Original: [Equation]
Revised: [Equation]

Original: where applying the layer normalization to a representation matrix H means applying the layer normalization to all its components h 1 , . .
Revised: where applying the layer normalization to a representation matrix H means applying the layer normalization to all its components h 1 , . .

Original: , h n .
Revised: , h n .

Original: We empirically compare the two proposed methods and find their performance comparable, while the second one performs slightly better.
Revised: We empirically compare the two proposed methods and find their performance comparable, while the second one performs slightly better.

Original: Therefore, we adopt the second update form and name it Contrastive Normalization ( ContraNorm ).
Revised: Therefore, we adopt the second update form and name it Contrastive Normalization ( ContraNorm ).

Original: The ContraNorm layer can be added after any representation layer to reduce the uniformity loss and help relieve the dimensional collapse.
Revised: The ContraNorm layer can be added after any representation layer to reduce the uniformity loss and help relieve the dimensional collapse.

Original: We discuss the best place to plug our ContraNorm in Appendix E.
Revised: We discuss the best place to plug our ContraNorm in Appendix E.

Revised: Given H b ∈ R n × d where n and d is the number of samples in the batch and hidden dimension, the time complexity of ContraNorm is O ( n 2 d ) , which is the same order as the self-attention operation in Transformer.

Revised: For enhancing scalability, we propose a modified version of ContraNorm with a linear complexity in the number of samples.

Revised: See Appendix F for more details.

Original: 3. T HEORETICAL A NALYSIS
Revised: 3. T HEORETICAL A NALYSIS

Original: In this part, we will give a theoretical result on our proposed ContraNorm method.
Revised: In this part, we will give a theoretical result on our proposed ContraNorm method.

Original: We will show that with a different form of the uniformity loss, the ContraNorm update can help alleviate the dimensional collapse.
Revised: We will show that with a different form of the uniformity loss, the ContraNorm update can help alleviate the dimensional collapse.

Original: Proposition 2.
Revised: Proposition 2.

Original: Consider the update form
Revised: Consider the update form

Original: [Equation]
Revised: [Equation]

Original: (10) let σ max be the largest singular value of H b .
Revised: let σ max be the largest singular value of H b .

Original: For s > 0 satisfying 1 + (1 − σ 2max ) s > 0 , we have erank( H t ) >
Revised: For s > 0 satisfying 1 + (1 − σ 2max ) s > 0 , we have erank( H t ) >

Original: erank( H b ) .
Revised: erank( H b ) .

Original: Proposition 2 gives a promise of alleviating dimensional collapse under a special update form.
Revised: Proposition 2 gives a promise of alleviating dimensional collapse under a special update form.

Original: Denote ˆ L dim = tr (( I − HH ⊤ ) 2 ) / 4 , then ˆ L dim shares a similar form with Barlow Twins (Zbontar et al., 2021).
Revised: Denote ˆ L dim = tr (( I − HH ⊤ ) 2 ) / 4 , then ˆ L dim shares a similar form with Barlow Twins (Zbontar et al., 2021).

Revised: This loss tries to equate the diagonal elements of the similarity matrix HH ⊤

Revised: [Figure]

Revised: Fig.(3a) presents performances of models with block number=12, 24, 36, 48, 60, 72, 84.

Original: This loss tries to equate the diagonal elements of the similarity matrix HH ⊤ to 1 and equate the off-diagonal elements of the matrix to 0, which drives different features becoming orthogonal, thus helping the features become more uniform in the space.
Revised: Fig.(3b) and Fig.(3c) show the tendency of features variance and effective rank for models with 72 blocks. to 1 and equate the off-diagonal elements of the matrix to 0, which drives different features becoming orthogonal, thus helping the features become more uniform in the space.

Original: Note that ∂ ˆ L dim ∂ H =
Revised: Note that ∂ ˆ L dim ∂ H =

Original: ( I − HH ⊤ ) H , therefore, Eq.(10) can be rewritten as H t
Revised: ( I − HH ⊤ ) H , therefore, Eq.(10) can be rewritten as H t

Original: = H b
Revised: = H [Equation] s

Original: + s ∂ ˆ L dim ( H b ) ∂ H b , which implies that this update form is in fact a ContraNorm with a different uniformity loss.
Revised: [Equation] L [Equation] H b , which implies that this update form is in fact a ContraNorm with a different uniformity loss.

Original: Propositionclaims that this update can increase the effective rank of the representation matrix, when s satisfies 1+(1 − σ max ) s > 0 .
Revised: Propositionclaims that this update can increase the effective rank of the representation matrix, when s satisfies 1+(1 − σ max ) s > 0 .

Original: Note that no matter whether σ max > 0 or not, if s is sufficiently close to 0 , the condition will be satisfied.
Revised: Note that no matter whether σ max > 0 or not, if s is sufficiently close to 0 , the condition will be satisfied.

Original: Under this situation, the update will alleviate the dimensional collapse.
Revised: Under this situation, the update will alleviate the dimensional collapse.

Original: [Figure]

Original: Fig.(3a) presents performances of models with block number=12, 24, 36, 48, 60, 72, 84.

Original: Fig.(3b) and Fig.(3c) show the tendency of features variance and effective rank for models with 72 blocks

Original: [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we demonstrate the effectiveness of ContraNorm by experiments including 1) language understanding tasks on GLUE datasets with BERT and ALBERT (Lan et al., 2020) as the backbones; 2) image classification task on ImageNet100 and ImageNet1k datasets with DeiT as the base model; 3) fully supervised graph node classification task on popular graph datasets with GCN as the backbone.
Revised: In this section, we demonstrate the effectiveness of ContraNorm by experiments including 1) language understanding tasks on GLUE datasets with BERT and ALBERT (Lan et al., 2020) as the backbones; 2) image classification task on ImageNet100 and ImageNet1k datasets with DeiT as the base model; 3) fully supervised graph node classification task on popular graph datasets with GCN as the backbone.

Original: Moreover, we conduct ablative studies on ContraNorm variants comparison and hyperparameters sensitivity analysis.
Revised: Moreover, we conduct ablative studies on ContraNorm variants comparison and hyperparameters sensitivity analysis.

Original: 4. E XPERIMENTS ON L ANGUAGE M ODELS
Revised: 4. E XPERIMENTS ON L ANGUAGE M ODELS

Original: To corroborate the potential of ContraNorm, we integrate it into two transformer structured models:
Revised: To corroborate the potential of ContraNorm, we integrate it into two transformer structured models:

Original: BERT and ALBERT, and evaluate it on GLUE datasets.
Revised: BERT and ALBERT, and evaluate it on GLUE datasets.

Original: GLUE includes three categories of tasks: (i) single-sentence tasks CoLA and SST-2; (ii) similarity and paraphrase tasks MRPC, QQP, and
Revised: GLUE includes three categories of tasks: (i) single-sentence tasks CoLA and SST-2; (ii) similarity and paraphrase tasks MRPC, QQP, and

Original: STS-B; (iii) inference tasks MNLI, QNLI, and RTE.
Revised: STS-B; (iii) inference tasks MNLI, QNLI, and RTE.

Original: For MNLI task, we experiment on both the matched (MNLI-m) and mismatched (MNLI-mm) versions.
Revised: For MNLI task, we experiment on both the matched (MNLI-m) and mismatched (MNLI-mm) versions.

Original: Setup.
Revised: Setup.

Original: We default plug ContraNorm after the self-attention module and residual connection (more position choices are in Appendix E).
Revised: We default plug ContraNorm after the self-attention module and residual connection (more position choices are in Appendix E).

Original: We use a batch size of 32 and fine-tune for 5 epochs over the data for all GLUE tasks.
Revised: We use a batch size of 32 and fine-tune for 5 epochs over the data for all GLUE tasks.

Original: For each task, we select the best scale factor s in Eq.(6) among (0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2) .
Revised: For each task, we select the best scale factor s in Eq.(6) among (0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2) .

Original: We use base models (BERT-base and ALBERT-base) of 12 stacked blocks with hyperparameters fixed for all tasks: number of hidden size 128, number of attention heads 8, maximum sequence length 384.
Revised: We use base models (BERT-base and ALBERT-base) of 12 stacked blocks with hyperparameters fixed for all tasks: number of hidden size 128, number of attention heads 12, maximum sequence length 384.

Original: We use Adam (Kingma & Ba, 2014) optimizer with theinitial learning rate 0.0001 and the warm-up technique.
Revised: We use Adam (Kingma & Ba, 2014) optimizer with the learning rate of 2 e − 5 .

Original: Results.
Revised: Results.

Original: As shown in Table 1, ContraNorm substantially improves results on all datasets compared with vanilla BERT.
Revised: As shown in Table 1, ContraNorm substantially improves results on all datasets compared with vanilla BERT.

Original: Specifically, our ContraNorm improves the previous average performance from 82.59% to 83.39% for BERT backbone and from 83.74% to 84.67% for ALBERT backbone.
Revised: Specifically, our ContraNorm improves the previous average performance from 82.59% to 83.39% for BERT backbone and from 83.74% to 84.67% for ALBERT backbone.

Original: We also submit our trained model to GLUE benchmark leaderboard and the results can be found in Appendix
Revised: We also submit our trained model to GLUE benchmark leaderboard and the results can be found in Appendix

Original: F. It is observed that BERT with ContraNorm also outperforms vanilla model across all datasets.
Revised: G. It is observed that BERT with ContraNorm also outperforms vanilla model across all datasets.

Original: To verify the de-oversmoothing effect of ContraNorm.
Revised: To verify the de-oversmoothing effect of ContraNorm.

Original: We build models with/without ContraNorm on various layer depth settings.
Revised: We build models with/without ContraNorm on various layer depth settings.

Original: The performance comparison is shown in Fig. 3a.
Revised: The performance comparison is shown in Fig. 3a.

Original: Constant stack of blocks causes obvious deterioration in vanilla BERT, while BERT with ContraNorm maintains competitive advantage.
Revised: Constant stack of blocks causes obvious deterioration in vanilla BERT, while BERT with ContraNorm maintains competitive advantage.

Original: Moreover, for deep models, we also show the tendency of variance and effective rank in Fig.
Revised: Moreover, for deep models, we also show the tendency of variance and effective rank in Fig.

Original: 3b and Fig.
Revised: 3b and Fig.

Original: 3c, which verifies the power of ContraNorm in alleviating complete collapse and dimensional collapse, respectively. 
Revised: 3c, which verifies the power of ContraNorm in alleviating complete collapse and dimensional collapse, respectively. [Table] [Table] # L= # L= # L=

Original: [Table]
Revised: [Table]

Revised: DeiT

Revised: [Table]

Revised: [Table]

Revised: +ContraNorm 79.

Revised: [Table]

Revised: [Table]

Original: 4. E XPERIMENTS ON V ISION T RANSFORMERS
Revised: 4. E XPERIMENTS ON V ISION T RANSFORMERS

Original: We also validate effectiveness of ContraNorm in computer vision field.
Revised: We also validate effectiveness of ContraNorm in computer vision field.

Original: We choose DeiT as backbone and models are trained from scratch.
Revised: We choose DeiT as backbone and models are trained from scratch.

Original: Experiments with different depth settings are evaluated on ImageNet100 and ImageNet1k datasets.
Revised: Experiments with different depth settings are evaluated on ImageNet100 and ImageNet1k datasets.

Original: Based on the Timm (Wightman, 2019) and DeiT repositories, we insert ContraNorm into base model intermediately following self-attention module.
Revised: Based on the Timm (Wightman, 2019) and DeiT repositories, we insert ContraNorm into base model intermediately following self-attention module.

Original: Setup.
Revised: Setup.

Original: We follow the training recipe of Touvron et al.
Revised: We follow the training recipe of Touvron et al.

Original: (2021), and make minimal changes to hyperparameters.
Revised: (2021), and make minimal changes to hyperparameters.

Original: Specifically, we use AdamW (Loshchilov & Hutter, 2019) optimizer with cosine learning rate decay.
Revised: Specifically, we use AdamW (Loshchilov & Hutter, 2019) optimizer with cosine learning rate decay.

Original: We train each model for 150 epochs and the batch size is set to 1024.
Revised: We train each model for 150 epochs and the batch size is set to 1024.

Original: Augmentation techniques are used to boost the performance.
Revised: Augmentation techniques are used to boost the performance.

Original: For all experiments, the image size is set to be 224 × 224 .
Revised: For all experiments, the image size is set to be 224 × 224 .

Original: For Imagenet100, we set the scale factor s to 0.2 for all layer depth settings.
Revised: For Imagenet100, we set the scale factor s to 0.2 for all layer depth settings.

Original: For
Revised: For

Original: Imagenet1k, we set s to 0.2 for models with 12 and 16 blocks, and raise it to 0.3 for models withblocks.
Revised: Imagenet1k, we set s to 0.2 for models with 12 and 16 blocks, and raise it to 0.3 for models withblocks.

Original: Results.
Revised: Results.

Original: In Table 2, DeiT with ContraNorm outperforms vanilla DeiT with all layer settings on both datasets.
Revised: In Table 2, DeiT with ContraNorm outperforms vanilla DeiT with all layer settings on both datasets.

Original: Typically, our method shows a gain of nearly 5% on test accuracy for ImageNet100.
Revised: Typically, our method shows a gain of nearly 5% on test accuracy for ImageNet100.

Original: For
Revised: For

Original: ImageNet1k, we boost the performance of DeiT with 24 blocks from 77.69% to 78.67%.
Revised: ImageNet1k, we boost the performance of DeiT with 24 blocks from 77.69% to 78.67%.

Original: 4. E XPERIMENTS ON G RAPH N EURAL N ETWORKS
Revised: 4. E XPERIMENTS ON G RAPH N EURAL N ETWORKS

Original: We implement ContraNorm as a simple normalization layer after each graph convolution of GCN, and evaluate it on fully supervised graph node classification task.
Revised: We implement ContraNorm as a simple normalization layer after each graph convolution of GCN, and evaluate it on fully supervised graph node classification task.

Original: For datasets, we choose two popular citation networks Cora (McCallum et al., 2000) and Citeseer (Giles et al., 1998), and Wikipedia article networks Chameleon and Squirrel (Rozemberczki et al., 2021).
Revised: For datasets, we choose two popular citation networks Cora (McCallum et al., 2000) and Citeseer (Giles et al., 1998), and Wikipedia article networks Chameleon and Squirrel (Rozemberczki et al., 2021).

Original: More information of datasets is deferred to Appendix G. We compare ContraNorm against a popular normalization layer designed for preventing GCN oversmoothing PairNorm (Zhao & Akoglu, 2020).
Revised: More information of datasets is deferred to Appendix H. We compare ContraNorm against a popular normalization layer designed for preventing GCN oversmoothing PairNorm (Zhao & Akoglu, 2020).

Original: We also take LayerNorm as a baseline by setting the scale s = 0 .
Revised: We also take LayerNorm as a baseline by setting the scale s = 0 .

Original: Setup.
Revised: Setup.

Original: We follow data split setting in Kipf & Welling (2017) with train/validation/test splits of 60%, 20%, 20%, respectively.
Revised: We follow data split setting in Kipf & Welling (2017) with train/validation/test splits of 60%, 20%, 20%, respectively.

Original: To keep fair in comparison, we fix the number of channels to 32, dropout rate to 0.6 and L 2 regularization with penalty 5 · 10 − 4 as reported in Zhao & Akoglu (2020).
Revised: To keep fair in comparison, we fix the number of channels to 32, dropout rate to 0.6 and L 2 regularization with penalty 5 · 10 − 4 as reported in Zhao & Akoglu (2020).

Original: We choose the best of scale controller s in range of { 0 . 2 , 0 . 5 , 0 . 8 , 1 . 0 } for both PairNorm and ContraNorm.
Revised: We choose the best of scale controller s in range of { 0 . 2 , 0 . 5 , 0 . 8 , 1 . 0 } for both PairNorm and ContraNorm.

Original: For PairNorm, we choose the best variant presented by Zhao & Akoglu (2020).
Revised: For PairNorm, we choose the best variant presented by Zhao & Akoglu (2020).

Original: Results.
Revised: Results.

Original: As shown in Table 3, in shallow layers (e.g., two layer)
Revised: As shown in Table 3, in shallow layers (e.g., two layer)

Original: , the addition of ContraNorm reduces the accuracy of vanilla GCN by a small margin, while it helps prevent the performance from sharply deteriorating as the layer goes deeper.
Revised: , the addition of ContraNorm reduces the accuracy of vanilla GCN by a small margin, while it helps prevent the performance from sharply deteriorating as the layer goes deeper.

Original: ContraNorm outperforms PairNorm and LayerNorm in the power of de-oversmoothing.
Revised: ContraNorm outperforms PairNorm and LayerNorm in the power of de-oversmoothing.

Original: Here, we show the staple in diluting oversmoothing is
Revised: Here, we show the staple in diluting oversmoothing is

Original: ContraNorm, and LayerNorm alone fails to prevent GCN from oversmoothing, even amplifying the negative effect on Cora with more than 16 layers. 
Revised: ContraNorm, and LayerNorm alone fails to prevent GCN from oversmoothing, even amplifying the negative effect on Cora with more than 16 layers. [Table] [Table] [Table]

Original: 4. A BLATION S TUDY
Revised: A BLATION S TUDY

Original: Comparison of different ContraNorm variants.
Revised: Comparison of different ContraNorm variants.

Original: Recalling Section 3.3, we improve ContraNorm with stop-gradient technique (SG) by masking the second term.
Revised: Recalling Section 3.3, we improve ContraNorm with stop-gradient technique (SG) by masking the second term.

Original: For solving data representation instability, we apply layer normalization (LN) to the original version, while for the convenience of theoretical analysis, layer normalization is replaced by L 2 normalization ( L 2 N).
Revised: For solving data representation instability, we apply layer normalization (LN) to the original version, while for the convenience of theoretical analysis, layer normalization is replaced by L 2 normalization ( L 2 N).

Original: Here, we investigate the effect of these tricks and the results are shown in Table 4.
Revised: Here, we investigate the effect of these tricks and the results are shown in Table 4.

Original: Compared with the variant with only LayerNorm, ContraNorm with both stop gradient and layer normalization presents better performance.
Revised: Compared with the variant with only LayerNorm, ContraNorm with both stop gradient and layer normalization presents better performance.

Original: As for the two normalization methods, they are almost comparable to our methods, which verifies the applicability of our theoretical analysis.[Table] [Table]
Revised: As for the two normalization methods, they are almost comparable to our methods, which verifies the applicability of our theoretical analysis.

Original: 83.11 ✓ ✓ 59.

Original: 68.95 83.29 ✓ ✓ 59.

Original: [-0.02 0.04 0.06 0.08 0.10 scale     a cc u r a c y (a) RTE 0.02 0.04 0.
-]
Revised: [-0.02 0.04 0.06 0.08 0.10 scale     a cc u r a c y (a) RTE 0.02 0.04 0.
-]

Original: 0.08 0.10 scale    m a tt h e w s c

Original: o

Original: rr

Original: e l a t i o n (b) CoLA 0.02 0.04 0.06 0.08 0.10 scale     s p e a r m a n r (c) STS-B 0.02 0.04 0.06 0.08 0.10 scale

Original: F(d) MRPC

Original: Hyperparameters sensitivity analysis.
Revised: Hyperparameters sensitivity analysis.

Original: We conduct an ablation study regarding the gains of ContraNorm while using different values of scale factor s .
Revised: We conduct an ablation study regarding the gains of ContraNorm while using different values of scale factor s .

Original: Experiments are conducted on GLUE datasets.
Revised: Experiments are conducted on GLUE datasets.

Original: For each dataset, we vary the normalization scale around the best choice and report the test performance.
Revised: For each dataset, we vary the normalization scale around the best choice and report the test performance.

Original: The results in Fig.
Revised: The results in Fig.

Original: 4 illustrate that model with ContraNorm is robust in an appropriate range of normalization scaling factor.
Revised: 5 illustrate that model with ContraNorm is robust in an appropriate range of normalization scaling factor.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we point out the deficiencies of current metrics in characterizing over-smoothing, and highlight the importance of taking dimensional collapse as the entry point for oversmoothing analysis.
Revised: In this paper, we point out the deficiencies of current metrics in characterizing over-smoothing, and highlight the importance of taking dimensional collapse as the entry point for oversmoothing analysis.

Original: Inspired by the uniformity loss in contrastive learning, we design an optimization-induced normalization layer ContraNorm.
Revised: Inspired by the uniformity loss in contrastive learning, we design an optimization-induced normalization layer ContraNorm.

Original: Our theoretical findings indicate ContraNorm mitigates dimensional collapse successfully by reducing variance and effective rank of representations.
Revised: Our theoretical findings indicate ContraNorm mitigates dimensional collapse successfully by reducing variance and effective rank of representations.

Original: Experiments show that ContraNorm boosts the performance of Transformers and GNNs on various tasks.
Revised: Experiments show that ContraNorm boosts the performance of Transformers and GNNs on various tasks.

Original: Our work provides a new perspective from contrastive learning on solving the oversmoothing problem, which helps motivate the following extensive research
Revised: Our work provides a new perspective from contrastive learning on solving the oversmoothing problem, which helps motivate the following extensive research. A CKNOWLEDGEMENT

Revised: Yisen Wang is partially supported by the National Key R&D Program of China (2022ZD0160304), the National Natural Science Foundation of China (62006153), Open Research Projects of Zhejiang

Revised: Lab (No. 2022RC0AB05) and Huawei Technologies Inc.

Revised: Xiaojun Guo thanks for Hanqi Yan for her kind guidance on the implementation of BERT models.

Revised: We thank anonymous reviewers for their constructive advice on this work

Original: A I LLUSTRATION ON THE S TOP -G RADIENT O PERATION IN E Q .(7)
Revised: A I LLUSTRATION ON THE S TOP -G RADIENT O PERATION IN E Q .(7)

Original: In this section, we will illustrate the stop-gradient operation in Eq.(7) by using the framework proposed by (Tao et al., 2022).
Revised: In this section, we will illustrate the stop-gradient operation in Eq.(7) by using the framework proposed by (Tao et al., 2022).

Original: The original update form should be Eq.(6):
Revised: The original update form should be Eq.(6):

Original: [Equation]
Revised: [Equation]

Original: We take SimCLR(Chen et al., 2020) as the contrastive learning framework.
Revised: We take SimCLR(Chen et al., 2020) as the contrastive learning framework.

Original: Tao et al.
Revised: Tao et al.

Original: (2022) have studied the stop-gradient form of SimCLR and illustrated that the stop-gradient operation will make a similar performance with the original one.
Revised: (2022) have studied the stop-gradient form of SimCLR and illustrated that the stop-gradient operation will make a similar performance with the original one.

Original: Based on this, we will elaborate on how AD − 1 is removed in the following part.
Revised: Based on this, we will elaborate on how AD − 1 is removed in the following part.

Original: In fact, we can directly illustrate how the second term in Eq(4) can be omitted.
Revised: In fact, we can directly illustrate how the second term in Eq(4) can be omitted.

Original: [Equation]
Revised: [Equation]

Original: In SimCLR, by denoting the normalized features from the online branch as u oi , i = 1 , 2 , . . . , n , the normalized features from the target branch (although the two branches have no differences in
Revised: In SimCLR, by denoting the normalized features from the online branch as u oi , i = 1 , 2 , . . . , n , the normalized features from the target branch (although the two branches have no differences in

Original: SimCLR) are u ti , i = 1 , 2 , . .
Revised: SimCLR) are u ti , i = 1 , 2 , . .

Original: . , n and V = {
Revised: . , n and V = {

Original: u o 1 , u o 2 , . .
Revised: u o 1 , u o 2 , . .

Original: , u on , u t 1 , u t 2 , . . .
Revised: , u on , u t 1 , u t 2 , . . .

Original: , u tn } , the SimCLR loss can be represented as
Revised: , u tn } , the SimCLR loss can be represented as

Original: [Equation]
Revised: [Equation]

Original: where u 1 and u ′ 1 are positive pairs and τ is the temperature.
Revised: where u 1 and u ′ 1 are positive pairs and τ is the temperature.

Original: Then the gradient of L on u o 1 can be calculated as∂L
Revised: Then the gradient of L on u o 1 can be calculated as

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: is the softmax results over similarities between u o 1 and other samples, and
Revised: is the softmax results over similarities between u o 1 and other samples, and

Original: [Equation]
Revised: [Equation]

Original: is computed over similarities between sample v and its contrastive samples V /v .
Revised: is computed over similarities between sample v and its contrastive samples V /v .

Original: We can see that the first term of ∂L/∂u o 1 comes from the part which takes u o 1 as the anchor, and the second term comes from the part which takes the other feature as the anchor.
Revised: We can see that the first term of ∂L/∂u o 1 comes from the part which takes u o 1 as the anchor, and the second term comes from the part which takes the other feature as the anchor.

Original: Tao et al.
Revised: Tao et al.

Original: (2022) proposes to stop the second term and verifies that stopping the second gradient term will not affect the performance empirically.
Revised: (2022) proposes to stop the second term and verifies that stopping the second gradient term will not affect the performance empirically.

Original: Note that the u t 1 term in the gradient is from the alignment loss.
Revised: Note that the u t 1 term in the gradient is from the alignment loss.

Original: So the gradient of the uniformity loss on u o 1 can be written as ∂L 
Revised: So the gradient of the uniformity loss on u o 1 can be written as ∂L uniform

Original: [Equation]
Revised: [Equation]

Original: It is noteworthy that by writing V
Revised: It is noteworthy that by writing V

Original: = { h 1 , h 2 , . . .
Revised: = { h 1 , h 2 , . . .

Original: , h N } , Eq(4) shares the same form as Eq(11).
Revised: , h N } , Eq(4) shares the same form as Eq(11).

Original: By adopting the stop-gradient method just as (Tao et al., 2022) takes, we remove the second term in
Revised: By adopting the stop-gradient method just as (Tao et al., 2022) takes, we remove the second term in

Original: Eq(4), which is just the AD − 1 term in Eq(6).
Revised: Eq(4), which is just the AD − 1 term in Eq(6).

Original: Empirically, we draw the singular value distribution of embeddings for vanilla BERT and +ContraNorm with only AD − 1 term or D − 1 A term on RTE task.
Revised: Empirically, we draw the singular value distribution of embeddings for vanilla BERT and +ContraNorm with only AD − 1 term or D − 1 A term on RTE task.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5, compared with vanilla BERT with a long-tail distribution (dimensional collapse), adding ContraNorm with D − 1 A and AD − 1 both reduce the number of insignificant (nearly zero) singular values and make a more balanced distribution.
Revised: 4, compared with vanilla BERT with a long-tail distribution (dimensional collapse), adding ContraNorm with D − 1 A and AD − 1 both reduce the number of insignificant (nearly zero) singular values and make a more balanced distribution.

Original: The similar singular value distributions mean that they play a similar role in alleviating dimensional collapse.
Revised: The similar singular value distributions mean that they play a similar role in alleviating dimensional collapse.

Original: [Figure]
Revised: [Figure]

Original: BERT + ContraNorm with D 1 A + ContraNormwith AD
Revised: BERT [Figure] ContraNorm with D 1 A + ContraNorm

Original: [Figure]
Revised: [Figure]

Original: B P ROOFS FOR P ROPOSITIONS IN S ECTION 4. 
Revised: B P ROOFS FOR P ROPOSITIONS IN S ECTION 4.

Original: We give out two lemmas.
Revised: We give out two lemmas.

Original: Lemma B.1.
Revised: Lemma B.1.

Original: Denote e = [1 , 1 , . .
Revised: Denote e = [1 , 1 , . .

Original: , 1] ⊤ / √ n .
Revised: , 1] ⊤ / √ n .

Original: For any update form X 1 = P X 0 and λ > 0 , if the eigenvalues of ( I − ee ⊤ ) − λ P ⊤ ( I − ee ⊤ ) P are all not larger than zero, then we have
Revised: For any update form X 1 = P X 0 and λ > 0 , if the eigenvalues of ( I − ee ⊤ ) − λ P ⊤ ( I − ee ⊤ ) P are all not larger than zero, then we have

Original: Var( X 1 ) ≥ λ − 1 · Var( X 0 ) .
Revised: Var( X 1 ) ≥ λ − 1 · Var( X 0 ) .

Original: Proof.
Revised: Proof.

Original: We denote X 0 X ⊤= Y diag ( ω 1 , . .
Revised: We denote X 0 X ⊤= Y diag ( ω 1 , . .

Original: . , ω n ) Y ⊤ for the eigen-decomposition of X 0 X ⊤ 0 , where Y =
Revised: . , ω n ) Y ⊤ for the eigen-decomposition of X 0 X ⊤ 0 , where Y =

Original: [ y 1 , y 2 , . . .
Revised: [ y 1 , y 2 , . . .

Original: , y n ] is the orthogonal basis and all ω i ≥ 0 .
Revised: , y n ] is the orthogonal basis and all ω i ≥ 0 .

Original: Note that ( I − ee ⊤ ) 

Revised: Note that ( I − ee ⊤ )

Original: =
Revised: =

Original: ( I − ee ⊤ ) .
Revised: ( I − ee ⊤ ) .

Original: Therefore,
Revised: Therefore,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Similarly, we have
Revised: Similarly, we have

Original: Var( X 1 ) =
Revised: Var( X 1 ) =

Original: n (cid:88) 
Revised: n (cid:88) i =

Original: i =
Revised: [Equation]

Original: ω i y 
Revised: i y ⊤

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: Thus, if the eigenvalues of ( I − ee ⊤ ) − λ P ⊤ ( I − ee ⊤ ) P := Σ are all not larger than zero, Σ is semi-negative definite, then we have
Revised: Thus, if the eigenvalues of ( I − ee ⊤ ) − λ P ⊤ ( I − ee ⊤ ) P := Σ are all not larger than zero, Σ is semi-negative definite, then we have

Original: [Equation]
Revised: [Equation]

Original: which implies that Var( X 0 ) − λ Var( X 1 ) ≤ 0 .
Revised: which implies that Var( X 0 ) − λ Var( X 1 ) ≤ 0 .

Original: Therefore, Var( X 1 ) ≥ λ − 1 · Var( X 0 ) .
Revised: Therefore, Var( X 1 ) ≥ λ − 1 · Var( X 0 ) .

Original: The second lemma is from the Eq.(13) in (Fulton).
Revised: The second lemma is from the Eq.(13) in (Fulton).

Original: Lemma B.2.
Revised: Lemma B.2.

Original: Let A, B, C be symmetric matrices and C
Revised: Let A, B, C be symmetric matrices and C

Original: =
Revised: =

Original: A + B. Suppose the eigenvalues of A are α 1 ≥ α 2 ≥ · · · ≥ α n , the eigenvalues of B are β 1 ≥ β 2 ≥ · · · ≥ β n , and the eigenvalues of C are γ 1 ≥ γ 2 ≥ · · · ≥ γ n .
Revised: A + B. Suppose the eigenvalues of A are α 1 ≥ α 2 ≥ · · · ≥ α n , the eigenvalues of B are β 1 ≥ β 2 ≥ · · · ≥ β n , and the eigenvalues of C are γ 1 ≥ γ 2 ≥ · · · ≥ γ n .

Original: Then we have the inequality
Revised: Then we have the inequality

Original: [Equation]
Revised: [Equation]

Original: We can now start to prove Proposition 1.
Revised: We can now start to prove Proposition 1.

Original: Proposition 1.
Revised: Proposition 1.

Original: Let e =
Revised: Let e =

Original: . , 1) ⊤ / √ n .
Revised: . , 1) ⊤ / √ n .

Original: For attention matrix ¯ A = softmax( H b H ⊤ b ) , let σ min be the smallest eigenvalue of P =
Revised: For attention matrix ¯ A = softmax( H b H ⊤ b ) , let σ min be the smallest eigenvalue of P =

Original: ( I − ee ⊤ )( I − ¯ A )
Revised: ( I − ee ⊤ )( I − ¯ A )

Original: + ( I − ¯ A ) ⊤ ( I − ee ⊤ ) .
Revised: + ( I − ¯ A ) ⊤ ( I − ee ⊤ ) .

Original: For the ContraNorm update H t =
Revised: For the ContraNorm update H t =

Original: ((1 + s )
Revised: ((1 + s )

Original: I − s ¯ A ) H b , s > 0 , we have V ar ( H t ) ≥ (1 − sσ min ) − 1 · V ar ( H b ) .
Revised: I − s ¯ A ) H b , s > 0 , we have V ar ( H t ) ≥ (1 − sσ min ) − 1 · V ar ( H b ) .

Original: Especially, if σ min ≥ 0 , we have V ar ( H t ) ≥
Revised: Especially, if σ min ≥ 0 , we have V ar ( H t ) ≥

Original: V ar ( H b ) .
Revised: V ar ( H b ) .

Original: Proof.
Revised: Proof.

Original: We denote Σ
Revised: We denote Σ

Original: = ( I − ee ⊤ ) − λ ((1 + s )
Revised: = ( I − ee ⊤ ) − λ ((1 + s )

Original: I − s
Revised: I − s

Original: ¯ A ) ⊤ ( I − ee ⊤ )((1 + s )
Revised: ¯ A ) ⊤ ( I − ee ⊤ )((1 + s )

Original: I − s ¯ A ) .
Revised: I − s ¯ A ) .

Original: Then, Σ = (1 − λ )
Revised: Then, Σ = (1 − λ )

Original: I − λ
Revised: I − λ

Original: ( s ( I − ¯ A
Revised: ( s ( I − ¯ A

Original: ) ⊤ ( I − ee ⊤ ) − s ( I − ee ⊤ )( I − ¯ A ) − s 2 ( I − ¯ A ) ⊤ ( I − ee ⊤ )( I − ¯ A )) =
Revised: ) ⊤ ( I − ee ⊤ ) − s ( I − ee ⊤ )( I − ¯ A ) − s 2 ( I − ¯ A ) ⊤ ( I − ee ⊤ )( I − ¯ A )) =

Original: (1 − λ )
Revised: (1 − λ )

Original: I − λs P − λs
Revised: I − λs P − λs

Original: ( I − ¯ A ) ⊤ ( I − ee ⊤ )( I − ¯ A ) .
Revised: ( I − ¯ A ) ⊤ ( I − ee ⊤ )( I − ¯ A ) .

Original: Let α 1 ≥ α 2 ≥ · · · ≥ α n be the eigenvalues of Σ .
Revised: Let α 1 ≥ α 2 ≥ · · · ≥ α n be the eigenvalues of Σ .

Original: Since I − ee ⊤ has a eigenvalue of 0 and n −eigenvalues of 1, I − ee ⊤ is a semi-definite positive matrix.
Revised: Since I − ee ⊤ has a eigenvalue of 0 and n −eigenvalues of 1, I − ee ⊤ is a semi-definite positive matrix.

Original: Thus, s 2 ( I − ¯ A ) ⊤ ( I − ee ⊤ )( I − ¯ A ) is also a semi-definite positive matrix.
Revised: Thus, s 2 ( I − ¯ A ) ⊤ ( I − ee ⊤ )( I − ¯ A ) is also a semi-definite positive matrix.

Original: Notice that the largest eigenvalue of (1 − λ )
Revised: Notice that the largest eigenvalue of (1 − λ )

Original: I − s P is (1 − λ ) − sσ min and the largest eigenvalue of − s 2 ( I − ¯ A ) ⊤ ( I − ee ⊤ )
Revised: I − s P is (1 − λ ) − sσ min and the largest eigenvalue of − s 2 ( I − ¯ A ) ⊤ ( I − ee ⊤ )

Original: ( I − ¯ A ) is 0 .
Revised: ( I − ¯ A ) is 0 .

Original: Therefore, by Lemma B.2, the largest eigenvalue of Σ is less or equal to (1 − λ ) − sσ min .
Revised: Therefore, by Lemma B.2, the largest eigenvalue of Σ is less or equal to (1 − λ ) − sσ min .

Original: Let λ = 1 − sσ min , then the largest eigenvalue of Σ is less or equal to 0 .
Revised: Let λ = 1 − sσ min , then the largest eigenvalue of Σ is less or equal to 0 .

Original: By Lemma B.1, we have V ar ( H t ) ≥ (1 − sσ min ) − 1 · V ar ( H b ) .
Revised: By Lemma B.1, we have V ar ( H t ) ≥ (1 − sσ min ) − 1 · V ar ( H b ) .

Original: Moreover, if σ min ≥ 0 , then (1 − sσ min ) − 1 ≥ 1 , leading to V ar ( H t ) ≥ V ar ( H b ) .
Revised: Moreover, if σ min ≥ 0 , then (1 − sσ min ) − 1 ≥ 1 , leading to V ar ( H t ) ≥ V ar ( H b ) .

Original: Remark.
Revised: Remark.

Original: Now we discuss some sufficient conditions for σ min ≥ 0 .
Revised: Now we discuss some sufficient conditions for σ min ≥ 0 .

Original: If P is a diagonally dominant matrix, then we will have the result σ min ≥ 0 .
Revised: If P is a diagonally dominant matrix, then we will have the result σ min ≥ 0 .

Original: Denote a ij = ¯
Revised: Denote a ij = ¯

Original: A ij , b ij = σ ij − a ij and Q =
Revised: A ij , b ij = σ ij − a ij and Q =

Original: ( I − ee ⊤ )( I − ¯ A ) , where σ ij = 1 if i = j and σ ij = 

Revised: ( I − ee ⊤ )( I − ¯ A ) , where σ ij = 1 if i = j and σ ij =

Original: if i ̸
Revised: if i ̸

Original: = j , then we have
Revised: = j , then we have

Original: [Equation]
Revised: [Equation]

Original: If we have (cid:80) nk =1 a kj ≤ 1 + na ij for any i, j , then we will have
Revised: If we have (cid:80) nk =1 a kj ≤ 1 + na ij for any i, j , then we will have

Original: [Equation]
Revised: [Equation]

Original: Notice that (cid:80) nk =1 b kj = 0 , we have
Revised: Notice that (cid:80) nk =1 b kj = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Since ¯ A is an attention matrix, we also have
Revised: Since ¯ A is an attention matrix, we also have

Original: [Equation]
Revised: [Equation]

Original: Q jk | .
Revised: Q jk | .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: which indicates that P is a diagonally dominated matrix, thus σ min ≥ 0 .
Revised: which indicates that P is a diagonally dominated matrix, thus σ min ≥ 0 .

Original: Therefore, (cid:80) nk
Revised: Therefore, (cid:80) nk

Original: =
Revised: =

Original: a kj ≤ 1 + na ij for any i, j is just a sufficient condition.
Revised: a kj ≤ 1 + na ij for any i, j is just a sufficient condition.

Original: A special case for this is (cid:80) nk =1 a kj = 1 , ∀ k .
Revised: A special case for this is (cid:80) nk =1 a kj = 1 , ∀ k .

Original: We now move on to the proof of Proposition 2.
Revised: We now move on to the proof of Proposition 2.

Original: We first give a lemma on the property of effective rank.
Revised: We first give a lemma on the property of effective rank.

Original: Lemma B.3.
Revised: Lemma B.3.

Original: Let the eigenvalues of AA ⊤ be λ 1 ≥ λ 2 ≥ · · · ≥ λ n and the eigenvalues of BB ⊤ be σ 1 ≥ σ 2 ≥ . . .
Revised: Let the eigenvalues of AA ⊤ be λ 1 ≥ λ 2 ≥ · · · ≥ λ n and the eigenvalues of BB ⊤ be σ 1 ≥ σ 2 ≥ . . .

Original: σ n .
Revised: σ n .

Original: If σ i /λ i is increasing as i increases, then we have erank( B ) ≥ erank( A ) .
Revised: If σ i /λ i is increasing as i increases, then we have erank( B ) ≥ erank( A ) .

Original: This lemma can be proved just by using the definition of the effective rank.
Revised: This lemma can be proved just by using the definition of the effective rank.

Original: Proposition 2.
Revised: Proposition 2.

Original: Consider the update form
Revised: Consider the update form

Original: [Equation]
Revised: [Equation]

Original: let σ max be the largest singular value of H b .
Revised: let σ max be the largest singular value of H b .

Original: For s > 0 satisfying 1 + (1 − σ 2max ) s > 0 , we have erank( H t ) >
Revised: For s > 0 satisfying 1 + (1 − σ 2max ) s > 0 , we have erank( H t ) >

Original: erank( H b ) .
Revised: erank( H b ) .

Original: Proof.
Revised: Proof.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Suppose λ 1 ≥ λ 2 ≥ · · · ≥ λ n are the eigenvalues of H b H ⊤ b , then H t H Tt has the same eigenvectors as H b H ⊤ b , and its eigenvalues are ( λ i s − (1 + s )) 2 λ i .
Revised: Suppose λ 1 ≥ λ 2 ≥ · · · ≥ λ n are the eigenvalues of H b H ⊤ b , then H t H Tt has the same eigenvectors as H b H ⊤ b , and its eigenvalues are ( λ i s − (1 + s )) 2 λ i .

Original: Since s satisfies 1 + (1 − σ 2max s ) > 0 , we have 1 + s >
Revised: Since s satisfies 1 + (1 − σ 2max s ) > 0 , we have 1 + s >

Original: λ i s .
Revised: λ i s .

Original: Therefore, ( λ i s − (1 + s )) 2 is increasing as i increases, resulting the fact that erank( H t ) >
Revised: Therefore, ( λ i s − (1 + s )) 2 is increasing as i increases, resulting the fact that erank( H t ) >

Original: erank( H b ) by using Lemma B.3.
Revised: erank( H b ) by using Lemma B.3.

Original: C M ETRICS CALCULATING FEATURE AND ATTENTION MAP COSINE SIMILARITY
Revised: C M ETRICS CALCULATING FEATURE AND ATTENTION MAP COSINE SIMILARITY

Original: Following Wang et al.
Revised: Following Wang et al.

Original: , given feature map H ∈ R N × D and attention map of the h -th head
Revised: , given feature map H ∈ R N × D and attention map of the h -th head

Original: A ( h ) ∈ R N × N with batch size N and hidden embedding size D , the feature cosine similarity and the attention cosine similarity is computed as[Equation] N
Revised: A ( h ) ∈ R N × N with batch size N and hidden embedding size D , the feature cosine similarity and the attention cosine similarity is computed as

Original: [Equation]
Revised: [Equation]

Original: where h i denotes the i -th row of H , a i is the i -th column of A , and H is the number of attention heads.
Revised: where h i denotes the i -th row of H , a i is the i -th column of A , and H is the number of attention heads.

Original: D A PPLYING THE U NIFORMITY L OSS D IRECTLY
Revised: D A PPLYING THE U NIFORMITY L OSS D IRECTLY

Original: We conduct a comparative experiment on BERT model with straightforwardly applied uniformity loss and our proposed ContraNorm.
Revised: We conduct a comparative experiment on BERT model with straightforwardly applied uniformity loss and our proposed ContraNorm.

Original: Specifically, we add the uniformity loss ( loss uni ) to the classification loss (MSELoss, CrossEntropyLoss or BCELoss depending on the task type, denoted by loss cls ).
Revised: Specifically, we add the uniformity loss ( loss uni ) to the classification loss (MSELoss, CrossEntropyLoss or BCELoss depending on the task type, denoted by loss cls ).

Original: Formally, the final loss is written as
Revised: Formally, the final loss is written as

Original: [Equation]
Revised: [Equation]

Original: where loss uni = (cid:80) Ni =1 log (cid:80) Nj =

Revised: where loss uni = (cid:80) Ni =1 log (cid:80) Nj =

Original: exp( f ( x i ) T f ( x j ) /τ ) and N is the number of samples in the batch.
Revised: exp( f ( x i ) T f ( x j ) /τ ) and N is the number of samples in the batch.

Original: We tune τ in the range of [0 . 5 , 0 . 8 , 1 . 0 , 1 . 2 , 1 . 5] and choose the best one in terms of average performance.
Revised: We tune τ in the range of [0 . 5 , 0 . 8 , 1 . 0 , 1 . 2 , 1 . 5] and choose the best one in terms of average performance.

Original: Other hyperparameters are kept the same as settings of ContraNorm.
Revised: Other hyperparameters are kept the same as settings of ContraNorm.

Original: The results are shown in the Table.
Revised: The results are shown in the Table.

Original: We can see that +ContraNorm gets the best score in 8 / 9 tasks, while +Uniformity loss reaches the best only in 1 / 9 tasks.
Revised: We can see that +ContraNorm gets the best score in 8 / 9 tasks, while +Uniformity loss reaches the best only in 1 / 9 tasks.

Original: ContraNorm also has the highest average score among all tasks.
Revised: ContraNorm also has the highest average score among all tasks.

Original: The reason is that updating the total loss is a combined process for objectives of correct classification and uniform distribution.
Revised: The reason is that updating the total loss is a combined process for objectives of correct classification and uniform distribution.

Original: Thus, a lower loss total may be only caused by a lower classification loss while uniformity loss is kept the same, which cannot ensure a more uniform distribution of representations.
Revised: Thus, a lower loss total may be only caused by a lower classification loss while uniformity loss is kept the same, which cannot ensure a more uniform distribution of representations.

Original: In contrast, ContraNorm acts directly on representations in each layer and enforces the uniform property.
Revised: In contrast, ContraNorm acts directly on representations in each layer and enforces the uniform property.

Original: In fact, there are many methods in GNNs such as Yang et al.
Revised: In fact, there are many methods in GNNs such as Yang et al.

Original: (2021) and Zhu et al.
Revised: (2021) and Zhu et al.

Original: (2021), which design the propagation mechanism under the guidance of the corresponding objective.
Revised: (2021), which design the propagation mechanism under the guidance of the corresponding objective.

Original: The welldesigned propagation mechanism is shown to be the most fundamental part of GNNs (Zhu et al.,
Revised: The welldesigned propagation mechanism is shown to be the most fundamental part of GNNs (Zhu et al.,

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: Instead of directly using the loss function, these methods transfer the loss function into a specific propagation method and achieve superior performance

Original: [Table]
Revised: [Table]

Original: Instead of directly using the loss function, these methods transfer the loss function into a specific propagation method and achieve superior performance, which indicates that changing the network may be more effective than directly adding the objective to the loss function.
Revised: which indicates that changing the network may be more effective than directly adding the objective to the loss function.

Original: E C HOICES OF C ONTRA N ORM PLUGGING POSITION
Revised: E C HOICES OF C ONTRA N ORM PLUGGING POSITION

Original: We explore two ways to integrate ContraNorm into BERT and ALBERT.
Revised: We explore two ways to integrate ContraNorm into BERT and ALBERT.

Original: Concretely, consider the update of H ( l ) in l -th block of Transformers
Revised: Concretely, consider the update of H ( l ) in l -th block of Transformers

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: whereX
Revised: where

Original:  = H b is the input tensor.
Revised: X = H b is the input tensor.

Original: We choose positions 1) between Eq.(26) and Eq.(27), named as before-residual ; 2) between Eq.(27) and Eq.(28), named as after-residual .
Revised: We choose positions 1) between Eq.(26) and Eq.(27), named as before-residual ; 2) between Eq.(27) and Eq.(28), named as after-residual .

Original: The performance comparison between the two positions on
Revised: The performance comparison between the two positions on

Original: GLUE datasets is shown in Table.
Revised: GLUE datasets is shown in Table.

Original: It is observed that putting ContraNorm after the residual connection slightly outperforms that before residual.
Revised: It is observed that putting ContraNorm after the residual connection slightly outperforms that before residual.

Original: Therefore, we choose the after-residual variant as our basic ContraNorm.
Revised: Therefore, we choose the after-residual variant as our basic ContraNorm.

Original: F R ESULTS ON TEST SET OF GLUE DATASETS
Revised: F T IME C OMPLEXITY A NALYSIS AND A M ODIFIED C ONTRA N ORM

Revised: The most time-consuming operation of ContraNorm is the matrix multiplication. Given H ∈ R n × d , where n and d denote the number of samples in a batch and feature size respectively, the time complexity of ContraNorm is O ( n 2 d ) , which is the same order as the self-attention operation in Transformer.

Revised: Empirically, we report the training time of BERT with or without ContraNorm on

Revised: GLUE tasks in Table 7.

Revised: All experiments are conducted on a single NVIDIA GeForce RTX 3090.

Revised: On average, we raise the performance of BERT on GLUE tasks from 82.59% to 83.54% (see Table 1) with less than 4 minutes overhead.

Revised: We think the time overhead is acceptable considering the benefits it brings.

Revised: Moreover, to enhance scalability we propose a modified version of ContraNorm inspired by Ali et al.

Revised: As already alluded to, ContraNorm resolves dimensional collapse by pushing samples away from each other with the uniformity loss as the energy function.

Revised: A different choice of the underlying function generates a new form of normalization layer.

Revised: Here, to maintain the same goal of settling dimensional collapse, we choose another contrastive loss named Barlow Twins (Zbontar et al., 2021)

Revised: [Equation]

Revised: [Table]

Revised: with C ij = (cid:80) b f ( x Ab,i ) f

Revised: ( x Bb,j ) / (cid:113)(cid:80) b ( f ( x Ab,i )) 2 (cid:113)(cid:80) b ( f ( x Bb,j )) 2 , where λ is a trading-off constant, b is the batch index, i, j index the vector dimension, and x A ,

Revised: x B denotes different augmented views of x .

Revised: Dislike the sample-wise loss InfoNCE, Barlow Twins decorrelates the dimensions and can be seen as contrastive between the dimensions of the embeddings (Garrido et al., 2022).

Revised: Despite seeming different, recent works empirically show strong similarities between the two loss functions, and theoretically prove the equivalence between them under limited assumptions (Garrido et al., 2022;

Revised: Tao et al., 2022; Balestriero & LeCun, 2022; Huang et al., 2021).

Revised: Therefore, the layer generated by unfolding iterations of Barlow Twins is believed to work equally well as ContraNorm.

Revised: Specifically, we reorder the matrix multiplication from HH T ∈ R n × n to H T

Revised: H ∈ R d × d to enforce the channels of embeddings different from each other.

Revised: Then the modified version of ContraNorm becomes

Revised: [Equation]

Revised: The transposed alternative has a linear complexity in the number of samples,

Revised: i

Revised: [Table]

Revised: e. O ( nd 2 ) .

Revised: In the case of n ≫ d , the modified version greatly alleviates the computation burden.

Revised: To verify whether it performs equally well as ContraNorm

Revised: [Table]

Revised: we conduct experiments on the validation set of GLUE tasks.

Revised: The learning rate of BERT with ContraNorm-v2 is set to 4 e − 5 and other experiment setups are the same in Section 4.1.

Revised: As shown from Table 8, for each task the performance of BERT with ContraNorm-v2 surpasses the vanilla BERT, and the average performance is raised from 82 .

Revised: 59% to 84 .

Revised: The results imply effectiveness of this modified version of ContraNorm, which can also be explained with the relationship between Gram matrix ( G = HH T ∈ R n × n ) and covariance matrix (

Revised: C = H T

Revised: H ∈ R d × d ) (Ali et al., 2021).

Original:  We submit our trained model to GLUE benchmark leaderboard and the resultant feedback of performance is shown in Table (7).
Revised: [Table] G R ESULTS ON TEST SET OF GLUE DATASETS We submit our trained model to GLUE benchmark leaderboard and the resultant feedback of performance is shown in Table (9).

Original: [Table]
Revised: [Table]

Original: G I NTRODUCTION OF GRAPH DATASETS
Revised: H I NTRODUCTION OF GRAPH DATASETS

Original: Citation Network.
Revised: Citation Network.

Original: Cora, CiteSeer are three popular citation graph datasets.
Revised: Cora, CiteSeer are three popular citation graph datasets.

Original: In these graphs, nodes represent papers and edges correspond to the citation relationship between two papers.
Revised: In these graphs, nodes represent papers and edges correspond to the citation relationship between two papers.

Original: Nodes are classified according to academic topics.
Revised: Nodes are classified according to academic topics.

Original: Wikipedia Network.
Revised: Wikipedia Network.

Original: Chameleon and Squirrel are Wikipedia page networks on specific topics, where nodes represent web pages and edges are the mutual links between them.
Revised: Chameleon and Squirrel are Wikipedia page networks on specific topics, where nodes represent web pages and edges are the mutual links between them.

Original: Node features are the bag-of-words representation of informative nouns.
Revised: Node features are the bag-of-words representation of informative nouns.

Original: The nodes are classified into four categories according to the number of the average monthly traffic of the page.
Revised: The nodes are classified into four categories according to the number of the average monthly traffic of the page.

Original: Statics of datasets are shown in Table. 
Revised: Statics of datasets are shown in Table. [Table]

Original: H A BLATION STUDY FULL RESULTS
Revised: I A BLATION STUDY FULL RESULTS

Original: In Fig.(6), we show the performance of BERT+ContraNorm with varying scaling factor s on GLUE datasets.
Revised: In Fig.(5), we show the performance of BERT+ContraNorm with varying scaling factor s on GLUE datasets.

Original: Other parameters remain consistent.
Revised: Other parameters remain consistent.

Revised: The results illustrate that model with ContraNorm is robust in an appropriate range of normalization scaling factor.

Original:  [Figure] [Figure]
Revised: [Table] [Figure] [Table]

Original: m
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: tt
Revised: r

Original: [Figure]
Revised: [Figure]

Original: e
Revised: y

Original: [Figure]
Revised: [Figure]

Original: rr
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: i
Revised: r a c y

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: cc

Original: [Figure]
Revised: [Figure]