Original: R EDUCING O VERSMOOTHING IN GRAPH NEURAL NETWORKS BY CHANGING THE ACTIVATION FUNCTION
Revised: R EDUCING O VERSMOOTHING IN GRAPH NEURAL NETWORKS BY CHANGING THE ACTIVATION FUNCTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The performance of Graph Neural Networks (GNNs) deteriorates as the depth of the network increases.
Revised: The performance of Graph Neural Networks (GNNs) deteriorates as the depth of the network increases.

Original: That performance drop is mainly attributed to oversmoothing, which leads to similar node representations through repeated graph convolutions.
Revised: That performance drop is mainly attributed to oversmoothing, which leads to similar node representations through repeated graph convolutions.

Original: We show that in deep GNNs the activation function plays a crucial role in oversmoothing.
Revised: We show that in deep GNNs the activation function plays a crucial role in oversmoothing.

Original: We explain theoretically why this is the case and propose a simple modification to the slope of ReLU to reduce oversmoothing.
Revised: We explain theoretically why this is the case and propose a simple modification to the slope of ReLU to reduce oversmoothing.

Original: The proposed approach enables deep architectures without the need to change the network architecture or to add residual connections.
Revised: The proposed approach enables deep architectures without the need to change the network architecture or to add residual connections.

Original: We verify the theoretical results experimentally and further show that deep networks, which do not suffer from oversmoothing, are beneficial in the presence of the “cold start” problem, i e. when there is no feature information about unlabeled nodes
Revised: We verify the theoretical results experimentally and further show that deep networks, which do not suffer from oversmoothing, are beneficial in the presence of the “cold start” problem, i e. when there is no feature information about unlabeled nodes

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph Neural Networks (GNNs) utilize message passing or neighborhood aggregation schemes to extract representations for nodes and their neighborhoods.
Revised: Graph Neural Networks (GNNs) utilize message passing or neighborhood aggregation schemes to extract representations for nodes and their neighborhoods.

Original: GNNs have achieved good results on a variety of graph analytics tasks, such as node classification (Zhang et al., 2018; Hamilton et al., 2017), link prediction (Liben-Nowell & Kleinberg, 2003; Zhang & Chen, 2018) and graph classification (Klicpera et al., 2020).
Revised: GNNs have achieved good results on a variety of graph analytics tasks, such as node classification (Zhang et al., 2018; Hamilton et al., 2017), link prediction (Liben-Nowell & Kleinberg, 2003; Zhang & Chen, 2018) and graph classification (Klicpera et al., 2020).

Original: As a result, they play a key role in graph representation learning.
Revised: As a result, they play a key role in graph representation learning.

Original: One of the most prominent GNN models is the Graph Convolutional Network (GCN) (Kipf & Welling, 2017), which creates node representations, by averaging the representations (embeddings) of its immediate neighbors.
Revised: One of the most prominent GNN models is the Graph Convolutional Network (GCN) (Kipf & Welling, 2017), which creates node representations, by averaging the representations (embeddings) of its immediate neighbors.

Original: Several studies have shown that, the performance of GCNs deteriorates, when their architecture becomes deeper (Li et al., 2018).
Revised: Several studies have shown that, the performance of GCNs deteriorates, when their architecture becomes deeper (Li et al., 2018).

Original: The success of deep CNNs on many tasks, like image classification, naturally led to several attempts towards building deep GNNs for node classification (Kipf & Welling, 2017; Li et al., 2018;
Revised: The success of deep CNNs on many tasks, like image classification, naturally led to several attempts towards building deep GNNs for node classification (Kipf & Welling, 2017; Li et al., 2018;

Original: Xu et al., 2018).
Revised: Xu et al., 2018).

Original: Increasing the model’s depth (and the number of parameters it has), would allow more accurate representational learning to occur.
Revised: Increasing the model’s depth (and the number of parameters it has), would allow more accurate representational learning to occur.

Original: Most of the existing approaches have failed to develop a sufficiently deep architecture that achieves good performance.
Revised: Most of the existing approaches have failed to develop a sufficiently deep architecture that achieves good performance.

Original: Therefore, there is a need to design new models which can efficiently scale to a large number of layers.
Revised: Therefore, there is a need to design new models which can efficiently scale to a large number of layers.

Original: The aim of this paper is to investigate the contributing factors, that compromise the performance of deep GNNs and develop a method to address them.
Revised: The aim of this paper is to investigate the contributing factors, that compromise the performance of deep GNNs and develop a method to address them.

Original: The performance drop of deep GNNs, is associated with several factors, including vanishing gradients, overfitting, as well as the phenomenon called oversmoothing (Li et al., 2018; Xu et al., 2018; Klicpera et al., 2018; Wu et al., 2019).
Revised: The performance drop of deep GNNs, is associated with several factors, including vanishing gradients, overfitting, as well as the phenomenon called oversmoothing (Li et al., 2018; Xu et al., 2018; Klicpera et al., 2018; Wu et al., 2019).

Original: Oversmoothing has been shown to be associated with graph convolution, a type of Laplacian operator.
Revised: Oversmoothing has been shown to be associated with graph convolution, a type of Laplacian operator.

Original: Li et al.
Revised: Li et al.

Original: (2018) proved that applying that operator repeatedly, makes node representations converge to a stationary point.
Revised: (2018) proved that applying that operator repeatedly, makes node representations converge to a stationary point.

Original: At that point, all of the initial information (i.e. node features’ information) is lost through the Laplacian smoothing.
Revised: At that point, all of the initial information (i.e. node features’ information) is lost through the Laplacian smoothing.

Original: Consequently, oversmoothing hurts the performance by making node features indistinguishable across different classes.
Revised: Consequently, oversmoothing hurts the performance by making node features indistinguishable across different classes.

Original: In this work, we address the oversmoothing problem in deep GNNs.
Revised: In this work, we address the oversmoothing problem in deep GNNs.

Original: We provide what is to the best of our knowledge the first study regarding the role of the activation function and the learning rate per layer of the model to oversmoothing and propose a new method to address the problem.
Revised: We provide what is to the best of our knowledge the first study regarding the role of the activation function and the learning rate per layer of the model to oversmoothing and propose a new method to address the problem.

Original: We confirm our hypotheses both experimentally and theoretically.
Revised: We confirm our hypotheses both experimentally and theoretically.

Original: The new method is shown to prevent node embeddings from converging to the same point, thus leading to better node representations.
Revised: The new method is shown to prevent node embeddings from converging to the same point, thus leading to better node representations.

Original: We summarize our main contributions as follows.
Revised: We summarize our main contributions as follows.

Original: • Role of Activation Function in Oversmoothing :
Revised: • Role of Activation Function in Oversmoothing :

Original: We prove theoretically the connection between
Revised: We prove theoretically the connection between

Original: the activation function and oversmoothing.
Revised: the activation function and oversmoothing.

Original: In fact, we show the relation between the slope of ReLU and the singular values of weight matrices, which are known to be associated with oversmoothing (Oono & Suzuki, 2020; Cai & Wang, 2020).
Revised: In fact, we show the relation between the slope of ReLU and the singular values of weight matrices, which are known to be associated with oversmoothing (Oono & Suzuki, 2020; Cai & Wang, 2020).

Original: We have also verified our theoretical results experimentally.
Revised: We have also verified our theoretical results experimentally.

Original: • Role of Learning Rate in Oversmoothing :
Revised: • Role of Learning Rate in Oversmoothing :

Original: Our analysis on the effect of the slope of ReLU to oversmoothing has a direct extension to the learning rates used per layer of the network.
Revised: Our analysis on the effect of the slope of ReLU to oversmoothing has a direct extension to the learning rates used per layer of the network.

Original: We conducted further experiments to study the effect of tuning the learning rates, showing that this approach could also reduce oversmoothing, but it is less practical.
Revised: We conducted further experiments to study the effect of tuning the learning rates, showing that this approach could also reduce oversmoothing, but it is less practical.

Original: • The power of Deep GNNs: We have performed extensive experiments using up to 64-layer networks, tackling oversmoothing with the proposed method.
Revised: • The power of Deep GNNs: We have performed extensive experiments using up to 64-layer networks, tackling oversmoothing with the proposed method.

Original: We further show the benefits that such deep GNNs can provide in the presence of reduced information, such as in a “cold start” situation, where node features are available only for the labeled nodes in a node classification setting.
Revised: We further show the benefits that such deep GNNs can provide in the presence of reduced information, such as in a “cold start” situation, where node features are available only for the labeled nodes in a node classification setting.

Original:  N OTATIONS AND P RELIMINARIES 2. N OTATIONS
Revised:  N OTATIONS AND P RELIMINARIES 2. N OTATIONS

Original: In order to illustrate the problem of oversmoothing, we consider the task of semi-supervised node classification on a graph.
Revised: In order to illustrate the problem of oversmoothing, we consider the task of semi-supervised node classification on a graph.

Original: The graph to be analysed is G ( V , E ,X), with | V | =
Revised: The graph to be analysed is G ( V , E ,X), with | V | =

Original: N nodes u i ∈ V , edges ( u i , u j ) ∈ E and X =
Revised: N nodes u i ∈ V , edges ( u i , u j ) ∈ E and X =

Original: [ x 1 , ..., x N ] T ∈ R N × C denotes the initial node features.
Revised: [ x 1 , ..., x N ] T ∈ R N × C denotes the initial node features.

Original: The edges form an adjacency matrix A ∈ R N × N where edge ( u i , u j ) is associated with element A i,j .
Revised: The edges form an adjacency matrix A ∈ R N × N where edge ( u i , u j ) is associated with element A i,j .

Original: A i,j can take arbitrary real values indicating the weight (strength) of edge ( u i , u j ) .
Revised: A i,j can take arbitrary real values indicating the weight (strength) of edge ( u i , u j ) .

Original: Node degrees are represented through a diagonal matrix D ∈ R N × N , where each element d i represents the sum of edge weights connected to node i. During training, only the labels of a subset V l ∈ V are available.
Revised: Node degrees are represented through a diagonal matrix D ∈ R N × N , where each element d i represents the sum of edge weights connected to node i. During training, only the labels of a subset V l ∈ V are available.

Original: The task is to learn a node classifier, that predicts the label of each node using the graph topology and the given feature vectors.
Revised: The task is to learn a node classifier, that predicts the label of each node using the graph topology and the given feature vectors.

Original: GCN , originally proposed by Kipf & Welling (2017), utilizes a feed forward propagation as:
Revised: GCN , originally proposed by Kipf & Welling (2017), utilizes a feed forward propagation as:

Original: [Equation]
Revised: [Equation]

Original: where H ( l ) =
Revised: where H ( l ) =

Original: [ h ( l ) 1 , ..., h ( l ) N ] are node representations (or hidden vectors or embeddings) at the l -th layer, with h li standing for the hidden representation of node i;
Revised: [ h ( l ) 1 , ..., h ( l ) N ] are node representations (or hidden vectors or embeddings) at the l -th layer, with h li standing for the hidden representation of node i;

Original: ˆ A =
Revised: ˆ A =

Original: ˆ D − 1 / 

Revised: ˆ D − 1 /

Original: ( A + I ) ˆ D − 1 / 2 denotes the augmented adjacency matrix after self-loop addition, where ˆ D corresponds to the degree matrix; σ ( · ) is a nonlinear element-wise function, i.e. the activation function, ReLU; and W ( l ) is the trainable weight matrix of the l -th layer.
Revised: ( A + I ) ˆ D − 1 / 2 denotes the augmented adjacency matrix after self-loop addition, where ˆ D corresponds to the degree matrix; σ ( · ) is a nonlinear element-wise function, i.e. the activation function, ReLU; and W ( l ) is the trainable weight matrix of the l -th layer.

Original: A generic ReLU function with slope α is defined as ReLU( x ) = max ( α · x, 0) .
Revised: A generic ReLU function with slope α is defined as ReLU( x ) = max ( α · x, 0) .

Original: 2. U NDERSTANDING O VERSMOOTHING
Revised: 2. U NDERSTANDING O VERSMOOTHING

Original: GNNs achieve state-of-the-art performance in a variety of graph-based tasks.
Revised: GNNs achieve state-of-the-art performance in a variety of graph-based tasks.

Original: Despite their success, models like GCN (Kipf & Welling, 2017) and GAT (Velickovic et al., 2018) experience a performance drop, when stacking multiple layers.
Revised: Despite their success, models like GCN (Kipf & Welling, 2017) and GAT (Velickovic et al., 2018) experience a performance drop, when stacking multiple layers.

Original: To a large extent, this is attributed to oversmoothing due to repeated graph convolutions.
Revised: To a large extent, this is attributed to oversmoothing due to repeated graph convolutions.

Original: In their analysis Li et al.
Revised: In their analysis Li et al.

Original: (2018), showed that graph convolution is a special form of Laplacian smoothing.
Revised: (2018), showed that graph convolution is a special form of Laplacian smoothing.

Original: In fact, they proved that the new representation of each node is formed by a weighted average of its own representation and that of its neighbors.
Revised: In fact, they proved that the new representation of each node is formed by a weighted average of its own representation and that of its neighbors.

Original: This mechanism allows the node representations within each (graph) cluster, i.e. highly connected group of nodes, to become more similar and improves the performance on semi-supervised tasks on graphs.
Revised: This mechanism allows the node representations within each (graph) cluster, i.e. highly connected group of nodes, to become more similar and improves the performance on semi-supervised tasks on graphs.

Original: When stacking multiple layers, the smoothing operation is repeated multiple times leading to oversmoothing of node representations, i.e., the hidden representations of all nodes become similar, resulting in information loss.
Revised: When stacking multiple layers, the smoothing operation is repeated multiple times leading to oversmoothing of node representations, i.e., the hidden representations of all nodes become similar, resulting in information loss.

Original: 2. D EEP GNN LIMITATIONS
Revised: 2. D EEP GNN LIMITATIONS

Original: Therefore, oversmoothing leads node representations to converge to a fixed point as the network’s depth increases (Li et al., 2018).
Revised: Therefore, oversmoothing leads node representations to converge to a fixed point as the network’s depth increases (Li et al., 2018).

Original: At that point, node representations contain information relevant to the graph topology and disregard the input features.
Revised: At that point, node representations contain information relevant to the graph topology and disregard the input features.

Original: Oono & Suzuki (2020) have generalized the idea in Li et al.
Revised: Oono & Suzuki (2020) have generalized the idea in Li et al.

Original: (2018) by taking into consideration, that the ReLU activation function maps to a positive cone.
Revised: (2018) by taking into consideration, that the ReLU activation function maps to a positive cone.

Original: They explain oversmoothing as the convergence to a subspace, instead of the convergence to a fixed point.
Revised: They explain oversmoothing as the convergence to a subspace, instead of the convergence to a fixed point.

Original: A similar approach is presented in (Cai & Wang, 2020), offering a different perspective to the oversmoothing problem using Dirichlet energy.
Revised: A similar approach is presented in (Cai & Wang, 2020), offering a different perspective to the oversmoothing problem using Dirichlet energy.

Original: We now look closer to the proposal of Oono & Suzuki (2020), which we will use as a basis for our analysis.
Revised: We now look closer to the proposal of Oono & Suzuki (2020), which we will use as a basis for our analysis.

Original: The main result in (Oono & Suzuki, 2020) is the definition of the convergence speed towards a subspace M, where the distance between node representations tends to zero.
Revised: The main result in (Oono & Suzuki, 2020) is the definition of the convergence speed towards a subspace M, where the distance between node representations tends to zero.

Original: We denote as d M ( X ) the distance between the feature vector X and the subspace, where the oversmoothing is prevalent.
Revised: We denote as d M ( X ) the distance between the feature vector X and the subspace, where the oversmoothing is prevalent.

Original: When that distance approaches to zero it indicates that node representations have been over-smoothed.
Revised: When that distance approaches to zero it indicates that node representations have been over-smoothed.

Original: For this distance, Oono & Suzuki (2020) prove an interesting property.
Revised: For this distance, Oono & Suzuki (2020) prove an interesting property.

Original: Theorem 1 (Oono & Suzuki (2020))
Revised: Theorem 1 (Oono & Suzuki (2020))

Original: Let the largest singular value of the weight matrix
Revised: Let the largest singular value of the weight matrix

Original: W lh be s lh and s l = H l (cid:81) h
Revised: W lh be s lh and s l = H l (cid:81) h

Original: =1 s lh , where W lh is the weight matrix of layer h and H l be the network’s depth, following the notation of the paper.
Revised: =1 s lh , where W lh is the weight matrix of layer h and H l be the network’s depth, following the notation of the paper.

Original: Then it holds that d M ( f l ( X )) ≤ s l d M ( X ) for any X ∈ R N × C , where f( · ) is the forward pass of a GNN layer (i.e. σ ( AXW lh ) ).
Revised: Then it holds that d M ( f l ( X )) ≤ s l d M ( X ) for any X ∈ R N × C , where f( · ) is the forward pass of a GNN layer (i.e. σ ( AXW lh ) ).

Original: The above theorem indicates, that the deeper the network the smaller the distance of node representations from the subspace M. If the maximum singular values are small then node representations asymptotically approach M, for any initial values of node features.
Revised: The above theorem indicates, that the deeper the network the smaller the distance of node representations from the subspace M. If the maximum singular values are small then node representations asymptotically approach M, for any initial values of node features.

Original: Extending the above theorem, the authors conclude to the following estimate about the speed of convergence to the oversmoothing subspace.
Revised: Extending the above theorem, the authors conclude to the following estimate about the speed of convergence to the oversmoothing subspace.

Original: Corollary 1 (Oono & Suzuki (2020))
Revised: Corollary 1 (Oono & Suzuki (2020))

Original: Let s = sup l ∈ N + s l · then d M ( X ( l ) )
Revised: Let s = sup l ∈ N + s l · then d M ( X ( l ) )

Original: =
Revised: =

Original: O (( sλ ) l ) , where l is the layer number and if sλ < 1 the distance from oversmoothing subspace exponentially approaches zero.
Revised: O (( sλ ) l ) , where l is the layer number and if sλ < 1 the distance from oversmoothing subspace exponentially approaches zero.

Original: Where λ is the smallest non-zero eigenvalue of I - ˆ A .
Revised: Where λ is the smallest non-zero eigenvalue of I - ˆ A .

Original: According to the authors, sufficiently deep GCN will certainly suffer from oversmoothing under some conditions (details can be found in Oono & Suzuki (2020)).
Revised: According to the authors, sufficiently deep GCN will certainly suffer from oversmoothing under some conditions (details can be found in Oono & Suzuki (2020)).

Original: We build upon this result, aiming to develop a consistent approach that reduces oversmoothing and enables deep architectures.
Revised: We build upon this result, aiming to develop a consistent approach that reduces oversmoothing and enables deep architectures.

Original: 2. N ORMALIZATION IN DEEP NEURAL NETWORKS
Revised: 2. N ORMALIZATION IN DEEP NEURAL NETWORKS

Original: Of relevance to oversmoothing are also methods that perform normalization for deep neural networks (Ioffe & Szegedy, 2015; Ba et al., 2016).
Revised: Of relevance to oversmoothing are also methods that perform normalization for deep neural networks (Ioffe & Szegedy, 2015; Ba et al., 2016).

Original: In such methods, the output of each neuron is normalized, in order to keep a portion of the initial feature variance.
Revised: In such methods, the output of each neuron is normalized, in order to keep a portion of the initial feature variance.

Original: Other recent work proposed
Revised: Other recent work proposed

Original: Self Normalized Networks (SNN), that use a different activation function (SeLU) (Klambauer et al., 2017).
Revised: Self Normalized Networks (SNN), that use a different activation function (SeLU) (Klambauer et al., 2017).

Original: These models perform self normalization inside each neuron, in order to keep the variance between consecutive layers stable.
Revised: These models perform self normalization inside each neuron, in order to keep the variance between consecutive layers stable.

Original: They have managed to enable deep Fully Connected models and achieve good performance.
Revised: They have managed to enable deep Fully Connected models and achieve good performance.

Original: A recent work regarding normalization in GNNs is the Pairnorm method (Zhao & Akoglu, 2020), which aims to keep constant the total pairwise distance between node representations.
Revised: A recent work regarding normalization in GNNs is the Pairnorm method (Zhao & Akoglu, 2020), which aims to keep constant the total pairwise distance between node representations.

Original: Compared to
Revised: Compared to

Original: SNNs, Pairnorm performs normalization needing a constant value as hyper-parameter determined per dataset, while SNNs manage to normalize the output utilizing their activation function, which can be used in a family of neural networks.
Revised: SNNs, Pairnorm performs normalization needing a constant value as hyper-parameter determined per dataset, while SNNs manage to normalize the output utilizing their activation function, which can be used in a family of neural networks.

Original: The activation function proposed in (Klambauer et al., 2017) (i.e. SeLU) has a saturating region to reduce the variance of data and a slope slightly greater than 1 in order to increase the data variance, when needed.
Revised: The activation function proposed in (Klambauer et al., 2017) (i.e. SeLU) has a saturating region to reduce the variance of data and a slope slightly greater than 1 in order to increase the data variance, when needed.

Original:  U NDERSTANDING AND DEALING WITH OVERSMOOTHING
Revised:  U NDERSTANDING AND DEALING WITH OVERSMOOTHING

Original: Using the mathematical definition of oversmoothing in Oono & Suzuki (2020), in this section we establish a connection with the training process of GNNs.
Revised: Using the mathematical definition of oversmoothing in Oono & Suzuki (2020), in this section we establish a connection with the training process of GNNs.

Original: In particular, we analyse the role of the activation function and the learning rate and we propose modified versions of GNNs to address the issue.
Revised: In particular, we analyse the role of the activation function and the learning rate and we propose modified versions of GNNs to address the issue.

Original: 3. T HEORETICAL A NALYSIS
Revised: 3. T HEORETICAL A NALYSIS

Original: We start by establishing the connection between oversmoothing and variance reduction of node representations.
Revised: We start by establishing the connection between oversmoothing and variance reduction of node representations.

Original: Consider oversmoothing as the convergence to a subspace, where node representations are almost the same.
Revised: Consider oversmoothing as the convergence to a subspace, where node representations are almost the same.

Original: In that particular subspace, the initial variance of feature vectors has been massively reduced.
Revised: In that particular subspace, the initial variance of feature vectors has been massively reduced.

Original: This unwanted oversmoothing effect is ‘harsh’, due to the fact that ReLU performs a zero mapping, when the input is negative.
Revised: This unwanted oversmoothing effect is ‘harsh’, due to the fact that ReLU performs a zero mapping, when the input is negative.

Original: We use the term ‘harsh’ to indicate the smoothing case, where representations are mapped to a single point (i.e. zero), instead of converging to a subspace.
Revised: We use the term ‘harsh’ to indicate the smoothing case, where representations are mapped to a single point (i.e. zero), instead of converging to a subspace.

Original: We use the notation of ‘converging to a subspace’ following the oversmoothing analysis presented in Oono & Suzuki (2020), where the subspace M (where oversmoothing is prevalent) is defined.
Revised: We use the notion of ‘converging to a subspace’ following the oversmoothing analysis presented in Oono & Suzuki (2020), where the subspace M (where oversmoothing is prevalent) is defined.

Original: We will now show how the slope of the ReLU activation function affects oversmoothing, starting with some important assumptions: (a) the non-exploding gradients assumption, and (b) ReLU’s probability not to output zero.
Revised: We will now show how the slope of the ReLU activation function affects oversmoothing, starting with some important assumptions: (a) the non-exploding gradients assumption, and (b) ReLU’s probability not to output zero.

Original: We use these assumptions to extract bounds on the weights of the network and then use these bounds to determine the relationship between the largest singular value and the elements of each weight matrix.
Revised: We use these assumptions to extract bounds on the weights of the network and then use these bounds to determine the relationship between the largest singular value and the elements of each weight matrix.

Original: Given that relationship, we connect our analysis with existing literature regarding oversmoothing through Theorem 1.
Revised: Given that relationship, we connect our analysis with existing literature regarding oversmoothing through Theorem 1.

Original: Assumption 1 For each layer l there exists a number G l which is the upper bound to the gradients of the output of the subsequent layer (l+1) with respect to the weight elements of W ( l )
Revised: Assumption 1 For each layer l there exists a number G l which is the upper bound to the gradients of the output of the subsequent layer (l+1) with respect to the weight elements of W ( l )

Original: [Equation]
Revised: [Equation]

Original: where O ( l +1) is the output of (l+1)-th layer.
Revised: where O ( l +1) is the output of (l+1)-th layer.

Original: This assumption needs to hold in all cases, in order to avoid the exploding gradient case, that would not allow the learning process to converge.
Revised: This assumption needs to hold in all cases, in order to avoid the exploding gradient case, that would not allow the learning process to converge.

Original: Assumption 2 The probability, that the output of the ReLU function not equals zero at layer l is independent of the outputs of the previous layers, in a feed-forward ReLU neural network, i.e.
Revised: Assumption 2 The probability, that the output of the ReLU function not equals zero at layer l is independent of the outputs of the previous layers, in a feed-forward ReLU neural network, i.e.

Original: [Equation]
Revised: [Equation]

Original: where h ( l ) is the node representation at layer l , ReLU is applied piece-wise on that representation and ⃗ 0 is the all zero vector.
Revised: where h ( l ) is the node representation at layer l , ReLU is applied piece-wise on that representation and ⃗ 0 is the all zero vector.

Original: Following the results of Lu et al.
Revised: Following the results of Lu et al.

Original: we get that P ( ReLU ( h ( l ) ) ̸
Revised: we get that P ( ReLU ( h ( l ) ) ̸

Original: = ⃗ 0 |∀ h ( j ) , j < l ) = 

Revised: = ⃗ 0 |∀ h ( j ) , j < l ) =

Original: /for feed-forward networks (FFs), on the condition that the output of the previous layers is positive.
Revised: /for feed-forward networks (FFs), on the condition that the output of the previous layers is positive.

Original: That condition does not necessary hold in GNNs, due to the role of the adjacency matrix in the aggregation scheme.
Revised: That condition does not necessary hold in GNNs, due to the role of the adjacency matrix in the aggregation scheme.

Original: In FFs, when the output of a layer is zero, so are the outputs of all subsequent layers.
Revised: In FFs, when the output of a layer is zero, so are the outputs of all subsequent layers.

Original: In GNNs there might be cases, where the output of layer l (a node representation) is zero but the aggregation using the adjacency matrix produces a non-zero representation in layer ( l + 1).
Revised: In GNNs there might be cases, where the output of layer l (a node representation) is zero but the aggregation using the adjacency matrix produces a non-zero representation in layer ( l + 1).

Original: Apart from this difference, GNNs can be considered a special case of FFs, with the proper wiring due to the adjacency matrix aggregation.
Revised: Apart from this difference, GNNs can be considered a special case of FFs, with the proper wiring due to the adjacency matrix aggregation.

Original: Therefore, the above assumption holds also for GNNs.
Revised: Therefore, the above assumption holds also for GNNs.

Original: Lemma 1 For a network of depth dep the total gradient reaching to the l-th layer
Revised: Lemma 1 For a network of depth dep the total gradient reaching to the l-th layer

Original: [Equation]
Revised: [Equation]

Original: in order to update W ( l ) is bounded by:
Revised: in order to update W ( l ) is bounded by:

Original: [Equation]
Revised: [Equation]

Original: Where J is the model’s loss function (i.e. Cross Entropy), α stands for the ReLU slope and G l is the upper bound of gradients of the output of the subsequent layer (l+1) with respect to the weight elements of W ( l ) .
Revised: Where J is the model’s loss function (i.e. Cross Entropy), α stands for the ReLU slope and G l is the upper bound of gradients of the output of the subsequent layer (l+1) with respect to the weight elements of W ( l ) .

Original: The proof of Lemma 1 is shown in Appendix A and is based on the chain rule for backpropagation.
Revised: The proof of Lemma 1 is shown in Appendix A and is based on the chain rule for backpropagation.

Original: Given the derivative of the ReLU function and the upper bound G l of Assumption 1, we can derive the bound of Equation 4.
Revised: Given the derivative of the ReLU function and the upper bound G l of Assumption 1, we can derive the bound of Equation 4.

Original: To compute the gradient with respect to weight element w ( l ) i,j of layer l , we repeatedly differentiate nested ReLU functions, leading to a product of their slopes (or zero).
Revised: To compute the gradient with respect to weight element w ( l ) i,j of layer l , we repeatedly differentiate nested ReLU functions, leading to a product of their slopes (or zero).

Original: In the final differentiation step, we get the gradient of the output of ( l + 1)-th layer with respect to w ( l ) i,j , which is bounded by G l .
Revised: In the final differentiation step, we get the gradient of the output of ( l + 1)-th layer with respect to w ( l ) i,j , which is bounded by G l .

Original: Lemma 

Revised: Lemma

Original: While model’s loss, through gradients, flows backwards some weight elements do not receive updates, because we have dying ReLUs (i.e. ReLUs outputting zero)
Revised: While model’s loss, through gradients, flows backwards some weight elements do not receive updates, because we have dying ReLUs (i.e. ReLUs outputting zero)

Original: (Lu et al., 2019).
Revised: (Lu et al., 2019).

Original: The total number of weight elements getting updated at layer l is bounded by:
Revised: The total number of weight elements getting updated at layer l is bounded by:

Original: [Equation]
Revised: [Equation]

Original: where d is the largest of the two dimensions of W ( l ) , i.e. there are at most d 2 elements in W ( l ) , if it is a square matrix.
Revised: where d is the largest of the two dimensions of W ( l ) , i.e. there are at most d 2 elements in W ( l ) , if it is a square matrix.

Original: The proof of Lemma 2 appears in Appendix B. Given Lemma 1, the gradient flowing backwards, with respect to a weight element w ( l ) i,j contains a product of ReLU derivatives.
Revised: The proof of Lemma 2 appears in Appendix B. Given Lemma 1, the gradient flowing backwards, with respect to a weight element w ( l ) i,j contains a product of ReLU derivatives.

Original: In order for w ( l ) i,j to get an update, all ReLU derivatives need to be non-zero, otherwise the gradient will be zeroed and no update will be applied to w ( l ) i,j .
Revised: In order for w ( l ) i,j to get an update, all ReLU derivatives need to be non-zero, otherwise the gradient will be zeroed and no update will be applied to w ( l ) i,j .

Original: Additionally, due to Assumption 2 weight elements located in the lower weight matrices tend to receive fewer updates.
Revised: Additionally, due to Assumption 2 weight elements located in the lower weight matrices tend to receive fewer updates.

Original: This is due to the fact, that the further the gradient flows backwards the more ReLU derivative factors appear in it.
Revised: This is due to the fact, that the further the gradient flows backwards the more ReLU derivative factors appear in it.

Original: Based on Assumption 2, the probability of all of them to be non-zero decreases.
Revised: Based on Assumption 2, the probability of all of them to be non-zero decreases.

Original: Regarding the singular values (denoted by s l ( · ) ) of a weight matrix W ( l ) at the l -th layer, the largest of them is given by: max ( s l ( W ( l ) )
Revised: Regarding the singular values (denoted by s l ( · ) ) of a weight matrix W ( l ) at the l -th layer, the largest of them is given by: max ( s l ( W ( l ) )

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: l ) i,j | 2 , where || · || F is the Frobenius norm and || · || 2 is the spectral norm.
Revised: l ) i,j | 2 , where || · || F is the Frobenius norm and || · || 2 is the spectral norm.

Original: Using the general weight updating rule (cid:18) i.e., w
Revised: Using the general weight updating rule (cid:18) i.e., w

Original: [Equation]
Revised: [Equation]

Original: ) new i,
Revised: ) new i,

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: , where J is the model’s loss and η is the learning rate, we arrive at the main theorem of this work.
Revised: , where J is the model’s loss and η is the learning rate, we arrive at the main theorem of this work.

Original: Theorem 

Revised: Theorem

Original: The upper bound of the largest singular value of the weight matrix W ( l ) at layer l for a
Revised: The upper bound of the largest singular value of the weight matrix W ( l ) at layer l for a

Original: GNN model, utilizing a ReLU activation function, depends on the slope of the function.
Revised: GNN model, utilizing a ReLU activation function, depends on the slope of the function.

Original: That bound is given per layer and shows the effect of each iteration of updates on the weight matrix.
Revised: That bound is given per layer and shows the effect of each iteration of updates on the weight matrix.

Original: We denote with W old and W new the weight matrices before and after the update during an iteration of the training process respectively.
Revised: We denote with W old and W new the weight matrices before and after the update during an iteration of the training process respectively.

Original: [Equation]
Revised: [Equation]

Original: where B l = η · G l , dep is the network’s depth, d is the largest dimension of the W ( l ) matrix, p is the upper bound of the probability of ReLU not to output zero and α is ReLU’s slope.
Revised: where B l = η · G l , dep is the network’s depth, d is the largest dimension of the W ( l ) matrix, p is the upper bound of the probability of ReLU not to output zero and α is ReLU’s slope.

Original: The proof of Theorem 2 appears in Appendix C and uses the upper bound of the largest singular value by the Frobenius norm, expanded according to the weight update rule.
Revised: The proof of Theorem 2 appears in Appendix C and uses the upper bound of the largest singular value by the Frobenius norm, expanded according to the weight update rule.

Original: Separating weight elements into two sets (updated and not updated) we identify the gradient values responsible for the weight updates.
Revised: Separating weight elements into two sets (updated and not updated) we identify the gradient values responsible for the weight updates.

Original: These values are bounded (Lemma 1), as is the number of updated elements (Lemma 2), leading to Equation 6.
Revised: These values are bounded (Lemma 1), as is the number of updated elements (Lemma 2), leading to Equation 6.

Original: The resulting upper bound shows the connection of the slope of
Revised: The resulting upper bound shows the connection of the slope of

Original: ReLU and the upper bound of the probability of ReLU not to output zero with the largest singular value of the weight matrix.
Revised: ReLU and the upper bound of the probability of ReLU not to output zero with the largest singular value of the weight matrix.

Original: That largest singular value is connected with the oversmoothing problem and as we mentioned above could act as a resource to reduce it.
Revised: That largest singular value is connected with the oversmoothing problem and as we mentioned above could act as a resource to reduce it.

Original: 3. A LLEVIATING O VERSMOOTHING
Revised: 3. A LLEVIATING O VERSMOOTHING

Original: We transfer the idea of SeLU to GNNs, focusing on the part that increases the variance, because the repetition of the Laplacian operator acts as a variance reducer.
Revised: We transfer the idea of SeLU to GNNs, focusing on the part that increases the variance, because the repetition of the Laplacian operator acts as a variance reducer.

Original: In order to avoid oversmoothing in deep GNNs using this approach, we need to identify a ‘sweet’ spot, where variance reduction from graph convolution counteracted as needed by the slope of the activation function.
Revised: In order to avoid oversmoothing in deep GNNs using this approach, we need to identify a ‘sweet’ spot, where variance reduction from graph convolution counteracted as needed by the slope of the activation function.

Original: Typically in GNNs, ReLU is used with a slope value α = 1 .
Revised: Typically in GNNs, ReLU is used with a slope value α = 1 .

Original: As a result, the second exponential factor in
Revised: As a result, the second exponential factor in

Original: Equation 6 can be ignored (it is always equal to 1).
Revised: Equation 6 can be ignored (it is always equal to 1).

Original: This pushes the bound for the largest singular value of weight matrices towards a fixed low value, when the architecture of the network gets deeper ( dep increases), and especially at the lower layers (small l ) of the network.
Revised: This pushes the bound for the largest singular value of weight matrices towards a fixed low value, when the architecture of the network gets deeper ( dep increases), and especially at the lower layers (small l ) of the network.

Original: This is due to the first exponential factor of (6) converging to zero for large values of dep .
Revised: This is due to the first exponential factor of (6) converging to zero for large values of dep .

Original: The restriction of the largest singular values to low values, increases the speed of convergence to the oversmoothng subspace, as stated in Corollary 1 from Oono & Suzuki (2020).
Revised: The restriction of the largest singular values to low values, increases the speed of convergence to the oversmoothng subspace, as stated in Corollary 1 from Oono & Suzuki (2020).

Original: According to the corollary, the speed depends on λ and s. The former parameter ( λ ) is a property of the data (largest eigenvalue of the adjacency matrix), while the latter (s) is the product of the largest singular values of the weight matrices.
Revised: According to the corollary, the speed depends on λ and s. The former parameter ( λ ) is a property of the data (largest eigenvalue of the adjacency matrix), while the latter (s) is the product of the largest singular values of the weight matrices.

Original: Realizing the importance of the slope of ReLU in the training process, we move on to propose a modification of the activation function that reduces oversmoothing.
Revised: Realizing the importance of the slope of ReLU in the training process, we move on to propose a modification of the activation function that reduces oversmoothing.

Original: Following (Lu et al., 2019), we proceed our analysis with p = 1 / 2 (see Assumption 2), which simplifies calculations.
Revised: Following (Lu et al., 2019), we proceed our analysis with p = 1 / 2 (see Assumption 2), which simplifies calculations.

Original: In particular, we observe that a slope of 2 makes the second exponential factor prevail over the first one, leading to a new combined factor of 2( dep − l 2 ) .
Revised: In particular, we observe that a slope of 2 makes the second exponential factor prevail over the first one, leading to a new combined factor of 2( dep − l 2 ) .

Original: This in turn increases the upper bound for the largest singular value, restricting the influence of the layer index ( l ) and depth ( dep ) in Equation 6.
Revised: This in turn increases the upper bound for the largest singular value, restricting the influence of the layer index ( l ) and depth ( dep ) in Equation 6.

Original: It is worth noting, that the upper bound should not be constant across all layers (which could be achieved by setting a corresponding value for the slope), because different layers of the network have different roles and the goal of a GNN is to bring intra-class representations close, while keeping inter-class representations apart.
Revised: It is worth noting, that the upper bound should not be constant across all layers (which could be achieved by setting a corresponding value for the slope), because different layers of the network have different roles and the goal of a GNN is to bring intra-class representations close, while keeping inter-class representations apart.

Original: Using the results of (Oono & Suzuki, 2020) we can further connect the choice of slope for the activation function to the speed of convergence to the oversmoothing subspace.
Revised: Using the results of (Oono & Suzuki, 2020) we can further connect the choice of slope for the activation function to the speed of convergence to the oversmoothing subspace.

Original: Hence, the proposed method increases the bound for each largest singular value, which in turn decreases the speed of convergence to the oversmoothing subspace, according to Corollary 1.
Revised: Hence, the proposed method increases the bound for each largest singular value, which in turn decreases the speed of convergence to the oversmoothing subspace, according to Corollary 1.

Original: 3. M ODIFYING THE SLOPE OF R E LU: L IMITATIONS
Revised: 3. M ODIFYING THE SLOPE OF R E LU: L IMITATIONS

Original: Oono & Suzuki (2020) have proved that any GNN deep enough will eventually reach the oversmoothing subspace.
Revised: Oono & Suzuki (2020) have proved that any GNN deep enough will eventually reach the oversmoothing subspace.

Original: In theory, this also holds for our method, when the GNN gets very deep.
Revised: In theory, this also holds for our method, when the GNN gets very deep.

Original: Could this be avoided by increasing the slope of ReLU further?
Revised: Could this be avoided by increasing the slope of ReLU further?

Original: Unfortunately not, as we cannot make the slope too large.
Revised: Unfortunately not, as we cannot make the slope too large.

Original: It is known, that by increasing the slope of ReLU too much, one may face the problem of exploding gradients (Bengio et al., 1994), that impedes the learning procedure.
Revised: It is known, that by increasing the slope of ReLU too much, one may face the problem of exploding gradients (Bengio et al., 1994), that impedes the learning procedure.

Original: In fact, if we increase the slope of ReLU too much, the upper bound in Equation 6 becomes too loose and the performance degrades.
Revised: In fact, if we increase the slope of ReLU too much, the upper bound in Equation 6 becomes too loose and the performance degrades.

Original: Our proof suggests, that a slope equal to 2 avoids oversmoothing and our experiments have shown that it also avoids the exploding gradients problem.
Revised: Our proof suggests, that a slope equal to 2 avoids oversmoothing and our experiments have shown that it also avoids the exploding gradients problem.

Original: 3. M ODIFYING THE LEARNING RATE , INSTEAD OF THE SLOPE OF R E LU
Revised: 3. M ODIFYING THE LEARNING RATE , INSTEAD OF THE SLOPE OF R E LU

Original: Instead of changing the slope of ReLU, we could opt to modify the Learning Rate (LR) per layer in order to increase the upper bound in Equation 6.
Revised: Instead of changing the slope of ReLU, we could opt to modify the Learning Rate (LR) per layer in order to increase the upper bound in Equation 6.

Original: In particular, a different learning rate ( η ) per layer leads to a different B l (= η · G l ) per layer, counteracting the problematic first exponential factor of
Revised: In particular, a different learning rate ( η ) per layer leads to a different B l (= η · G l ) per layer, counteracting the problematic first exponential factor of

Original: Equation 6.
Revised: Equation 6.

Original: However, determining the right LR per layer is a non-trivial task.
Revised: However, determining the right LR per layer is a non-trivial task.

Original: Intuitively, LR should be larger in the lower layers and smaller in the upper ones.
Revised: Intuitively, LR should be larger in the lower layers and smaller in the upper ones.

Original: Keeping the slope equal to 1, the first problematic exponential term is smaller (note p < 1 ) in the lower layers reducing the value of the upper bound.
Revised: Keeping the slope equal to 1, the first problematic exponential term is smaller (note p < 1 ) in the lower layers reducing the value of the upper bound.

Original: To avoid this, one needs to tune LR carefully.
Revised: To avoid this, one needs to tune LR carefully.

Original: Additionally, LR affects heavily the learning process and large LR values might lead to oscillations and poor learning performance.
Revised: Additionally, LR affects heavily the learning process and large LR values might lead to oscillations and poor learning performance.

Original: Modifying the slope and the learning rate combined led to negligible improvements.
Revised: Modifying the slope and the learning rate combined led to negligible improvements.

Original: Preliminary results with modified
Revised: Preliminary results with modified

Original: LR values are shown in Appendix D.
Revised: LR values are shown in Appendix D.

Original: 3. W HY DO WE NEED DEEP GNN S ?
Revised: 3. W HY DO WE NEED DEEP GNN S ?

Original: Oversmoothing is only a problem for deep GNNs, leading to the question of whether and when deep
Revised: Oversmoothing is only a problem for deep GNNs, leading to the question of whether and when deep

Original: GNNs are really needed.
Revised: GNNs are really needed.

Original: Most of the existing benchmark datasets for various graph analytics tasks do not seem to justify the need for deep networks.
Revised: Most of the existing benchmark datasets for various graph analytics tasks do not seem to justify the need for deep networks.

Original: Due to their homophilic nature, useful information for each node resides in its close neighbors (usually 2 or 3 hops away).
Revised: Due to their homophilic nature, useful information for each node resides in its close neighbors (usually 2 or 3 hops away).

Original: A task where deeper architectures could be needed is the “cold start” problem; namely the situation where many node features are missing.
Revised: A task where deeper architectures could be needed is the “cold start” problem; namely the situation where many node features are missing.

Original: The problem of missing features is called “cold start”, because it resembles the situation of a new product/user arriving to a recommender system.
Revised: The problem of missing features is called “cold start”, because it resembles the situation of a new product/user arriving to a recommender system.

Original: The system has no prior knowledge about the new arrival, yet has to make some recommendations.
Revised: The system has no prior knowledge about the new arrival, yet has to make some recommendations.

Original: In this scenario, the hope is that deeper GNNs could recover features from more distant nodes, in order to create informative representations.
Revised: In this scenario, the hope is that deeper GNNs could recover features from more distant nodes, in order to create informative representations.

Original:  E XPERIMENTS 4. E XPERIMENTAL S ETUP
Revised:  E XPERIMENTS 4. E XPERIMENTAL S ETUP

Original: Datasets: Aligned to most of the literature, we use four well-known benchmark datasets: Cora,
Revised: Datasets: Aligned to most of the literature, we use four well-known benchmark datasets: Cora,

Original: CiteSeer, Pubmed and the less homophilic dataset Texas .
Revised: CiteSeer, Pubmed and the less homophilic dataset Texas .

Original: The statistics of the datasets are reported in the Appendix.
Revised: The statistics of the datasets are reported in the Appendix.

Original: For Cora, CiteSeer and Pubmed we use the same data splits as in Kipf & Welling (2017), where all nodes except the ones used for training and validation are used for testing.
Revised: For Cora, CiteSeer and Pubmed we use the same data splits as in Kipf & Welling (2017), where all nodes except the ones used for training and validation are used for testing.

Original: For
Revised: For

Original: Texas we use the same splits as in (Pei et al., 2020).
Revised: Texas we use the same splits as in (Pei et al., 2020).

Original: Models: We utilize two different GNN models as our base models for the proposed methodology; namely GCN (Kipf & Welling, 2017) and GAT (Velickovic et al., 2018).
Revised: Models: We utilize two different GNN models as our base models for the proposed methodology; namely GCN (Kipf & Welling, 2017) and GAT (Velickovic et al., 2018).

Original: We compare the results of these models with and without the modification of the slope of ReLU for varying number of layers.
Revised: We compare the results of these models with and without the modification of the slope of ReLU for varying number of layers.

Original: We do not compare against methods utilizing residual connections, because the aim of this work is to show that oversmoothing could be avoided without “short-circuiting” initial information to latter layers.
Revised: We do not compare against methods utilizing residual connections, because the aim of this work is to show that oversmoothing could be avoided without “short-circuiting” initial information to latter layers.

Revised: We use residual GNNs as a baseline in the Extended Experiments section, Appendix F.

Original: Hyperparameters: We set the number of hidden units (of each layer) of GCN and GAT to 128 for both models across all datasets.
Revised: Hyperparameters: We set the number of hidden units (of each layer) of GCN and GAT to 128 for both models across all datasets.

Original: The L 2 regularization is set with penalty 5 · 10 − 4 for both models.
Revised: The L 2 regularization is set with penalty 5 · 10 − 4 for both models and learning rate set to 10 − 3 .

Original: We vary the depth between 2 and 64 layers.
Revised: We vary the depth between 2 and 64 layers.

Original: The number of attention heads for GAT is set to 1.
Revised: The number of attention heads for GAT is set to 1.

Original: Configuration: Each experiment is run 10 times and we report the average performance over these runs.
Revised: Configuration: Each experiment is run 10 times and we report the average performance over these runs.

Original: We train all models for 200 epochs using Cross Entropy as a loss function.
Revised: We train all models for 200 epochs using Cross Entropy as a loss function.

Original: For our experiments we used an RTX 3080ti GPU.
Revised: For our experiments we used an RTX 3080ti GPU.

Original: 4. E XPERIMENTAL R ESULTS Reducing oversmoothing:
Revised: 4. E XPERIMENTAL R ESULTS Reducing oversmoothing:

Original: Table 1 presents the classification performance of the two base models (GCN and GAT) on all four datasets, with and without the modified slope of ReLU (called Slope2GNN) for varying number of layers.
Revised: Table 1 presents the classification performance of the two base models (GCN and GAT) on all four datasets, with and without the modified slope of ReLU (called Slope2GNN) for varying number of layers.

Original: Additionally, it presents results using SeLU, instead of ReLU, since it also modifies the slope of the function, while additionally normalizing node representations, as explained in section 2.4.
Revised: Additionally, it presents results using SeLU, instead of ReLU, since it also modifies the slope of the function, while additionally normalizing node representations, as explained in section 2.4.

Original: To the best of our knowledge, this is the first time SeLU is used with GNNs.
Revised: To the best of our knowledge, this is the first time SeLU is used with GNNs.

Original: Table 1 shows that methods with a modified activation function reduce oversmoothing significantly.
Revised: Table 1 shows that methods with a modified activation function reduce oversmoothing significantly.

Original: As expected, baselines maintain a high performance for shallow architectures in homophilic datasets, where oversmoothing is not a problem.
Revised: As expected, baselines maintain a high performance for shallow architectures in homophilic datasets, where oversmoothing is not a problem.

Original: The modified activation functions (Slope2GNN and SeLU) consistently improve the testing accuracy as the number of layers ( # L ) increases.
Revised: The modified activation functions (Slope2GNN and SeLU) consistently improve the testing accuracy as the number of layers ( # L ) increases.

Original: Note that even the proposed method suffers performance degradation for very deep GNNs, due to the unavoidable nature of the oversmoothing (Oono & Suzuki, 2020).
Revised: Note that even the proposed method suffers performance degradation for very deep GNNs, due to the unavoidable nature of the oversmoothing (Oono & Suzuki, 2020).

Original: GCN and GAT average over information in the close neighborhood of the node, assuming homophily, which is not the case in the Texas dataset.
Revised: GCN and GAT average over information in the close neighborhood of the node, assuming homophily, which is not the case in the Texas dataset.

Original: Interestingly, low homophily in this dataset makes oversmoothing appear at a larger depth compared to other datasets.
Revised: Interestingly, low homophily in this dataset makes oversmoothing appear at a larger depth compared to other datasets.

Original: This is because connected nodes do not have similar representations and the model needs more Laplacian operations to oversmooth them.
Revised: This is because connected nodes do not have similar representations and the model needs more Laplacian operations to oversmooth them.

Original: The effect of deeper networks on different activation functions:
Revised: The effect of deeper networks on different activation functions:

Original: Focusing on the use of SeLU, although it seems to make the models resistant to oversmoothing, the effect seems to be lost for GCNs with 64 layers.
Revised: Focusing on the use of SeLU, although it seems to make the models resistant to oversmoothing, the effect seems to be lost for GCNs with 64 layers.

Original: This may be due to the sensitivity of the function on the choice of slope value, which is lower than 2.
Revised: This may be due to the sensitivity of the function on the choice of slope value, which is lower than 2.

Original: Moreover, the saturating region (part of SeLU used to reduce variance of data, if needed) of SeLU might further reduce the variance of node representations, acting in favor of oversmoothing.
Revised: Moreover, the saturating region (part of SeLU used to reduce variance of data, if needed) of SeLU might further reduce the variance of node representations, acting in favor of oversmoothing.

Original: Finally, SNNs aim to keep the variance of the model stable, ignoring the effect of the Laplacian smoothing in GNNs.
Revised: Finally, SNNs aim to keep the variance of the model stable, ignoring the effect of the Laplacian smoothing in GNNs.

Original: To further examine the behavior of each method, as the number of layers increases, in Figure 1, we present a more detailed progression of the accuracy of GCNs, as the networks get deeper on the
Revised: To further examine the behavior of each method, as the number of layers increases, in Figure 1, we present a more detailed progression of the accuracy of GCNs, as the networks get deeper on the

Original: Cora dataset.
Revised: Cora dataset.

Original: The effect of oversmoothing and its reduction by the two methods that modify ReLU is very apparent here.
Revised: The effect of oversmoothing and its reduction by the two methods that modify ReLU is very apparent here.

Original: Furthermore, the simpler Slope2GNN approach seems to be more resistant to the increase in the depth of the network.
Revised: Furthermore, the simpler Slope2GNN approach seems to be more resistant to the increase in the depth of the network.

Original: The need for deep networks:
Revised: The need for deep networks:

Original: Having shown the benefit of using the modified activation functions in deep GNNs, we move to a set of experiments that aims to highlight also the value of using such deep architectures.
Revised: Having shown the benefit of using the modified activation functions in deep GNNs, we move to a set of experiments that aims to highlight also the value of using such deep architectures.

Original: In particular, we report our experimental results in the “cold start” scenario, as described in section 3.5.
Revised: In particular, we report our experimental results in the “cold start” scenario, as described in section 3.5.

Original: The cold-start datasets that we use in these experiments are generated by removing feature vectorsfrom unlabeled nodes and replacing them with all-zero vectors. For each combination of GNN and
Revised: The cold-start datasets that we use in these experiments are generated by removing feature vectors

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: from unlabeled nodes and replacing them with all-zero vectors.

Original: [Table] [Table] [Table] activation function, we present in Table 2 the best performance achieved and the depth at which themodel attains that performance.
Revised: For each combination of GNN and activation function, we present in Table 2 the best performance achieved and the depth at which the attains that performance.

Original: The main observation in these results is the improvement in terms of accuracy with the use of deeper GNNs.
Revised: The main observation in these results is the improvement in terms of accuracy with the use of deeper GNNs.

Original: This improvement is only attainable with the modified activation functions that reduce the effect of oversmoothing.
Revised: This improvement is only attainable with the modified activation functions that reduce the effect of oversmoothing.

Original: The vanilla versions of GCN and GAT cannot go further than the performance of their shallow versions.
Revised: The vanilla versions of GCN and GAT cannot go further than the performance of their shallow versions.

Original: Worth-mentioning is also the fact that the simple Slope2GNN approach manages to benefit from the deeper architectures even in the hard Texas dataset.
Revised: Worth-mentioning is also the fact that the simple Slope2GNN approach manages to benefit from the deeper architectures even in the hard Texas dataset.

Original: This is not the case for SeLU, which may require tuning of the slope of the activation function, as discussed above.
Revised: This is not the case for SeLU, which may require tuning of the slope of the activation function, as discussed above.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Li et al.
Revised: Li et al.

Original: (2018) were the first to introduce the problem of oversmoothing in GNNs.
Revised: (2018) were the first to introduce the problem of oversmoothing in GNNs.

Original: Subsequent work, (Oono & Suzuki, 2020; Cai & Wang, 2020) further analyzed and theoretically proved theexistence
Revised: An initial approach of activation function alternatives to avoid reduction of the rank of the feature space is the

Original: [Table]
Revised: [Table]

Original:  of the problem
Revised: one provided by Luan et al. (2019), where tanh is used instead of ReLU.

Original:  and showed that it is unavoidable as the depth increases.
Revised: Subsequent work, (Oono & Suzuki, 2020; Cai & Wang, 2020) further analyzed and theoretically proved the existence of the problem and showed that it is unavoidable as the depth increases.

Original: Xu et al.
Revised: Xu et al.

Original: (2018) proposed 
Revised: (2018) proposed Jumping

Original: Jumping Knowledge Networks (JK-Networks) as the first attempt to address the problem, by injecting skip connections to the GNNs.
Revised: Knowledge Networks (JK-Networks) as the first attempt to address the problem, by injecting skip connections to the GNNs.

Original: The model kept information from lower layers and combined them directly with the final layer, before labeling the node.
Revised: The model kept information from lower layers and combined them directly with the final layer, before labeling the node.

Original: Following a similar approach, Chen et al.
Revised: Following a similar approach, Chen et al.

Original: (2020) proposed the use of residual connections between layers to enable deep architectures and alleviate the problem.
Revised: (2020) proposed the use of residual connections between layers to enable deep architectures and alleviate the problem.

Original: Both of these approaches explicitly inject part of the initial information into higher layers of the network, implicitly downgrading the importance of intermediate layers.
Revised: Both of these approaches explicitly inject part of the initial information into higher layers of the network, implicitly downgrading the importance of intermediate layers.

Original: On the other hand, our method enables deep architectures without the need to inject initial information and allows the model to learn how to compress the initial information and what amount of it to maintain.
Revised: On the other hand, our method enables deep architectures without the need to inject initial information and allows the model to learn how to compress the initial information and what amount of it to maintain.

Original: We do not compare against these methods, because the aim of this work is to show that oversmoothing could be avoided without “short-circuiting” initial information to latter layers.
Revised: We do not compare against these methods, because the aim of this work is to show that oversmoothing could be avoided without “short-circuiting” initial information to latter layers.

Original: DropEdge (Rong et al., 2020) and DropNode (Do et al., 2021) are two alternative methods that address oversmoothing by altering the graph topology.
Revised: DropEdge (Rong et al., 2020) and DropNode (Do et al., 2021) are two alternative methods that address oversmoothing by altering the graph topology.

Original: In particular, they remove either edges or nodes at random, in order to slow down the message passing speed.
Revised: In particular, they remove either edges or nodes at random, in order to slow down the message passing speed.

Original: In contrast to such approaches, our method does not change the graph topology, which may have unpredictable effects, but rather relies on a small adjustment of the slope of the activation function.
Revised: In contrast to such approaches, our method does not change the graph topology, which may have unpredictable effects, but rather relies on a small adjustment of the slope of the activation function.

Original: An additional advantage of the approach proposed in this paper is that it does not have any hyper-parameters to be tuned, in contrast to all of the alternatives mentioned above.
Revised: An additional advantage of the approach proposed in this paper is that it does not have any hyper-parameters to be tuned, in contrast to all of the alternatives mentioned above.

Original: Hyperparameter tuning is also required by the Pairnorm method, which was mentioned in section 2.4.
Revised: Hyperparameter tuning is also required by the Pairnorm method, which was mentioned in section 2.4.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we have shown the important role that the slope of the ReLU activation function plays in the oversmoothing problem in GNNs.
Revised: In this paper we have shown the important role that the slope of the ReLU activation function plays in the oversmoothing problem in GNNs.

Original: We have further proposed a simple modification that reduces drastically the problem.
Revised: We have further proposed a simple modification that reduces drastically the problem.

Original: We have illustrated the benefits of the approach in a set of experiments with different datasets and GNNs of different depth.
Revised: We have illustrated the benefits of the approach in a set of experiments with different datasets and GNNs of different depth.

Original: Additionally, we showed the improvement in accuracy one can achieve through deep architectures that do not suffer from oversmoothing.
Revised: Additionally, we showed the improvement in accuracy one can achieve through deep architectures that do not suffer from oversmoothing.

Original: This was evident in a set of experiments that simulated the “cold start” problem of missing node features in GNNs.
Revised: This was evident in a set of experiments that simulated the “cold start” problem of missing node features in GNNs.

Original: The simple link between the activation function and oversmoothing unveiled in this paper, opens up a range of interesting hypotheses to be investigated in the future.
Revised: The simple link between the activation function and oversmoothing unveiled in this paper, opens up a range of interesting hypotheses to be investigated in the future.

Original: In particular, we would like to study the effect of changing the activation function in various GNN architectures and learning methods.
Revised: In particular, we would like to study the effect of changing the activation function in various GNN architectures and learning methods.

Original: Additionally, we would like to study alternative activation functions that could provide further benefits.
Revised: Additionally, we would like to study alternative activation functions that could provide further benefits.

Original: Finally, we aim to study in more detail the approach of varying LR per layer.
Revised: Finally, we aim to study in more detail the approach of varying LR per layer.

Original: A PPENDIXA: L EMMA 1 P ROOF
Revised: A PPENDIX

Revised: A L EMMA P ROOF

Original: Lemma 1 For a network of depth dep the total gradient reaching to the l-th layer (cid:18) i.e., dJ dw ( l ) oldi,j (cid:19) in order to update W ( l ) is bounded by:
Revised: Lemma 1 For a network of depth dep the total gradient reaching to the l-th layer (cid:18) i.e., dJ dw ( l ) oldi,j (cid:19) in order to update W ( l ) is bounded by:

Original: [Equation]
Revised: [Equation]

Original: Where J is the model’s loss function (i.e. Cross Entropy), α stands for the ReLU slope and G l is the upper bound of gradients of the output of the subsequent layer (l+1) with respect to the weight elements of W ( l ) .
Revised: Where J is the model’s loss function (i e. Cross Entropy), α stands for the ReLU slope and G l is the upper bound of gradients of the output of the subsequent layer (l+1) with respect to the weight elements of W ( l ) .

Original: Proof: Firstly, let us define the gradient of the output of a ReLU function with respect to the input as follows:
Revised: Proof: Firstly, let us define the gradient of the output of a ReLU function with respect to the input as follows:

Original: [Equation]
Revised: [Equation]

Original: Let us consider now a function of nested ReLUs, like the neural networks under investigation, i.e. f ( x ) =
Revised: Let us consider now a function of nested ReLUs, like the neural networks under investigation, i.e. f ( x ) =

Original: g ( σ ( g ( σ ( ..g ( σ ( g ( x ))) .. )))) , where σ ( · ) is the ReLU function applied repeatedly n times, and g ( · ) is a function that multiplies its input with a value.
Revised: g ( σ ( g ( σ ( ..g ( σ ( g ( x ))) .. )))) , where σ ( · ) is the ReLU function applied repeatedly n times, and g ( · ) is a function that multiplies its input with a value.

Original: Then the gradient of f ( · ) with respect to x (x is a vector) is given as:
Revised: Then the gradient of f ( · ) with respect to x (x is a vector) is given as:

Original: [Equation]
Revised: [Equation]

Original: This formula indicates the relation between slope, layer index, depth and largest singular value (we only care about the largest).
Revised: This formula indicates the relation between slope, layer index, depth and largest singular value (we only care about the largest).

Original: Let as use the notion of J for the loss function of the network (i.e. Cross
Revised: Let as use the notion of J for the loss function of the network (i.e. Cross

Original: Entropy).
Revised: Entropy).

Original: The relation comes from the following formulas (for each weight element in the weight matrix):
Revised: The relation comes from the following formulas (for each weight element in the weight matrix):

Original: [Equation]
Revised: [Equation]

Original: Where the 0 value comes from Equation 7.
Revised: Where the 0 value comes from Equation 7.

Original: GNNs’ forward pass is similar to f ( · ) , hence performing the backpropagation leads to gradient calculation of the form of Equation 7.
Revised: GNNs’ forward pass is similar to f ( · ) , hence performing the backpropagation leads to gradient calculation of the form of Equation 7.

Original: So the update rule adds (or subtracts) to each weight a big number (contains exponential factor).
Revised: So the update rule adds (or subtracts) to each weight a big number (contains exponential factor).

Original: We prove, that dJ dw oldi,j tends to be closer to zero as the layer index gets smaller, because when the gradient travels backwards the more distant it travels the more probable it is to die.
Revised: We prove, that dJ dw oldi,j tends to be closer to zero as the layer index gets smaller, because when the gradient travels backwards the more distant it travels the more probable it is to die.

Original: There exist many components (gradients of ReLU functions) in the total gradient so it is probable one of them to be zero and the total gradient to be zero.
Revised: There exist many components (gradients of ReLU functions) in the total gradient so it is probable one of them to be zero and the total gradient to be zero.

Original: Given Assumption 1, we get an upper bound regarding the gradient of the output of the subsequent layer with respect to each weight element of W ( l ) (cid:18) i.e. [Equation]
Revised: Given Assumption 1, we get an upper bound regarding the gradient of the output of the subsequent layer with respect to each weight element of W ( l ) (cid:18) i.e., dO ( l +1) w ( l ) oldi,j ≤ G l (cid:19) .

Original: So we will have that:
Revised: So we will have that:

Original: [Equation]
Revised: [Equation]

Original: (8) where B l is an upper bound on the product of the learning rate η with G l , i.e. B l = η · G l .
Revised: where B l is an upper bound on the product of the learning rate η with G l , i.e. B l = η · G l .

Original: B: L EMMA 2 P ROOF
Revised: [Equation]

Original: Lemma 

Revised: Lemma

Original: While model’s loss, through gradients, flows backwards some weight elements do not receive updates, because we have dying ReLUs (i.e. ReLUs outputting zero)
Revised: While model’s loss, through gradients, flows backwards some weight elements do not receive updates, because we have dying ReLUs (i.e. ReLUs outputting zero)

Original: (Lu et al., 2019).
Revised: (Lu et al., 2019).

Original: The total number of weight elements getting updated at layer l is bounded by:
Revised: The total number of weight elements getting updated at layer l is bounded by:

Original: [Equation]
Revised: [Equation]

Original: where d is the largest of the two dimensions of W ( l ) , i.e. there are at most d 2 elements in W ( l ) , if it is a square matrix.
Revised: where d is the largest of the two dimensions of W ( l ) , i.e. there are at most d 2 elements in W ( l ) , if it is a square matrix.

Original: Proof: Let us assume that the number of weights that get updated is reduced as the gradient flows backward.
Revised: Proof: Let us assume that the number of weights that get updated is reduced as the gradient flows backward.

Original: This is due to the fact that, the further the gradient ‘travels’ the more gradients of ReLUs will exist within it.
Revised: This is due to the fact that, the further the gradient ‘travels’ the more gradients of ReLUs will exist within it.

Original: Hence the greater is the probability at least one of them to be zero.
Revised: Hence the greater is the probability at least one of them to be zero.

Original: In fact, we define the probability of a weight element to get updated and the number of weight elements to be updated in layer l as:
Revised: In fact, we define the probability of a weight element to get updated and the number of weight elements to be updated in layer l as:

Original: [Equation]
Revised: [Equation]

Original: Where i , j are the dimensions of the weight matrix and d is the largest of them.
Revised: Where i , j are the dimensions of the weight matrix and d is the largest of them.

Original: The probability for a weight element to get updated has the aforementioned form, because we have used the Assumption 2 regarding the probability of a ReLU unit not to output zero, which is at most p .
Revised: The probability for a weight element to get updated has the aforementioned form, because we have used the Assumption 2 regarding the probability of a ReLU unit not to output zero, which is at most p .

Original: Thus, the probability of getting updated is the probability of the gradient flowing backwards to reach the weight element, which in turn means not to have any ReLU component equal to zero.
Revised: Thus, the probability of getting updated is the probability of the gradient flowing backwards to reach the weight element, which in turn means not to have any ReLU component equal to zero.

Original: The total number of elements that get updated is upper bounded (using union bound) on the sum of the probabilities of all elements of the weight matrix.
Revised: The total number of elements that get updated is upper bounded (using union bound) on the sum of the probabilities of all elements of the weight matrix.

Original: In fact, every weight matrix has i · j elements where i, j are the matrix dimensions and d = max ( i, j ) .
Revised: In fact, every weight matrix has i · j elements where i, j are the matrix dimensions and d = max ( i, j ) .

Original: C: T HEOREM 2 P ROOF
Revised: C: T HEOREM 2 P ROOF

Original: Theorem 

Revised: Theorem

Original: The upper bound of the largest singular value of the weight matrix W ( l ) at layer l for a
Revised: The upper bound of the largest singular value of the weight matrix W ( l ) at layer l for a

Original: GNN model, utilizing a ReLU activation function, depends on the slope of the function.
Revised: GNN model, utilizing a ReLU activation function, depends on the slope of the function.

Original: That bound is given per layer and shows the effect of each iteration of updates on the weight matrix.
Revised: That bound is given per layer and shows the effect of each iteration of updates on the weight matrix.

Original: We denote with W old and W new the weight matrices before and after the update during an iteration of the training process respectively.
Revised: We denote with W old and W new the weight matrices before and after the update during an iteration of the training process respectively.

Original: [Equation]
Revised: [Equation]

Original: where B l = η · G l , dep is network’s depth, d is the largest dimension of W ( l ) matrix, p is the upper bound of the probability of ReLU not to output zero and α is ReLU’s slope.
Revised: where B l = η · G l , dep is network’s depth, d is the largest dimension of W ( l ) matrix, p is the upper bound of the probability of ReLU not to output zero and α is ReLU’s slope.

Original: Proof: Regarding the singular values of a matrix (denoted by s l ), it is known that:
Revised: Proof: Regarding the singular values of a matrix (denoted by s l ), it is known that:

Original: [Equation]
Revised: [Equation]

Original: the matrix here being W ( l ) , the weight matrix at layer l .
Revised: the matrix here being W ( l ) , the weight matrix at layer l .

Original: Based on Lemma 1, a new weight, during the weight update process will be given by: w new ∼ α ( depth − l ) · B l .
Revised: Based on Lemma 1, a new weight, during the weight update process will be given by: w new ∼ α ( depth − l ) · B l .

Original: Since only # { w ( l ) i,j } weight elements are getting updated per layer (Lemma 2), in each iteration the Frobenius norm increases by a value, that depends on layer index, depth and slope.
Revised: Since only # { w ( l ) i,j } weight elements are getting updated per layer (Lemma 2), in each iteration the Frobenius norm increases by a value, that depends on layer index, depth and slope.

Original: Specifically, if we define α ( depth − l ) · B l = K l , in order to simplify the formulas, we have: 
Revised: Specifically, if we define α ( depth − l ) · B l = K l , in order to simplify the formulas, we have: [Equation] l

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original:  Using the fact that w ( l ) old i,j < = K l , because K l contains an exponential term and weights are initialized to values close to zero we get:
Revised: [Equation] [Table] [Equation] Using the fact that w ( l ) old i,j < = K l , because K l contains an exponential term and weights are initialized to values close to zero we get:

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: In the second square root we sum over the updated weight elements and K l is independent of them.
Revised: In the second square root we sum over the updated weight elements and K l is independent of them.

Original: Hence, we get:
Revised: Hence, we get:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: So the slope of the function determines the upper bound of the Frobenius norm while training.
Revised: So the slope of the function determines the upper bound of the Frobenius norm while training.

Original: In turn this norm is directly connected as an upper bound to the largest singular value of the matrix.
Revised: In turn this norm is directly connected as an upper bound to the largest singular value of the matrix.

Original: In the aforementioned proof we have used previously defined Lemmas and Assumptions.
Revised: In the aforementioned proof we have used previously defined Lemmas and Assumptions.

Original: D: M ODIFYING LEARNING RATE : LR2GNN
Revised: D: M ODIFYING LEARNING RATE : LR2GNN

Original: As discussed in section 3.4, an alternative to changing the slope of ReLU is to modify the learning rate.
Revised: As discussed in section 3.4, an alternative to changing the slope of ReLU is to modify the learning rate.

Original: In order to test this hypothesis and compare the results to the modified activation function, we performed some preliminary experiments with GCN.
Revised: In order to test this hypothesis and compare the results to the modified activation function, we performed some preliminary experiments with GCN.

Original: In these experiments, we determined a different learning rate per layer, using the validation part of each dataset and the chosen values are shown in Table 3.
Revised: In these experiments, we determined a different learning rate per layer, using the validation part of each dataset and the chosen values are shown in Table 3.

Original: We did this only for the first 8 layers, as the process of tuning the rate proved very cumbersome and time-consuming.
Revised: We did this only for the first 8 layers, as the process of tuning the rate proved very cumbersome and time-consuming.

Original: The main observation seems to be the improved performance over the vanilla GCN for the three simpler datasets, coupled with a low accuracy for the Texas dataset.
Revised: The main observation seems to be the improved performance over the vanilla GCN for the three simpler datasets, coupled with a low accuracy for the Texas dataset.

Original: The comparison against the modified activation function seems inconclusive, but the latter approach wins due to its simplicity.
Revised: The comparison against the modified activation function seems inconclusive, but the latter approach wins due to its simplicity.

Original: If one opted for the modified learning rate, a method to automatically adapt the rate in different layers would be needed.
Revised: If one opted for the modified learning rate, a method to automatically adapt the rate in different layers would be needed.

Original: Deeper architectures extend the complexity and the pool, from which we would have to find the proper values of learning rates.
Revised: Deeper architectures extend the complexity and the pool, from which we would have to find the proper values of learning rates.

Original: Worth to mention, combining different values of LR per layer with Slope2GNN results in negligible improvements.
Revised: Worth to mention, combining different values of LR per layer with Slope2GNN results in negligible improvements.

Original: [Table]
Revised: [Table]

Revised: E: D ATASETS S TATISTICS

Original: [Table]
Revised: [Table]

Revised: F: E XTENDED E XPERIMENTS Extended experimentation, regular datasets:

Revised: Table 5 is an expanded version of Table 1, showing average test node classification accuracy along with the respective standard deviations.

Revised: We have also included GCNII (Chen et al., 2020), which is one of the best performing GNNs utilizing residual connections.

Revised: GCNII reduces oversmoothing through ‘short circuiting’ initial information to subsequent layers of the model.

Revised: The aim of the comparison against GCNII is to show that by changing the slope of ReLU simple models are capable of producing comparable (up to a limit) results to more sophisticated architectures, that use residual connections.

Revised: [Table]

Revised: Extended “Cold start”:

Revised: Table 6 is an expanded version of Table 2 including GCNII’s performance and the standard deviations for all models.

Revised: The “cold start” problem requires deeper models, because nodes containing informative feature vectors are located further from each test node.

Revised: Thus, architectures that utilize deep GNNs could effectively address the problem, if they avoid oversmoothing.

Revised: Once again our method effectively enables deep architectures using simple models (i

Revised: [Table]

Revised: e., GCN, GAT).

Revised: [Table]

Revised: Slope sensitivity results:

Revised: The choice of slope (i.e., α = 2 ) that we proposed is based on Equation 6.

Revised: We have verified experimentally that this slope value helps to reduce the effect of oversmoothing.

Revised: However, this is not the only slope value that could be used.

Revised: Therefore, we experimented with other slope values and present the results in Figure 2.

Revised: Based on the Figure, it is evident that any value between 1.8 and 2.

Revised: reduces oversmoothing, making the exact choice of value less important.

Revised: Visualising the effect of oversmoothing:

Revised: In order to understand the effect of oversmoothing in GNNs and show how our method alleviates it, we visualise node representations using t-SNE (van der Maaten & Hinton, 2008) for varying depth models.

Revised: Figures 3, 4 and 5 visualise the representations for different depth levels and different ReLU slope values.

Revised: Ideally, we would like nodes of the same color (same class) to be close together and well-separated from other classes.

Revised: In all three datasets, we observe how oversmoothing (at slope = 1) leads to a mixing of the classes.

Revised: We also see how the situation changes for slope = 2.

Revised: In those graphs, node representations do not rapidly mix as the depth of the network increases.