Original: C UTTING L ONG G RADIENT F LOWS : D ECOUPLING E ND - TO -E ND B ACKPROPAGATION B ASED ON S UPERVISED C ONTRASTIVE L EARNING
Revised: C UTTING L ONG G RADIENT F LOWS : D ECOUPLING E ND - TO -E ND B ACKPROPAGATION B ASED ON S UPERVISED C ONTRASTIVE L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: End-to-end backpropagation (BP) is the foundation of current deep learning technology.
Revised: End-to-end backpropagation (BP) is the foundation of current deep learning technology.

Original: Unfortunately, as a network becomes deeper, BP becomes inefficient for various reasons.
Revised: Unfortunately, as a network becomes deeper, BP becomes inefficient for various reasons.

Original: This paper proposes a new methodology for decoupling BP to transform a long gradient flow into multiple short ones in order to address the optimization issues caused by long gradient flows.
Revised: This paper proposes a new methodology for decoupling BP to transform a long gradient flow into multiple short ones in order to address the optimization issues caused by long gradient flows.

Original: We report thorough experiments conducted to illustrate the effectiveness of our model compared with BP and associated learning (AL), a state-of-the-art methodology for backpropagation decoupling.
Revised: We report thorough experiments conducted to illustrate the effectiveness of our model compared with BP, Early Exit and associated learning (AL), a state-of-the-art methodology for backpropagation decoupling.

Original: Wewill release the source code for the experiments after acceptance
Revised: We release the experimental code for reproducibility

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Current deep learning technology largely depends on backpropagation- and gradient-based learning methods (e.g., gradient descent) for model training.
Revised: Current deep learning technology largely depends on backpropagation and gradient-based learning methods (e.g., gradient descent) for model training.

Original: Meanwhile, many successful applications rely on extremely deep neural networks; for example, Transformer contains at least 12 layers (most have several sublayers)
Revised: Meanwhile, many successful applications rely on extremely deep neural networks; for example, Transformer contains at least 12 layers (most have several sublayers)

Original: (Vaswani et al., 2017), BERT has 12 to 24 layers (most also have several sublayers)
Revised: (Vaswani et al., 2017), BERT has 12 to 24 layers (most also have several sublayers)

Original: (Devlin et al., 2018), and GoogLeNet has 22 layers (many layers are Inception modules containing several sublayers)
Revised: (Devlin et al., 2018), and GoogLeNet has 22 layers (many layers are Inception modules containing several sublayers)

Original: (Szegedy et al., 2015).
Revised: (Szegedy et al., 2015).

Original: However, training a deepneural network based on backpropagation is inefficient for many reasons.
Revised: However, training a deep network based on backpropagation is inefficient for many reasons.

Original: First, a long gradient flow may suffer from gradient vanishing or explosion (Hochreiter, 1998).
Revised: First, a long gradient flow may suffer from gradient vanishing or explosion (Hochreiter, 1998).

Original: Second, a long gradient flow may lead to unstable gradients in the early layers (the layers close to the input layer)
Revised: Second, a long gradient flow may lead to unstable gradients in the early layers (the layers close to the input layer)

Original: (Nielsen, 2015).
Revised: (Nielsen, 2015).

Original: Third, backpropagation results in backward locking, meaning that the gradient of a network parameter can be computed only when all other gradients on which it depends have been computed (Jaderberg et al., 2017).
Revised: Third, backpropagation results in backward locking, meaning that the gradient of a network parameter can be computed only when all other gradients on which it depends have been computed (Jaderberg et al., 2017).

Original: These issues may become severe bottlenecks, especially when a network is deep.
Revised: These issues may become severe bottlenecks, especially when a network is deep.

Original: To train deep networks more efficiently, researchers have developed various strategies, such as batch normalization, gradient clipping, new activation functions (e.g., ReLU and leaky ReLU), new network architectures (e.g., LSTM (Hochreiter & Schmidhuber, 1997) and ResNet (He et al., 2016)), and many more.
Revised: To train deep networks more efficiently, researchers have developed various strategies, such as batch normalization, gradient clipping, new activation functions (e.g., ReLU and leaky ReLU), new network architectures (e.g., LSTM (Hochreiter & Schmidhuber, 1997)), and many more.

Original: Since a long gradient flow is a root cause of the above issues, a possible way to eliminate these issues is to shorten the length of the gradient flow, for example, by cutting a network into multiple components and assigning a local objective to each component.
Revised: Since a long gradient flow is a root cause of the above issues, a possible way to eliminate these issues is to shorten the length of the gradient flow, for example, by cutting a network into multiple components and assigning a local objective to each component.

Original: In this way, a long gradient flow can be divided into multiple shorter pieces, which should, at least partially, address the issues of vanishing/exploding gradients, unstable gradients in early layers, and backward locking.
Revised: In this way, a long gradient flow can be divided into multiple shorter pieces, which should, at least partially, address the issues of vanishing/exploding gradients, unstable gradients in early layers, and backward locking.

Original: Perhaps the most straightforward approach for assigning a local objective to a component is by adding a local auxiliary classifier that outputs a predicted ˆ y and updates the local parameters based on the difference between ˆ y and the ground-truth target y .
Revised: Perhaps the most straightforward approach for assigning a local objective to a component is by adding a local auxiliary classifier that outputs a predicted ˆ y and updates the local parameters based on the difference between ˆ y and the ground-truth target y .

Original: We call this strategy “Early Exit” in this paper because each such auxiliary classifier can be regarded as an exit of the neural network.
Revised: We call this strategy “Early Exit” in this paper because each such auxiliary classifier can be regarded as an exit of the neural network.

Original: The concept of Early Exit has been used in many previous studies, e.g., Mostafa et al.
Revised: The concept of Early Exit has been used in many previous studies, e.g., Mostafa et al.

Original: ; Teerapittayanon et al.
Revised: ; Teerapittayanon et al.

Original: ; Szegedy et al.
Revised: ; Szegedy et al.

Original: However, most of these studies have used Early Exit for other purposes, e.g., creating multiple prediction paths or helping to obtain gradients for the parameters in the early layers.
Revised: However, most of these studies have used Early Exit for other purposes, e.g., creating multiple prediction paths or helping to obtain gradients for the parameters in the early layers.

Original: Consequently, these studies have not investigated the separation of end-to-end backpropagation (BP) into multiple pieces, and the associated gradient flows are still long.
Revised: Consequently, these studies have not investigated the separation of end-to-end backpropagation (BP) into multiple pieces, and the associated gradient flows are still long.

Original: In addition, even if Early Exit is used to isolate the gradient flow, as shown in (Mostafa et al., 2018), the test accuracies are lower than those of models trained via BP.
Revised: In addition, even if Early Exit is used to isolate the gradient flow, as shown in (Mostafa et al., 2018), the test accuracies are lower than those of models trained via BP.

Original: There are other methods of cutting long gradient flows (Jaderberg et al., 2017; Czarnecki et al., 2017; Löwe et al., 2019; Wu et al., 2022; 𝒙 ! 𝒙 " 𝒙 𝒙 !
Revised: There are other methods of cutting long gradient flows (Jaderberg et al., 2017; Czarnecki et al., 2017; Löwe et al., 2019; Wu et al., 2022; Kao & Chen, 2021). However, most of these methods have been applied only to simple networks, and 𝒙 !

Original: ~𝑇 𝒙 "~𝑇 𝒓 ! 𝒓 "  𝒛 ! 𝒛 "𝑓 𝑔𝑓 𝑔
Revised: 𝒙 𝒙 " 𝒓 ! 𝒓 " 𝑇 𝑇 𝑓 𝑓 𝒛 ! 𝒛 " 𝑔 𝑔

Original: Small distance

Original: Contrastive learning 𝒙! # 𝒙 # 𝒙 !
Revised: Contrastive loss 𝒙 𝒙 # 𝒙 !

Original: # ~𝑇 𝒓 ! # 𝒛 !# 𝑓 𝑔 
Revised: 𝒓 ! 𝑓 𝒛 ! 𝑔 𝑇 𝒙 !

Original: Small distance Small distance Supervised contrastive learning 𝒙 !
Revised: # 𝒓 !

Original: ## 𝒙 ## 𝒙 ! ## ~𝑇 𝒓 ! ## 𝒛 !
Revised: # 𝑓 𝒛 !

Original: ## 𝑓 𝑔 
Revised: # 𝑔 𝑇

Original: Large distance Large distance Kao & Chen, 2021). However, most of these methods have been applied only to simple networks, and their test accuracies are still unsatisfactory; details will be discussed in Section 4.
Revised: Supervised contrastive loss their test accuracies are still unsatisfactory likely because the local objectives may not align with the global objective; details will be discussed in Section 4.

Original: This paper proposes a new methodology, Delog-SCL, which decouples the long gradient flow of a deep neural network by leveraging supervised contrastive learning (SCL).
Revised: This paper proposes a new methodology, Delog-SCL, which decouples the long gradient flow of a deep neural network by leveraging supervised contrastive learning (SCL).

Original: In this design, the forward path transforms an input x into the corresponding prediction
Revised: In this design, the forward path transforms an input x into the corresponding prediction

Original: ˆ y as usual.
Revised: ˆ y as usual.

Original: However, the gradient flow on the backward path is blocked between different components.
Revised: However, the gradient flow on the backward path is blocked between different components.

Original: Instead of using a global objective, we assign a local objective to each component and force each gradient flow to remain within one component.
Revised: Instead of using a global objective, we assign a local objective to each component and force each gradient flow to remain within one component.

Original: We present experiments conducted on multiple open datasets and compare the result with those of models trained via BP and associated learning (AL)
Revised: We present experiments conducted on multiple open datasets and compare the result with those of models trained via BP, Early Exit, and associated learning (AL)

Original: (Wu et al., 2022; Kao & Chen, 2021), a state-of-the-art methodology for BP decoupling that yields results comparable to those of BP.
Revised: (Wu et al., 2022; Kao & Chen, 2021), a state-of-the-art methodology for BP decoupling that yields results comparable to those of BP.

Original: We find that although our method requires fewer parameters, it outperforms AL in terms of test accuracy in most cases.
Revised: We find that Delog-SCL outperforms AL in terms of test accuracy in most cases with fewer parameters.

Original: Additionally, since our method has a more straightforward network architecture than AL, our method could be a favorable alternative to AL.
Revised: Additionally, since our method has a more straightforward network architecture than AL, our method could be a favorable alternative to AL.

Original: The rest of the paper is organized as follows.
Revised: The rest of the paper is organized as follows.

Original: In Section 2, we introduce our proposed model and its properties.
Revised: In Section 2, we introduce Delog-SCL and its properties.

Original: Section 3 presents a comparison of Delog-SCL with AL and BP in terms of their test accuracies and numbers of parameters.
Revised: Section 3 presents a comparison of Delog-SCL with BP, Early Exit, and AL in terms of their test accuracies and numbers of parameters.

Original: We also report the results of experiments focusing on certain properties of Delog-SCL in the same section.
Revised: We also report the results of analyses on certain properties of Delog-SCL in the same section.

Original: Section 4 reviews previous works on BP decoupling and presents a comparison of these works with our model.
Revised: Section 4 reviews previous works on BP decoupling and presents a comparison of these works with our model.

Original: Finally, we conclude our contribution in Section 5.
Revised: We conclude our contribution in Section 5.

Original:  M ETHODOLOGY 2. P RELIMINARIES : CONTRASTIVE LEARNING AND SUPERVISED CONTRASTIVELEARNING
Revised:  M ETHODOLOGY 2. P RELIMINARIES : CONTRASTIVE LEARNING AND SUPERVISED CONTRASTIVE

Revised: LEARNING

Original: Contrastive learning (CL) is a self-supervised technique for learning visual representations of images.
Revised: Contrastive learning (CL) is a self-supervised technique for learning visual representations of images.

Original: Referring to the inner rectangle in Figure 1, given an image x , CL involves generating different views (i.e., x 1 and x 2 in Figure 1) via the same family of data augmentations T .
Revised: Referring to the left of Figure 1, given an image x , CL involves generating different views (i.e., x 1 and x 2 in Figure 1) via the same family of data augmentations T .

Original: The generated views ( x 1 and x 2 ) are further transformed via an encoder function f and a projection head g to minimize the contrastive loss between the output vectors (i.e., z 1 and z 2 ).
Revised: The generated views ( xand x 2 ) are further transformed via an encoder function f and a projection head g to minimize the contrastive loss between the output vectors (i.e., z 1 and z 2 ).

Original: After training, the projection head g is disregarded, and only the encoder f is used to generate the visual representations of images (Chen et al., 2020).
Revised: After training, the projection head g is disregarded, and only the encoder f is used to generate the visual representations of images (Chen et al., 2020).

Original: In other words, given an anchor image x , CL relies on regarding its augmented images as positive instances and all other images as negative instances and considering that positive pairs should be close after encoding and projection.
Revised: In other words, given an anchor image x , CL relies on regarding its augmented images as positive instances and all other images as negative instances and considering that positive pairs should be close after encoding and projection.

Original: SCL refers to the extension of CL from a self-supervised setting to a fully supervised setting.
Revised: SCL refers to the extension of CL from a self-supervised setting to a fully supervised setting.

Original: Therefore, the training data for SCL consist of not only the training images themselves but also the classes of those images.
Revised: Therefore, the training data for SCL consist of not only the training images themselves but also the classes of those images.

Original: Referring to  Figure 1, given an anchor image x of class c , the positive instances 
Revised: Referring to the right of Figure 1, given an anchor image x of class c , the positive instances include the other images of class c in the same batch, whereas all other images in the same batch are regarded as negative instances (Khosla et al., 2020). 𝒙 ! 𝒓 "! 𝒓 #!

Revised: 𝒓 $!

Original:  [Figure]
Revised: #𝑦 ! 𝑓 " 𝑓 # 𝑓 $ 𝑓 % 𝑦 ! ℒ &’( 𝜕ℒ &’( 𝜕𝜃 ) [Figure]

Revised: 𝜕ℒ &’( 𝜕𝜃 ) " 𝜕ℒ &’( 𝜕𝜃 ) # 𝜕ℒ

Revised: [Figure]

Revised: ’( 𝜕𝜃 )

Revised: [Figure]

Revised: [Figure]

Revised: 𝑦

Revised: [Figure]

Revised: ℒ $’() 𝜕ℒ "%

Revised: [Figure]

Revised: 𝜕𝜃 * !

Revised: 𝜕ℒ "%

Revised: [Figure]

Revised: 𝜕𝜃 + !

Revised: 𝜕ℒ "%

Revised: [Figure]

Revised: 𝜕𝜃 + !

Revised: 𝜕ℒ "%

Revised: [Figure]

Revised: 𝜕𝜃 *

Revised: [Figure]

Revised: 𝑓 , 𝒓

Revised: [Figure]

Revised: 𝑓

Revised: [Figure]

Revised: 𝒛 , !

Revised: 𝑔 , 𝒛 ,$ 𝑔 , ℒ ,%

Revised: [Figure]

Revised: 𝜕ℒ ,%

Revised: [Figure]

Revised: 𝜕𝜃 * " 𝜕ℒ ,%

Revised: [Figure]

Revised: 𝜕𝜃

Revised: [Figure]

Revised: " 𝜕ℒ

Revised: [Figure]

Revised: [Figure]

Revised: 𝜕𝜃

Revised: [Figure]

Revised: " 𝜕ℒ

Revised: [Figure]

Revised: [Figure]

Revised: 𝜕𝜃 * "

Revised: [Figure]

Revised: 𝒓

Revised: [Figure]

Revised: 𝑓

Revised: [Figure]

Revised: 𝒛 -

Revised: [Figure]

Revised: 𝑔

Original: [Figure] 
Revised: [Figure] 𝒛 [Figure] 𝑔

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: 𝜕ℒ

Original: [Equation] [Figure]
Revised: [Figure] [Figure] [Figure]

Original: include not only the augmented images but also the other images of class c , whereas all other images in the same batch are regarded as negative instances (Khosla et al., 2020). If we assume that images x and x ′ in Figure 1 belong to the same class, then the distances between z 1 and z ′ 1 and between zand z ′ 1 should both be small. On the other hand, for an image x ′′ from a different class, the distances between z 1 and z ′′ 1 and between z 2 and z ′′ 1 should both be large.
Revised: [Figure]

Original: 2. D ECOUPLING END - TO - END BACKPROPAGATION VIA SUPERVISED CONTRASTIVE LEARNING
Revised: 2. D ECOUPLING END - TO - END BACKPROPAGATION VIA SUPERVISED CONTRASTIVE LEARNING

Original: This section presents Delog-SCL, which leverages the supervised contrastive loss to split one long gradient flow in a deep neural network into multiple shorter ones.
Revised: This section presents Delog-SCL, which leverages the supervised contrastive loss to split one long gradient flow in a deep neural network into multiple shorter ones.

Original: Let us first consider a standard neural network with 3 hidden layers as an example.
Revised: Let us first consider a standard neural network with 3 hidden layers as an example.

Original: As shown in
Revised: As shown in

Original: Figure 2, x ( i ) refers to an input image i , and the function f ℓ ( ℓ = 1 , . .
Revised: Figure 2, x ( i ) refers to an input image i , and the function f ℓ ( ℓ = 1 , . .

Original: . , 4) transforms r ( i ) ℓ − 1 into r ( i ) ℓ (under the assumptions that x ( i )
Revised: . , 4) transforms r ( i ) ℓ − 1 into r ( i ) ℓ (under the assumptions that x ( i )

Original: =
Revised: =

Original: r ( i ) 

Revised: r ( i )

Original: and the predicted class
Revised: and the predicted class

Original: ˆ y ( i ) =
Revised: ˆ y ( i ) =

Original: r ( i ) 4 ).
Revised: r ( i ) 4 ).

Original: Depending on the network architecture, the functions f ℓ could be various well-known neural network layers, such as fully connected layers, convolutional layers, pooling layers, or residual blocks.
Revised: Depending on the network architecture, the functions f ℓ could be various well-known neural network layers, such as fully connected layers, convolutional layers, pooling layers, or residual blocks.

Original: The objective L OUT is determined by the task type.
Revised: The objective L OUT is determined by the task type.

Original: For example, if we are addressing a classification task, we could use the cross-entropy loss between the predicted class ˆ y ( i ) and the ground-truth class y ( i ) as the objective
Revised: For example, if we are addressing a classification task, we could use the cross-entropy loss between the predicted class ˆ y ( i ) and the ground-truth class y ( i ) as the objective

Original: L OUT .
Revised: L OUT .

Original: We use backpropagation to obtain ∂ L OUT /∂θ f ℓ for each layer ℓ , where θ f ℓ represents the parameters of function f ℓ .
Revised: We use backpropagation to obtain ∂ L OUT /∂θ f ℓ for each layer ℓ , where θ f ℓ represents the parameters of function f ℓ .

Original: Once the gradients are obtained, we can use gradient-based optimization strategies, e.g., gradient descent, to update the parameter values.
Revised: Once the gradients are obtained, we can use gradient-based optimization strategies, e.g., gradient descent, to update the parameter values.

Original: Given a neural network with H hidden layers, it can be seen that the longest gradient flow is constructed as a product of H +local gradients.
Revised: Given a neural network with H hidden layers, it can be seen that the longest gradient flow is constructed as a product of H +local gradients.

Original: For example, to obtain ∂ L OUT /θ f 1 in a network with 3 hidden layers (as shown in
Revised: For example, to obtain ∂ L OUT /θ f 1 in a network with 3 hidden layers (as shown in

Original: Figure 2), we need the following:
Revised: Figure 2), we need the following:

Original: [Equation]
Revised: [Equation]

Original: The number of terms of this product grows linearly with the depth of the network.
Revised: The number of terms of this product grows linearly with the depth of the network.

Original: Therefore, as networks become deeper, their long gradient flows cause several optimization issues, as discussed in Section 1.
Revised: Therefore, as networks become deeper, their long gradient flows cause several optimization issues, as discussed in Section 1.

Original: We use Figure 3 to illustrate our strategy of cutting a long gradient flow into several local gradients for a neural network with 3 hidden layers.
Revised: We use Figure 3 to illustrate our strategy of cutting a long gradient flow into several local gradients for a neural network with 3 hidden layers.

Original: Let r ( i ) 

Revised: Let r ( i )

Original: (i.e., x ( i ) ) and r ( j )(i.e., x ( j ) ) be two image views in the same batch ( r ( i ) 0 and r ( j ) 

Revised: (i.e., x ( i ) ) and r ( j )(i.e., x ( j ) ) be two image views in the same batch ( r ( i ) 0 and r ( j )

Original: may or may not be augmented images, i.e., views, from the same image).
Revised: may or may not be augmented images, i.e., views, from the same image).

Original: We use f 1 to transform each of them, obtaining] r ( i ) 1 and r ( j ) 1 , and further use the function g 1 to convert them into z ( i ) 1 and z ( j ) 1 , respectively.
Revised: We use f 1 to transform each of them, obtaining r ( i ) 1 and r ( j ) 1 , and further use the function g 1 to convert them into z ( i ) 1 and z ( j ) 1 , respectively.

Original: The functions f 1 and g 1 can be regarded as the encoding function and the projection head, respectively, in CL (refer to Figure 1).
Revised: The functions f 1 and g 1 can be regarded as the encoding function and the projection head, respectively, in CL (refer to Figure 1).

Original: We repeat the same process for each hidden layer ℓ to form the corresponding component ℓ .
Revised: We repeat the same process for each hidden layer ℓ to form the corresponding component ℓ .

Original: If x ( i ) and x ( j ) are two different views of the same image or if y ( i ) (the class of x ( i ) ) is equal to y ( j ) (the class of x ( j ) ), then we should ensure that z ( i ) ℓ is close to z ( j ) ℓ for all ℓ .
Revised: If x ( i ) and x ( j ) are two different views of the same image or if y ( i ) (the class of x ( i ) ) is equal to y ( j ) (the class of x ( j ) ), then we should ensure that z ( i ) ℓ is close to z ( j ) ℓ for all ℓ .

Original: Otherwise, we should increase the distance between z ( i ) ℓ and z ( j ) ℓ .
Revised: Otherwise, we should increase the distance between z ( i ) ℓ and z ( j ) ℓ .

Original: In the last layer, we compute the distance between ˆ y ( i ) and y ( i ) as the loss L OUTi .
Revised: In the last layer, we compute the distance between ˆ y ( i ) and y ( i ) as the loss L OUTi .

Original: Eventually, we define the local supervised contrastive loss L SCℓ for batch B in layer ℓ as shown in Equation 2:
Revised: Eventually, we define the local supervised contrastive loss L SCℓ for batch B in layer ℓ as shown in Equation 2:

Original: [Equation]
Revised: [Equation]

Original: where B = 1 , 2 , . .
Revised: where B = 1 , 2 , . .

Original: . , N represents a batch of multiview images, P ( i ) is the set of all positive samples for an image i , τ is a hyperparameter, and I ( j ̸ = i ) ∈ { 0 , 1 } is an indicator function that returns 

Revised: . , N represents a batch of multiview images, P ( i ) is the set of all positive samples for an image i , τ is a hyperparameter, and I ( j ̸ = i ) ∈ { 0 , 1 } is an indicator function that returns

Original: if j ̸ = i and 0 otherwise.
Revised: if j ̸ = i and 0 otherwise.

Original: Ultimately, the global objective function is an accumulation of the local supervised contrastive losses and the losses in the output layer, as defined below:
Revised: Ultimately, the global objective function is an accumulation of the local supervised contrastive losses and the losses in the output layer, as defined below:

Original: [Equation]
Revised: [Equation]

Original: where H is the number of hidden layers and L OUTi is the i th loss in the output layer (refer to
Revised: where H is the number of hidden layers and L OUTi is the i th loss in the output layer (refer to

Original: Figure 3).
Revised: Figure 3).

Revised: The computation of L SCℓ and the pseudo code of Delog-SCL for a 3-layer vanilla ConvNet is given in Algorithm 1 and Algorithm 2 in Appendix A.5.

Original: 2. F ORWARD PATH , BACKWARD PATH , AND INFERENCE FUNCTION
Revised: 2. F ORWARD PATH , BACKWARD PATH , AND INFERENCE FUNCTION

Original: For a regular neural network (e.g., Figure 2), the forward path and the inference function are identical, and the backward path is simply obtained by inverting the direction of the forward path.
Revised: For a regular neural network (e.g., Figure 2), the forward path and the inference function are identical, and the backward path is simply obtained by inverting the direction of the forward path.

Original: However, the situation is more complicated in our case because we divide the global objective into several local ones.
Revised: However, the situation is more complicated in our case because we divide the global objective into several local ones.

Original: Consequently, we have multiple short forward paths, multiple short backward paths, and one inference path.
Revised: Consequently, we have multiple short forward paths, multiple short backward paths, and one inference path.

Original: Thus, the inference path and the forward paths are no longer identical in DelogSCL.
Revised: Thus, the inference path and the forward paths are no longer identical in DelogSCL.

Original: During training, each component ℓ has its own forward and backward paths.
Revised: During training, each component ℓ has its own forward and backward paths.

Original: Taking Figure 3 as an example, the forward path of component ℓ transforms each r ( i ) ℓ − 1 into r ( i ) ℓ via the local encoding function f ℓ and further transforms each r ( i ) ℓ into z ( i ) ℓ via the local projection head g ℓ .
Revised: Taking Figure 3 as an example, the forward path of component ℓ transforms each r ( i ) ℓ − 1 into r ( i ) ℓ via the local encoding function f ℓ and further transforms each r ( i ) ℓ into z ( i ) ℓ via the local projection head g ℓ .

Original: On the backward path, each hidden layer computes ∂ L SCℓ /∂θ g ℓ and ∂ L SCℓ /∂θ f ℓ based on the chain rule and updates the parameters by means of gradient-based optimization strategies.
Revised: On the backward path, each hidden layer computes ∂ L SC ℓ /∂θ g ℓ and ∂ L SC ℓ /∂θ f ℓ based on the chain rule and updates the parameters by means of gradient-based optimization strategies.

Original: We block the gradient flow between each component.
Revised: We block the gradient flow between each component.

Original: 1 As a result, each gradient flow remains within one component and is therefore short.
Revised: 1 As a result, each gradient flow remains within one component and is therefore short.

Original: Equation 4 and Equation 5 show these local gradient flows.
Revised: Equation 4 and Equation 5 show these local gradient flows.

Original: [Equation]
Revised: [Equation]

Original: Eventually, even if we construct a deep neural network, the cost of computing each ∂ L SC /∂θ f ℓ and each ∂ L SC /∂θ g ℓ remains constant.
Revised: Eventually, even if we construct a deep neural network, the cost of computing each ∂ L SC /∂θ f ℓ and each ∂ L SC /∂θ g ℓ remains constant.

Original: Additionally, the gradient flow in the output layer is also short: we simply compute ∂ L outk /∂θ f H +1 (where H is the number of hidden layers).
Revised: Additionally, the gradient flow in the output layer is also short: we simply compute ∂ L outk /∂θ f H +1 (where H is the number of hidden layers).

Original: This design alleviates various issues caused by long gradient flows.
Revised: This design alleviates various issues caused by long gradient flows.

Original: In the inference (prediction) phase, we need the functions f ℓ but notthe functions g ℓ , as shown by Equation 6:
Revised: In the inference (prediction) phase, we need the functions f ℓ but not g ℓ , as shown by Equation 6:

Original: [Equation]
Revised: [Equation]

Original: where ◦ is the function composition operator ( H = 3 for the example illustrated in Figure 3).
Revised: where ◦ is the function composition operator ( H = 3 for the example illustrated in Figure 3).

Original: Although our proposed method (e.g., Figure 3) involves more parameters than a standard neural network structure (e.g., Figure 2) during training, they have the same number of parameters during inference because both of them use only the functions f ℓ .
Revised: Although our proposed method (e.g., Figure 3) involves more parameters than a standard neural network structure (e.g., Figure 2) during training, they have the same number of parameters during inference because both of them use only the functions f ℓ .

Original: Therefore, they have the same hypothesis space.
Revised: Therefore, they have the same hypothesis space.

Original: The parameters that participate in the inference phase (denoted by θ f ℓ ) are called the effective parameters , and the parameters used during training but not during inference (denoted by θ g ℓ ) are called the affiliated parameters .
Revised: The parameters that participate in the inference phase (denoted by θ f ℓ ) are called the effective parameters , and the parameters used during training but not during inference (denoted by θ g ℓ ) are called the affiliated parameters .

Original: 2. P ROPERTIES
Revised: 2. P ROPERTIES

Original: [Table]
Revised: [Table]

Original: B

Original: Task Task Task... In this section, we discuss three properties of our proposed model — short gradient flows, a flexible structure, and the ability to perform parallel (pipelined) training.
Revised: In this section, we discuss three properties of our proposed model — short gradient flows, a flexible structure, and the ability to perform parallel (pipelined) training.

Original: As discussed in Section 2.3, training a regular neural network with BP requires a gradient flow of length
Revised: As discussed in Section 2.3, training a regular neural network with BP requires a gradient flow of length

Original: O ( H ) .
Revised: O ( H ) .

Original: In contrast, the length of each gradient flow in our model is independent of the number of layers; the length is always a constant.
Revised: In contrast, the length of each gradient flow in our model is independent of the number of layers; the length is always a constant.

Original: Therefore, the various optimization issues resulting from long gradient flows, as discussed in Section 1, are eliminated (or at least alleviated).
Revised: Therefore, the various optimization issues resulting from long gradient flows, as discussed in Section 1, are eliminated (or at least alleviated).

Original: The network structure is more flexible and perhaps easier to understand than that of associated learning (AL), a state-of-the-art methodology for decoupling BP in terms of test accuracy (Kao &
Revised: The network structure is more flexible and perhaps easier to understand than that of associated learning (AL), a state-of-the-art methodology for decoupling BP in terms of test accuracy (Kao &

Original: Chen, 2021; Wu et al., 2022).
Revised: Chen, 2021; Wu et al., 2022).

Original: Specifically, AL involves projecting the features x and the target y into the same latent space for each layer ℓ .
Revised: Specifically, AL involves projecting the features x and the target y into the same latent space for each layer ℓ .

Original: Although this design yields excellent test accuracies that are comparable to those of BP-trained models (Wu et al., 2022), it has at least two unconventional and perhaps mysterious characteristics.
Revised: Although this design yields excellent test accuracies that are comparable to those of BP-trained models (Wu et al., 2022), it has at least two unconventional and perhaps mysterious characteristics.

Original: First, AL involves projecting a one-hot-encoded target variable y into a latent vector t 1 and then transforming t 1 back into y .
Revised: First, AL involves projecting a one-hot-encoded target variable y into a latent vector t 1 and then transforming t 1 back into y .

Original: Interestingly, the length of t 1 is sometimes greater than the number of classes.
Revised: Interestingly, the length of t 1 is sometimes greater than the number of classes.

Original: This process corresponds to building an autoencoder whose bottleneck layer is larger than the input and output layers.
Revised: This process corresponds to building an autoencoder whose bottleneck layer is larger than the input and output layers.

Original: Although this unconventional approach works surprisingly well in practice (Wu et al., 2022; Kao & Chen, 2021), the fundamental reasons for this are still unclear.
Revised: Although this unconventional approach works surprisingly well in practice (Wu et al., 2022; Kao & Chen, 2021), the fundamental reasons for this are still unclear.

Original: Second, when converting a neural network into its AL form, we sometimes need to create extra fully connected layers.
Revised: Second, when converting a neural network into its AL form, we sometimes need to create extra fully connected layers.

Original: In contrast, our design is more natural because we need neither the autoencoder nor the extra fully connected layers.
Revised: In contrast, our design is more natural because we need neither the autoencoder nor the extra fully connected layers.

Original: Finally, since each component has its own local objective, we can parallelize the training procedure by means of pipelining.
Revised: Finally, since each component has its own local objective, we can parallelize the training procedure by means of pipelining.

Original: We use the network illustrated in Figure 3 as an example.
Revised: We use the network illustrated in Figure 3 as an example.

Original: Let Task ℓ denote the entire forward and backward process in layer ℓ ; then, we can illustrate the pipelining process as shown in Table 1.
Revised: Let Task ℓ denote the entire forward and backward process in layer ℓ ; then, we can illustrate the pipelining process as shown in Table 1.

Original: Specifically, in the first time unit t 1 , component 1 uses the first batch ( B 1 ) to perform Task 1.
Revised: Specifically, in the first time unit t 1 , component 1 uses the first batch ( B 1 ) to perform Task 1.

Original: At t 2 , component 2 performs Task 2 based on B 1 , and component 1 continues to performing Task 1 based on the second batch ( B 2 ).
Revised: At t 2 , component 2 performs Task 2 based on B 1 , and component 1 continues to performing Task 1 based on the second batch ( B 2 ).

Original: Starting at t 3 , all three components can perform forward propagation, backward propagation, and parameter updating simultaneously.
Revised: Starting at t 3 , all three components can perform forward propagation, backward propagation, and parameter updating simultaneously.

Original: However, we have shown here only that parallelization by means of pipelining is feasible; implementation of the pipeline mechanism is left for future work.
Revised: However, we have shown here only that parallelization by means of pipelining is feasible; implementation of the pipeline mechanism is left for future work.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We compare our proposed model with three baselines using different neural networks on different datasets.
Revised: We compare Delog-SCL with three baselines using different neural networks on different datasets.

Original: The baseline models include BP, the Early Exit mechanism introduced in Section 1, and AL, a state-of-the-art method for BP decoupling in terms of the test accuracy.
Revised: The baseline models include BP, the Early Exit mechanism introduced in Section 1, and AL, a stateof-the-art method for BP decoupling in terms of the test accuracy.

Original: We test three neural networks: a vanilla convolutional neural network (vanilla ConvNet), the VGG network (Simonyan & Zisserman, 2014), and the residual network (ResNet) (He et al., 2016).
Revised: We test three neural networks: a vanilla convolutional neural network (vanilla ConvNet), the VGG network (Simonyan & Zisserman, 2014), and the residual network (ResNet) (He et al., 2016).

Original: The experimental datasets include CIFAR-10 (consists of 50 , 000 color training images and 10 , 000 test images; each image belongs to 1 of 10 classes), CIFAR-100 (consists of 50 , 000 color training images and 10 , 000 test images; each image belongs to 1 of 100 classes), and Tiny-ImageNet (consists of 100 , 000 color training images, 10 , 000 validation images, and 10 , 000 test images; each image belongs to 1 ofclasses).
Revised: The experimental datasets include CIFAR-10 (consists of 50 , 000 color training images and 10 , 000 test images; each image belongs to 1 of 10 classes), CIFAR-100 (consists of 50 , 000 color training images and 10 , 000 test images; each image belongs to 1 of 100 classes), and Tiny-ImageNet (consists of 100 , 000 color training images, 10 , 000 validation images, and 10 , 000 test images; each image belongs to 1 of 200 classes).

Original: 3. A CCURACY COMPARISON
Revised: 3. A CCURACY COMPARISON

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: Table 2 shows the test accuracies of the various methods on the CIFAR-10 dataset.
Revised: Table 2 shows the test accuracies of the various methods on the CIFAR-10 dataset.

Original: The simple
Revised: The simple

Original: Early Exit mechanism can be used to learn the relationship between an image and its corresponding class.
Revised: Early Exit mechanism can be used to learn the relationship between an image and its corresponding class.

Original: However, the test accuracies of Early Exit are much worse than those of BP.
Revised: However, the test accuracies of Early Exit are much worse than those of BP.

Original: Both AL and our[Table] proposed Delog-SCL yield better test accuracies than BP based on the Vanilla ConvNet and VGG architectures.
Revised: Both AL and our proposed Delog-SCL yield better test accuracies than BP based on the Vanilla ConvNet and VGG architectures.

Original: However, when ResNet is used, BP yields the highest test accuracy.
Revised: However, when ResNet is used, BP yields the highest test accuracy.

Original: If we compare only the methods that involve BP decomposition, Delog-SCL performs the best among them.
Revised: If we compare only the methods that involve BP decomposition, Delog-SCL performs the best among them.

Original: We also tested BP,  AL, and Delog-SCL on CIFAR-100.
Revised: We also tested BP, Early Exit, AL, and Delog-SCL on CIFAR-100.

Original: We omitted Early Exit because this method performs much worse on more complicated datasets.

Original: The results, as shown in Table 3, are similar to those on CIFAR-10: Delog-SCL performs better than both AL and BP based on Vanilla ConvNet and VGG, whereas Delog-SCL performs worse than BP when ResNet is used.
Revised: The results, as shown in Table 3, are similar to those on CIFAR-10: Delog-SCL performs better than both AL and BP based on Vanilla ConvNet and VGG, whereas Delog-SCL performs worse than BP when ResNet is used.

Original: These results are also consistent with those reported in (Kao & Chen, 2021; Wu et al., 2022), which indicate that BP usually performs best for ResNet.
Revised: These results are also consistent with those reported in (Kao & Chen, 2021; Wu et al., 2022).

Original: This is probably because the residual connections assist in the transmission of the gradients.

Original: Table 4 gives the results obtained on Tiny-ImageNet.
Revised: Table 4 gives the results obtained on Tiny-ImageNet.

Original: When VGG is used, both AL and Delog-SCL outperform BP.
Revised: When VGG is used, both AL and Delog-SCL outperform BP.

Original: However, for ResNet, BP performs much better.
Revised: However, for ResNet, BP performs much better.

Revised: Delog-SCL is stable in training, as can be shown by Figure 5 in the Appendix.

Revised: 3.1. D ISCUSSION ON ACCURACY COMPARISON

Original: When BP is used, allof the parameters are updated to minimize  the distance between the prediction ˆ y and the target y .
Revised: When BP is used, all parameters are updated to minimize a global objective – the residual between the prediction ˆ y and the target y .

Revised: On the other hand, methods to decouple end-to-end backpropagation, such as Delog-SCL and AL, are composed of many local objectives, which may differ from the global objective.

Original: Therefore, it is surprising that Delog-SCL and AL outperform BP for some network structures.
Revised: Therefore, it is surprising that Delog-SCL and AL outperform BP for some network structures.

Original: The authors of AL proposed several conjectures to explain this remarkable result.
Revised: The authors of AL proposed several conjectures to explain this remarkable result.

Original: First, projecting the feature vector x and the target y into the same latent space may be helpful.
Revised: First, projecting the feature vector x and the target y into the same latent space may be helpful.

Original: Second, the autoencoder may implicitly perform somekind of feature extraction and regularization.
Revised: Second, the autoencoder may implicitly perform some feature extraction and regularization.

Original: Third, overparameterization may be helpful for optimization Arora et al. ; Chen & Chen (2020).
Revised: Third, overparameterization may be helpful for optimization (Arora et al., 2018; Chen & Chen, 2020).

Original: However,although the first and second conjectures do not apply for Delog-SCL, this method still yields better accuracies than BP and AL.
Revised: However, the first and second conjectures only apply to AL but not to Delog-SCL, but Delog-SCL still yields better accuracies than BP and AL in vanilla ConvNet and VGG.

Original: Therefore, the above conjectures may not fully explain the success of Delog-SCL.
Revised: Therefore, the above conjectures may not fully explain the success of Delog-SCL.

Original: Further investigation will be needed to uncover the fundamental reasons.
Revised: Further investigation will be needed to uncover the fundamental reasons.

Original: 3. N UMBER OF EFFECTIVE PARAMETERS
Revised: As for ResNet, its authors state that the main effect of a residual is not about promoting gradient flows (He et al., 2016). Instead, ResNet performs better than vanilla ConvNet because the latent representations H ℓ and H ℓ +1 at deep neighboring layers ℓ and ℓ + 1 are likely similar. Regular nonlinear transformations may be difficult to approximate an (almost) identical mapping from H ℓ to H ℓ +1 . However, the residual connection sets H ℓ +1 to be f ( H ℓ ) + H ℓ .

Revised: Even if f () is a nonlinear function, a solver is easier to make H ℓ +1 ≈ H ℓ by making f ( H ℓ ) ≈ 0 .

Revised: The property that H ℓ +1 ≈

Revised: H ℓ is likely true when a network is deep.

Revised: However, when using Delog-SCL or AL, each local network is short, so Delog-SCL and AL are unlikely to take advantage of the residual connections.

Revised: As a result, optimizing a ResNet by BP usually gives better results than by BP-decoupling methods, such as Delog-SCL and AL.

Revised: 3. N UMBER OF EFFECTIVE PARAMETERS

Original: [Table]
Revised: [Table]

Original: Table 5 shows the numbers of parameters required during training and inference for VGG and
Revised: Table 5 shows the numbers of parameters required during training and inference for VGG and

Original: ResNet (using CIFAR-10 as an example).
Revised: ResNet (using CIFAR-10 as an example).

Original: For BP, the training and prediction (testing) stages involve the same set of parameters.
Revised: For BP, the training and testing stages involve the same set of parameters.

Original: In contrast, the training process of AL requires additional bridge functions and encoding functions, which are not used  during testing.
Revised: In contrast, the training process of AL requires additional bridge functions and encoding functions, which are not used [Figure] during testing.

Original: As a result, the number of training parameters is much larger than the number of parameters required for BP.
Revised: So, the number of training parameters is much larger than the number of parameters required for BP.

Original: During testing, the extra fully connected layers in AL also require more parameters than are needed in BP.
Revised: During testing, the extra fully connected layers in AL also require more parameters than are needed in BP.

Original: Finally, Delog-SCL and BP have the same number of testing parameters.
Revised: Finally, Delog-SCL and BP have the same number of testing parameters.

Original: However, during training, Delog-SCL needs the projection heads g ℓ for each component; as a result, the number of required parameters during training is more than for BP but much fewer than for AL. 
Revised: However, during training, Delog-SCL needs the projection heads g ℓ for each component, so the number of required parameters during training is more than for BP but much fewer than for AL. The difference of the parameter counts may also be reflected on the training and inference speed.

Revised: We show the practical training and testing time of different methods in Table 7 in the Appendix.

Original: 3. A LARGER BATCH SIZE IMPROVES THE TRAINING QUALITY
Revised: 3. M ULTIPLE SHORT GRADIENT FLOWS ACCELERATE LEARNING

Original: [Table]
Revised: This section shows that dividing a long gradient flow into multiple short ones accelerates learning. Referring to Figure 3, our proposed Delog-SCL uses a local supervised contrastive loss to create a short local gradient flow L SCℓ .

Original: We also tested how the batch size influences the test accuracy.
Revised: We compare the standard Delog-SCL with a modification where only a single long gradient flow is used. In the compared baseline, we enable the global objective L OUT to pass through the entire network and remove all local supervised contrastive losses L SCℓ .

Original: As shown in Table 6, performing
Revised: The results are shown in Figure 4.

Original:  Delog-SCL training using a large batch size is helpful for both VGG andResNet, although of the two, the improvement on VGG is more evident.
Revised: We label the original Delog-SCL as “multiple short gradient flows” and the modification with single long gradient flow as “single gradient flow”.

Original: This finding is consistent with the results reported in previous studies, e.g., Chen et al.

Original: ; Henaff (2020); Bachman et al. (2019), in which the authors noted that because a larger batch tends to include more negative pairs (as shown inEquation 2), the model has access to more information that can be used to distinguish positive pairs from negative pairs. Although other studies, e.g., Mitrovic et al. (2020), have shown that the number of negative pairs may not be critical to the improvement of the test accuracy, most studies tend to agree that a larger batch size leads to better results.
Revised: Using multiple short gradient flows accelerates the learning speed, especially in the first 100 epochs.

Original: 3. A NONLINEAR PROJECTION HEADBENEFITS THE REPRESENTATION QUALITY OF THE LAYER BEFORE IT
Revised: 3. T HE EFFECT OF BATCH SIZE AND PROJECTION HEAD

Original: [Table] This section presents the influence ofdifferent projection heads.
Revised: We also experimented with how the batch size and the type of projection head influence learning.

Original: Table 7 compares the accuracies of VGG and ResNet on CIFAR-10 when 3 different types of projection heads are used: identity mapping (i.e., g ℓ (cid:16) r ( i ) ℓ (cid:17) = r ( i ) ℓ ), linear mapping (i.e., g ℓ (cid:16) r ( i ) ℓ (cid:17) = w T r ( i ) ℓ + w 0 ), and the default mapping based on a multilayer perceptron (MLP). The  results are similar to those reported in (Chen et al., 2020): the MLP mapping shows a 2 .
Revised: The experimental results show that a larger batch size improves the learning quality, which is consistent with previous studies (Chen et al., 2020; Henaff, 2020; Bachman et al., 2019).

Original: 6% improvement over linear projection, which outperforms identity projection by over 10% .
Revised: As for the projection head, using a nonlinear function benefits the representation quality of layers before it.

Original: The reason why using an MLP as the projection head is beneficial is probably that given in (Chen et al., 2020): the information loss induced by the contrastive loss is more severe when a simple projection head is used.
Revised: The result also matches the experiments conducted in Chen et al.

Original: In particular, since a projection head g ℓ (refer to Figure 3 and Figure 1) maximizes the agreement between augmented images, g ℓ may remove information relevant to image rotation, flipping, andother data augmentation operations, which could be useful for downstream tasks. When a simple projection head g ℓ is used, the information contained in r ( i ) ℓ will be similar to that in z ( i ) ℓ , which may mean that r ( i ) ℓ is invariant to data transformation. On the other hand, when a complex projection head such as an MLP is used, the information in r ( i ) ℓ may be very different from that in z ( i ) ℓ . As a result, even if z ( i ) ℓ loses information relevant to data transformation, r ( i ) ℓ may still preserve this information.
Revised: The experimental details of batch size and projection heads are presented in Section A.3 and Section A.4 in Appendix.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Studies on alternatives to BP mostly aim to address optimization and performance issues, such as gradient vanishing/explosion and training costs.
Revised: Studies on alternatives to BP mostly aim to address optimization and performance issues, such as gradient vanishing/explosion and training costs.

Original: We review some of these works that have particularly focused on the creation of local objectives and local gradient flows.
Revised: We review some of these works that have particularly focused on the creation of local objectives and local gradient flows.

Original:  The first type of BP alternative is target propagation (Lee et al., 2015; Meulemans et al., 2020;
Revised: [Table] The first type of BP alternative is target propagation (Lee et al., 2015; Meulemans et al. [Table]

Original:  Manchev & Spratling, 2020; Bengio, 2014), which assigns a local target for each layer via feedback[Table] (inverse) mapping.
Revised: [Table] Manchev & Spratling, 2020; Bengio, 2014), which assigns a local target for each layer via feedback (inverse) mapping.

Original: Such a methodology can alleviate the problem of vanishing/exploding gradients since each gradient flow is short.
Revised: Such a methodology can alleviate the problem of vanishing/exploding gradients since each gradient flow is short.

Original: However, the parameters are still updated in a layerwise fashion, so it could be challenging to learn the parameters in different layers simultaneously.
Revised: However, the parameters are still updated in a layerwise fashion, so it could be challenging to learn the parameters in different layers simultaneously.

Original: Methods of the second type model BP as a constrained optimization problem, in which the output of one layer is forced to equal the input to the next layer (Gotmare et al., 2018; Marra et al., 2020).
Revised: Methods of the second type model BP as a constrained optimization problem, in which the output of one layer is forced to equal the input to the next layer (Gotmare et al., 2018; Marra et al., 2020).

Original: Such a design shortens the gradient flows and enables parallel parameter updates.
Revised: Such a design shortens the gradient flows and enables parallel parameter updates.

Original: However, the experimental results show that the test accuracies are lower than that of standard BP.
Revised: However, the experimental results show that the test accuracies are lower than that of standard BP.

Original: Methods of the third type determine the local objectives through transformations of the target.
Revised: Methods of the third type determine the local objectives through transformations of the target.

Original: A representative method of this type is AL Wu et al.
Revised: A representative method of this type is AL Wu et al.

Original: ; Kao & Chen (2021), which transforms both the feature vector x and thecorresponding target y into the same set of latent spaces.
Revised: ; Kao & Chen (2021), which transforms both the feature vector x and the target y into the same set of latent spaces.

Original: To the best of our knowledge, AL is the only existing method that can achieve BP decoupling for a wide range of network architectures and yield test accuracies that are comparable to those obtained with BP.
Revised: To the best of our knowledge, AL is the only existing method that can achieve BP decoupling for a wide range of network architectures and yield test accuracies that are comparable to those obtained with BP.

Original: Our proposed Delog-SCL is motivated by both AL and Greedy InfoMax (GIM) (Löwe et al., 2019), which uses the contrastive loss as each local objective.
Revised: Our proposed Delog-SCL is motivated by both AL and Greedy InfoMax (GIM) (Löwe et al., 2019), which uses the contrastive loss as each local objective.

Original: However, GIM targets self-supervised learning tasks, whereas our Delog-SCL can handle supervised learning tasks because Delog-SCL uses the supervised contrastive loss in the hidden layers and the distance between the predicted and observed targets in the output layer.
Revised: However, GIM targets self-supervised learning tasks, whereas our Delog-SCL can handle supervised learning tasks because Delog-SCL uses the supervised contrastive loss in the hidden layers and the distance between the predicted and observed targets in the output layer.

Original: Since only Delog-SCL and AL yield test accuracies comparable to those of BP, in Table 8, we further compare the properties of these three methods.
Revised: Since only Delog-SCL and AL yield test accuracies comparable to those of BP, in Table 6, we further compare the properties of these three methods.

Original: First, BP requires no affiliated parameters because all parameters collaborate to reduce the global loss.
Revised: First, BP requires no affiliated parameters because all parameters collaborate to reduce the global loss.

Original: BP can be applied to almost all kinds of neural networks.
Revised: BP can be applied to almost all kinds of neural networks.

Original: However, its gradient flow is long (especially when the network is deep), and it is challenging to achieve pipelined training with this method.
Revised: However, its gradient flow is long (especially when the network is deep), and it is challenging to achieve pipelined training with this method.

Original: AL requires transforming both the feature vector x and the target y alongside each other.
Revised: AL requires transforming both the feature vector x and the target y alongside each other.

Original: As a result, AL usually requires additional fully connected layers, resulting in a large number of affiliated parameters and less structural flexibility.
Revised: As a result, AL usually requires additional fully connected layers, resulting in a large number of affiliated parameters and less structural flexibility.

Original: However, each gradient flow in AL is short, and parameters in different layers can be updated simultaneously via pipelined training.
Revised: However, each gradient flow in AL is short, and parameters in different layers can be updated simultaneously via pipelined training.

Original: Finally, because Delog-SCL requires computing the supervised contrastive loss in each hidden layer, this method also needs additional affiliated parameters (although much fewer than AL) during training.
Revised: Finally, because Delog-SCL requires computing the supervised contrastive loss in each hidden layer, this method also needs additional affiliated parameters (although much fewer than AL) during training.

Original: The introduction of the supervised contrastive loss also adds complexity in the network design.
Revised: The introduction of the supervised contrastive loss also adds complexity in the network design.

Original: The advantages of Delog-SCL are similar to those of
Revised: The advantages of Delog-SCL are similar to those of

Original: AL: the gradient flows are short, and parallel parameter updating is possible (via pipelining).
Revised: AL: the gradient flows are short, and parallel parameter updating is possible (via pipelining).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper presents Delog-SCL, a new methodology for decoupling the components of the BP process in a neural network.
Revised: This paper presents Delog-SCL, a new methodology for decoupling the components of the BP process in a neural network.

Original: Delog-SCL may address various optimization issues (e.g., vanishing/exploding gradients and unstable gradients in the early layers) resulting from the long gradient flows in deep neural networks.
Revised: Delog-SCL may address various optimization issues (e.g., vanishing/exploding gradients and unstable gradients in the early layers) resulting from the long gradient flows in deep neural networks.

Original: We report experiments conducted to show that Delog-SCL’s predictive power is comparable to (and frequently better than) that of either BP or AL, which is a state-of-the-art alternative to BP.
Revised: We report experiments conducted to show that Delog-SCL’s predictive power is comparable to (and frequently better than) that of either BP or AL, which is a state-of-the-art alternative to BP.

Original: Delog-SCL is more flexible than AL because Delog-SCL does not require additional fully connected layers, whereas AL usually does.
Revised: Delog-SCL is more flexible than AL because Delog-SCL does not require additional fully connected layers, whereas AL usually does.

Original: Therefore, Delog-SCL is a natural substitute for
Revised: Therefore, Delog-SCL is a natural substitute for

Original: AL and could be a promising alternative to BP.
Revised: AL and could be a promising alternative to BP.

Revised: A A PPENDIX A. T EST ACCURACY VS EPOCH

Revised: Figure 5 compares Delog-SCL and BP on vanilla CNN in terms of their dynamics of training accuracy when the epoch increases.

Revised: First, the test accuracy of Delog-SCL improves stably.

Revised: Second, the Delog-SCL outperforms BP after approximately 100 epochs.

Revised: BP is better than Delog-SCL at the beginning, likely because all the parameters in BP are updated to minimize a global objective.

Revised: On the contrary, most of the parameters in Delog-SCL are updated to fit local objective functions, which usually have no direct access to the target variable.

Revised: Different hyperparameter settings may lead to slightly different curves.

Revised: However, most of them follow a similar pattern.

Revised: Experiments on other datasets (CIFAR-10 and tiny-ImageNet) for the VGG network structure also show similar trends.

Revised: A. A COMPARISON OF TRAINING AND TEST TIME OF BP, AL, AND D ELOG -SCL

Revised: Table 7 compares Delog-SCL, BP, and AL in terms of their practical training and testing seconds per epoch on ConvNet, VGG, and ResNet using the CIFAR-10 dataset.

Revised: The empirical training and testing speed of Delog-SCL and BP are extremely close.

Revised: However, AL is apparently slower than the other two.

Revised: The experiments are tested on the Container Compute Service with NVIDIA Tesla V100 GPU.

Revised: [Figure]

Revised: [Table]

Revised: A. A LARGER BATCH SIZE IMPROVES THE LEARNING QUALITY

Revised: We also tested how the batch size influences the test accuracy.

Revised: As shown in Table 8, performing

Revised: Delog-SCL training using a large batch size is helpful

Revised: [Table]

Revised: and the improvement on VGG is more evident than in other networks.

Revised: This finding is consistent with the results reported in previous studies, e.g., Chen et al.

Revised: ; Henaff (2020); Bachman et al.

Revised: (2019), in which the authors noted that because a larger batch tends to include more negative pairs (as shown in Equation 2), the model has access to more information that can be used to distinguish positive pairs from negative pairs.

Revised: Although other studies, e.g., Mitrovic et al.

Revised: [Table]

Revised: have shown that the number of negative pairs may not be critical to the improvement of the test accuracy, most studies tend to agree that a larger batch size leads to better results.

Revised: Table

Revised: [Table]

Revised: [Table]

Revised: A. A NONLINEAR PROJECTION HEAD BENEFITS THE REPRESENTATION QUALITY OF THE LAYER BEFORE IT

Revised: This section presents the influence of different projection heads.

Revised: Table 9 compares the accuracies of VGG and ResNet on CIFAR-10 when 3 different types of projection heads are used: identity mapping (i.e., g ℓ (cid:16) r ( i ) ℓ (cid:17)

Revised: =

Revised: r ( i ) ℓ ), linear mapping (i.e., g ℓ (cid:16) r ( i ) ℓ (cid:17) =

Revised: w T r ( i ) ℓ + w 0 ), and the default mapping based on a multilayer perceptron (MLP).

Revised: The results are similar to those reported in (Chen et al., 2020): the MLP mapping shows a 2 .

Revised: 6% improvement over linear projection, which outperforms identity projection by over 10% .

Revised: Using an MLP as the projection head is beneficial likely because the information loss induced by the contrastive loss is more severe when a simple projection head is used (Chen et al., 2020).

Revised: In particular, since a projection head g ℓ (refer to Figure 3 and Figure 1) maximizes the agreement between augmented images, g ℓ may remove information relevant to image rotation, flipping, and other data augmentation operations, which could be useful for downstream tasks.

Revised: When a simple projection head g ℓ is used, the information contained in r ( i ) ℓ will be similar to that in z ( i ) ℓ , which means that r ( i ) ℓ is invariant to data augmentation.

Revised: On the other hand, when a complex projection head such as an MLP is used, the information in r ( i ) ℓ may be very different from that in z ( i ) ℓ .

Revised: As a result, even if z ( i ) ℓ loses information relevant to data augmentation, r ( i ) ℓ may still preserve this information.

Revised: A. P SEUDO CODE

Revised: Here we provide a PyTorch pseudocode for the creation of the local supervised contrastive losses (Algorithm 1) and Delog-SCL (Algorithm 2).

Revised: import torch

Revised: import torch.nn as nn class SupConLoss(nn.

Revised: Module): def __init__(self, dim): super.__init__() self.linear=nn.

Revised: Sequential(nn.

Revised: Linear(dim, 512), nn.ReLU(), nn.Linear(512, 1024)) self.temperature =0.

Revised: def forward(self, x, label): x = self.linear(x)

Revised: x = nn.functional.normalize(x) label =

Revised: label.view(-1, 1) bsz =label.shape[0] mask =torch.eq(label, label.

Revised: T).float() anchor_mask =torch.scatter(torch.ones_like(mask), 1, torch .arange(bsz).view(-1, 1), 0) logits =torch.div(torch.mm(x, x.T), self.temperature) deno =torch.exp(logits)*anchor_mask prob =

Revised: logits -torch.log(deno.sum(1, keepdim=True)) loss =

Revised: -(anchor\_mask *mask *prob).sum(1)/mask.sum() return loss.view(1, bsz).mean() import torch.nn as nn # A 3-layer vanilla ConvNet example for Delog-SCL class CNN_DelogSCL(nn.

Revised: Module): def __init__(self, dim): super.__init__()

Revised: CNNs=[] losses=[] channels=[3,128, 256, 512] self.shape = for i in range(3):

Revised: CNNs.

Revised: append

Revised: (nn.

Revised: Sequential(nn.

Revised: Conv2d

Revised: (channels[i], channels[i+1],padding=1),nn.

Revised: ReLU())

Revised: losses.append(SupConLoss(self.shape*self.shape*channels

Revised: [ i+1])) self.

Revised: CNN =nn.ModuleList(CNNs)

Revised: self.loss =nn.ModuleList(losses)

Revised: self.fc

Revised: =nn.Sequential(flatten(), nn.

Revised: Linear(self.shape

Revised: * self.shape*channels[-1], 10))

Revised: self.ce

Revised: =nn.

Revised: CrossEntropyLoss() def forward(self, x, label): loss

Revised: = for i in range(3): # .detach() prevents the computation graph from propagating gradients to the next layer x = self.

Revised: CNN[i](x.detach

Revised: ()) if self.training: loss+=self.loss[i](x, label)

Revised: y =

Revised: self.fc(x.detach())

Revised: if self.training: loss+=self.ce(y, label) return loss return y