Original: A SSOCIATED L EARNING : A M ETHODOLOGY TO D ECOMPOSE E ND - TO -E ND B ACKPROPAGATION ON CNN, RNN, AND T RANSFORMER
Revised: A SSOCIATED L EARNING : A M ETHODOLOGY TO D ECOMPOSE E ND - TO -E ND B ACKPROPAGATION ON CNN, RNN, AND T RANSFORMER

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study associated learning (AL), an alternative methodology to end-to-end backpropagation (BP).
Revised: We study associated learning (AL), an alternative methodology to end-to-end backpropagation (BP).

Original: We introduce the workflow to convert a neural network into an AL-form network such that AL can be used to learn parameters for various types of neural networks.
Revised: We introduce the workflow to convert a neural network into an AL-form network such that AL can be used to learn parameters for various types of neural networks.

Original: We compare AL and BP on some of the most successful neural networks: convolutional neural networks, recurrent neural networks and Transformers.
Revised: We compare AL and BP on some of the most successful neural networks: convolutional neural networks, recurrent neural networks, and Transformers.

Original: Experimental results show that AL consistently outperforms BP on open datasets.
Revised: Experimental results show that AL consistently outperforms BP on open datasets.

Original: We discuss possible reasons for AL’s success, its limitations and AL’s newly discovered properties.
Revised: We discuss possible reasons for AL’s success, its limitations, and AL’s newly discovered properties.

Revised: Our implementation is available at https://github.com/Hibb-bb/AL .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Backpropagation (BP) is the keystone of modern deep learning.
Revised: Backpropagation (BP) is the keystone of modern deep learning.

Original: Although BP is the standard way to learn network parameters, it is far from ideal.
Revised: Although BP is the standard way to learn network parameters, it is far from ideal.

Original: Some of the most discussed issues of BP are optimization difficulties (e.g., vanishing gradients and exploding gradients (Hochreiter et al., 2001)) and training performance (e.g., backward locking (Jaderberg et al., 2017)).
Revised: Some of the most discussed issues of BP are optimization difficulties (e g., vanishing gradients and exploding gradients (Hochreiter et al., 2001)) and training performance (e g., backward locking (Jaderberg et al., 2017))

Original: It appears that custom network structures may be needed for different types of learning tasks.
Revised: It appears that custom network structures may be needed for different types of learning tasks.

Original: Among the various forms, convolutional neural networks (CNNs), recurrent neural networks (RNNs), and
Revised: Among the various forms, convolutional neural networks (CNNs), recurrent neural networks (RNNs) and

Original: Transformer networks (along with their extensions, e.g., LSTM (Hochreiter & Schmidhuber, 1997) and VGG (Simonyan & Zisserman, 2015)) are particularly useful.
Revised: Transformer networks (along with their extensions, e g., LSTM (Hochreiter & Schmidhuber, 1997) and VGG (Simonyan & Zisserman, 2015)) are particularly useful.

Original: These networks have been successfully applied in fields as varied as computer vision, natural language processing, signal processing, and others (Goodfellow et al., 2016; Deng & Yu, 2014).
Revised: These networks have been successfully applied in fields as varied as computer vision, natural language processing, signal processing, and others (Goodfellow et al., 2016; Deng & Yu, 2014).

Original: This paper studies a new learning approach—associated learning (AL)—an alternative to end-toend backpropagation learning.
Revised: This paper studies a new learning approach—associated learning (AL)—an alternative to end-toend backpropagation learning.

Original: AL decomposes BP’s global end-to-end training strategy into several small local optimization targets such that each layer has an isolated gradient flow.
Revised: AL decomposes BP’s global end-to-end training strategy into several small local optimization targets such that each layer has an isolated gradient flow.

Original: However, since most layers in AL do not interact with the final loss directly, we would expect AL-training models to be less accurate than BP.
Revised: However, since most layers in AL do not interact with the final loss directly, we would expect AL-training models to be less accurate than BP.

Original: Surprisingly, the original AL paper compares AL and BP using the
Revised: Surprisingly, the original AL paper compares AL and BP using the

Original: CNN network (and its extensions, e.g., VGG) and shows impressive results based on image classification datasets (MNIST, CIFAR-10, and CIFAR-100) (Kao & Chen, 2021).
Revised: CNN network (and its extensions, e.g., VGG) and shows impressive results based on image classification datasets (MNIST, CIFAR-10, and CIFAR-100) (Kao & Chen, 2021).

Original: We continue this line of study in two ways.
Revised: We continue this line of study in two ways.

Original: First, we discover more interesting properties of AL.
Revised: First, we discover more interesting properties of AL.

Original: Second, we show how to apply AL on different network structures, including VGG (Simonyan & Zisserman, 2015),
Revised: Second, we show how to apply AL on different network structures, including VGG (Simonyan & Zisserman, 2015),

Original: LSTM (Hochreiter & Schmidhuber, 1997), and Transformer (Vaswani et al., 2017).
Revised: LSTM (Hochreiter & Schmidhuber, 1997), and Transformer (Vaswani et al., 2017).

Original: Eventually, we compare the networks learned via AL and via BP on various tasks (image classification, sentiment analysis, and topic classification) based on public datasets (CIFAR-10, CIFAR-100, IMDB movie reviews, AG’s News, Stanford Sentiment Treebank (SST), and DBpedia).
Revised: Eventually, we compare the networks learned via AL and via BP on various tasks (image classification, sentiment analysis, and topic classification) based on public datasets (CIFAR-10, CIFAR-100, IMDB movie reviews, AG’s News, Stanford Sentiment Treebank (SST), and DBpedia).

Original: We find that AL consistently outperforms BP in most datasets.
Revised: We find that AL consistently outperforms BP in most datasets.

Original: Additionally, AL requires fewer epochs than BP when using early stopping but still yields excellent accuracy.
Revised: Additionally, AL requires fewer epochs than BP when using early stopping but still yields excellent accuracy.

Original: These results suggest that AL is a strong alternative to BP, as AL is effective for various tasks and various network structures.
Revised: These results suggest that AL is a strong alternative to BP, as AL is effective for various tasks and various network structures.

Original: The rest of the paper is organized as follows.
Revised: The rest of the paper is organized as follows.

Original: In Section 2, we introduce associated learning and its properties.
Revised: In Section 2, we introduce associated learning and its properties.

Original: Section 3 presents the experimental results, including a comparison of the model accuracies and convergence speed of AL and BP on VGG, LSTM, and Transformer, a generalization test of 
Revised: Section 3 presents the experiments, including a comparison of the model accuracies and convergence speed of AL and BP on VGG, LSTM, and Transformer, a generalization test of AL and

Original: AL and BP, and an ablation study.
Revised: BP, and an ablation study.

Original: Section 4 reviews previous work on backpropagation alternatives and the network structures that may look similar to AL.
Revised: Section 4 reviews previous work on backpropagation alternatives and the network structures that may look similar to AL.

Original: Finally, we conclude our contribution in Section 5.
Revised: Finally, we conclude our contribution in Section 5.

Original:  A SSOCIATED L EARNING 2. A N O VERVIEW OF AL: AL F ORM , T RAINING , AND I NFERENCE
Revised:  A SSOCIATED L EARNING 2. A N O VERVIEW OF AL: AL F ORM , T RAINING , AND I NFERENCE

Original: [Figure]
Revised: [Figure]

Original: To apply AL, we must transform the network into a different structure, which we call AL form below.
Revised: To apply AL, we must transform the network into a different structure, which we call AL form below.

Original: Instead of learning a function to map a feature vector x to a target y , the AL form performs metric learning by searching for functions to transform x and y into latent representations such that the distance between these two latent representations is close.
Revised: Instead of learning a function to map a feature vector x to a target y , the AL form performs metric learning by searching for functions to transform x and y into latent representations such that the distance between these two latent representations is close.

Original: Algorithm 1 shows the procedure to convert a neural work into an AL form.
Revised: Algorithm 1 shows the procedure to convert a neural work into an AL form.

Original: We use Figure 1 (a neural network with 3 hidden layers) as an example to show the process.
Revised: We use Figure 1 (a neural network with 3 hidden layers) as an example to show the process.

Original: We can regard functions f 1 , f 2 , f 3 as the encoders to convert the input feature vector into a latent representation and clf as a classifier to transform the latent representation into a target.
Revised: We can regard functions f 1 , f 2 , f 3 as the encoders to convert the input feature vector into a latent representation and clf as a classifier to transform the latent representation into a target.

Original: When converting this network to its
Revised: When converting this network to its

Original: AL form (referring to Figure 2), we only keep the encoders
Revised: AL form (referring to Figure 2), we only keep the encoders

Original: f i -s and extend the model architecture by adding a bridge function b i and an autoencoder for each layer i .
Revised: f i -s and extend the model architecture by adding a bridge function b i and an autoencoder for each layer i .

Original: Let s i − 1 and t i − 1 be the inputs of layer
Revised: Let s i − 1 and t i − 1 be the inputs of layer

Original: i (assuming s 0 :
Revised: i (assuming s 0 :

Original: = x and t 0 := y ), functions f i and g i convert s i − 1 and t i − 1 into latent representations s i and t i .
Revised: = x and t 0 := y ), functions f i and g i convert s i − 1 and t i − 1 into latent representations s i and t i .

Original: The function f i can be any type of forwarding block, such as a convolution layer, an LSTM layer, or a residual block in ResNet.
Revised: The function f i can be any type of forwarding block, such as a convolution layer, an LSTM layer, or a residual block in ResNet.

Original: Each bridge function b i projects s i (the latent representation of x at layer i ) to be close to t i (the latent representation of y at layer i ), i.e., b i ( f i ( s i − 1 ))
Revised: Each bridge function b i projects s i (the latent representation of x at layer i ) to be close to t i (the latent representation of y at layer i ), i.e., b i ( f i ( s i − 1 ))

Original: ≈ g i
Revised: ≈ g i

Original: ( t i − 1 ) .
Revised: ( t i − 1 ) .

Original: Meanwhile, g i not only extracts the latent representation of y , but also serves as the encoder for an autoencoder such that h i ( g i ( t i − 1 ))
Revised: Meanwhile, g i not only extracts the latent representation of y , but also serves as the encoder for an autoencoder such that h i ( g i ( t i − 1 ))

Original: ≈ t i − 

Revised: ≈ t i −

Original: ( h i is the decoder).
Revised: ( h i is the decoder).

Original: For all the AL-form networks in our experiments, each of the functions g i and h i is constructed by a linear transform followed by a non-linear activation function (e.g., tanh), and b i is a linear transformation of the input vector or input matrix.
Revised: For all the AL-form networks in our experiments, each of the functions g i and h i is constructed by a linear transform followed by a non-linear activation function (e.g., tanh), and b i is a linear transformation of the input vector or input matrix.

Revised: We discuss the number of neurons for the input, output, and hidden layers of the autoencoders in Section 6.9.

Original: To summarize, we minimize the following objective for each layer i during training:
Revised: To summarize, we minimize the following objective for each layer i during training:

Original: [Equation]
Revised: [Equation]

Original: where L iA denotes the associated loss of layer i , and L iAE indicates the autoencoder loss of layer i .
Revised: where L iA denotes the associated loss of layer i , and L iAE indicates the autoencoder loss of layer i .

Original: Algorithm 2 gives the training algorithm for AL.
Revised: Algorithm 2 gives the training algorithm for AL.

Original: Since each layer has its objective function (Equation 3), we can parallelly update the parameters of different layers using a pipeline to increase the training throughput (details in Section 2.2.3).
Revised: Since each layer has its objective function (Equation 3), we can parallelly update the parameters of different layers using a pipeline to increase the training throughput (details in Section 2.2.3).

Original: For inference, if an AL network has L layers, an input x goes through f 1 , . . .
Revised: For inference, if an AL network has L layers, an input x goes through f 1 , . . .

Original: , f L to generate x ’s latent representation ( s L ), which is transformed to y ’s latent representation ( t L ) via b L .
Revised: , f L to generate x ’s latent representation ( s L ), which is transformed to y ’s latent representation ( t L ) via b L .

Original: Next, t L is converted to y via h L , . .
Revised: Next, t L is converted to y via h L , . .

Original: , h 1 .
Revised: , h 1 .

Original: The autoencoders’ encoding functions
Revised: The autoencoders’ encoding functions

Original: g i -s are not used during inference.
Revised: g i -s are not used during inference.

Original: Take Figure 2 as an example, the inference path is ( f 1 −→ f 2 −→ f 3 −→ b 3 −→ h 3 −→ h 2 −→ h 1 ) .
Revised: Take Figure 2 as an example, the inference path is ( f 1 −→ f 2 −→ f 3 −→ b 3 −→ h 3 −→ h 2 −→ h 1 ) .

Original: Although it could be unintuitive to concatenate two unconnected transformation functions as an inference path (i.e., h i +1 followed by h
Revised: Although it could be unintuitive to concatenate two unconnected transformation functions as an inference path (i.e., h i +1 followed by h

Original: i , i = 1 , 2 , 3 ), it is valid because h i +

Revised: i , i = 1 , 2 , 3 ), it is valid because h i +

Original: ( t i +1 )
Revised: ( t i +1 )

Original: =
Revised: =

Original: t ′ i
Revised: t ′ i

Original: ≈
Revised: ≈

Original: t i .
Revised: t i .

Original: Input a neural network N = ( f1 , f 2 , ...
Revised: Input a neural network N = ( f f f

Original: f L ), targety ∈ R 1 , feature vector x ∈ R m
Revised: ), target , feature vector x ∈ R m

Original: Output an AL network A 1: s 0 ← x ; t 0 ← y 2: for i = 1 to L do 3: s i =
Revised: Output an AL network A 1: s 0 ← x ; t 0 ← y 2: for i = 1 to L do 3: s i =

Original: f i ( s i − 1 ) 4: insert a function g i , where t
Revised: f i ( s i − 1 ) 4: insert a function g i , where t

Original: i =
Revised: i =

Original: g i ( t i − 1 ) 5: insert a function b i , where s ′ i
Revised: g i ( t i − 1 ) 5: insert a function b i , where s ′ i

Original: =
Revised: =

Original: b i ( s i )
Revised: b i ( s i )

Original: ▷ adding bridge function 6: insert a function h
Revised: ▷ adding bridge function 6: insert a function h

Original: i , where t ′ i − 

Revised: i , where t ′ i −

Original: = h i ( t i )
Revised: = h i ( t i )

Original: ▷ adding a decoder for an autoencoder 7:
Revised: ▷ adding a decoder for an autoencoder 7:

Original: L iAL =
Revised: L iAL =

Original: MSE ( s ′ i , t i ) + MSE ( t i − 1 , t ′ i − 1 )
Revised: MSE ( s ′ i , t i ) + MSE ( t i − 1 , t ′ i − 1 )

Original: ▷ associated loss and autoencoder loss 8: return A
Revised: ▷ associated loss and autoencoder loss 8: return A

Original: = ( a 1 , . .
Revised: = ( a 1 , . .

Original: . , a L ) , a i = {
Revised: . , a L ) , a i = {

Original: f i , b i , g i , h i } Li =
Revised: f i , b i , g i , h i } Li =

Original: Input an AL network A
Revised: Input an AL network A

Original: = ( a 1 , . .
Revised: = ( a 1 , . .

Original: . , a L ) , a i = {
Revised: . , a L ) , a i = {

Original: f i , b i , g i , h i } Li =1 , training features and targets ( X, Y )
Revised: f i , b i , g i , h i } Li =1 , training features and targets ( X, Y )

Original: Output a fine-tuned AL network A 1: repeat 2:
Revised: Output a fine-tuned AL network A 1: repeat 2:

Original: Sample x , y from X, Y .
Revised: Sample x , y from X, Y .

Original: 3: s 0 ← x ; t 0 ← y 4: for i = 1 to L do 5: s i ← f i ( s i − 1 ) 6: t i ← g i ( t i − 1 ) 7: s ′
Revised: 3: s 0 ← x ; t 0 ← y 4: for i = 1 to L do 5: s i ← f i ( s i − 1 ) 6: t i ← g i ( t i − 1 ) 7: s ′

Original: i ←
Revised: i ←

Original: b i ( s i ) 8: t ′ i − 1 ← h i ( t i ) 9:
Revised: b i ( s i ) 8: t ′ i − 1 ← h i ( t i ) 9:

Original: L iA ← MSE ( s ′ i , t i )
Revised: L iA ← MSE ( s ′ i , t i )

Original: ▷ associated loss 10:
Revised: ▷ associated loss 10:

Original: L i AE ← MSE ( t i − 1 , t ′ i − 1 )
Revised: L iAE ← MSE ( t i − 1 , t ′ i − 1 )

Original: ▷ autoencoder loss 11:
Revised: ▷ autoencoder loss 11:

Original: Update the parameter of f i , b i , g i according to ∇L A 12:
Revised: Update the parameter of f i , b i , g i according to ∇L A 12:

Original: Update the parameter of g i ,
Revised: Update the parameter of g i ,

Original: h i according to ∇L AE 13: until converges
Revised: h i according to ∇L AE 13: until converges

Original: 2.1. A PPLYING AL TO D IFFERENT N EURAL N ETWORK S TRUCTURES
Revised: 2.1. A PPLYING AL TO D IFFERENT N EURAL N ETWORK S TRUCTURES

Original: This section introduces details on converting some of the most successful neural network structures into their corresponding AL forms.
Revised: This section introduces details on converting some of the most successful neural network structures into their corresponding AL forms.

Original: We discuss vanilla CNN and VGG (used as the representative models for CNN-based models), LSTM (used as the representative model for RNN-based models), and Transformer.
Revised: We discuss vanilla CNN and VGG (used as the representative models for CNN-based models), LSTM (used as the representative model for RNN-based models), and Transformer.

Original: We also discuss how to integrate word embeddings into an AL-form network.
Revised: We also discuss how to integrate word embeddings into an AL-form network.

Original:  The AL form of an CNN architecture (here, vanilla CNN and VGG) uses f i -s to convert an input image x into latent representations by convolutions, just like a regular CNN or a VGG does.
Revised: CNN The AL form of an CNN architecture (here, vanilla CNN and VGG) uses f i -s to convert an input image x into latent representations by convolutions, just like a regular CNN or a VGG does.

Original: The associated loss at layer i is defined as the distance between b i ( s i ) (transforming the flattened feature map at layer i into t
Revised: The associated loss at layer i is defined as the distance between b i ( s i ) (transforming the flattened feature map at layer i into t

Original: i ’s shape) and t i (the latent representation of y ).
Revised: i ’s shape) and t i (the latent representation of y ).

Original: RNN The AL form of an RNN architecture (here, LSTM and Bi-LSTM) uses the internal state to iteratively process each element in the input sequence x and generates the new internal state, just like a regular RNN does.
Revised: RNN The AL form of an RNN architecture (here, LSTM and Bi-LSTM) uses the internal state to iteratively process each element in the input sequence x and generates the new internal state, just like a regular RNN does.

Original: After reading the entire input sequence x , we define s i the latent representation of x at layer i by the final internal state.
Revised: After reading the entire input sequence x , we define s i the latent representation of x at layer i by the final internal state.

Original: Consequently, the associated loss is defined as the distance between b i ( s i ) (transforming the final internal state at layer i into t
Revised: Consequently, the associated loss is defined as the distance between b i ( s i ) (transforming the final internal state at layer i into t

Original: i ’s shape) and t i (the latent representation of y ).
Revised: i ’s shape) and t i (the latent representation of y ).

Revised: Details are shown in Section 6.2.

Original: Transformer The AL form of a Transformer encodes the input sequence x into a list of vectors, as a regular Transformer does.
Revised: Transformer The AL form of a Transformer encodes the input sequence x into a list of vectors, as a regular Transformer does.

Original: We define s i (the latent representation of x ) by computing the mean-pooling on the encoded vectors.
Revised: We define s i (the latent representation of x ) by computing the mean-pooling on the encoded vectors.

Original: The associated loss is defined as the distance between b i ( s i ) (transforming the mean-pooling output into t i ’s shape) and t i (the latent representations of y ). 
Revised: The associated loss is defined as the distance between b i ( s i ) (transforming the mean-pooling output into t i ’s shape) and t i (the latent representations of y ). Details are shown in Section 6.2.

Original: Word Embedding Word embeddings are frequently used as the input of LSTM or Transformer for NLP tasks.
Revised: Embedding Word embeddings are frequently used as the input of LSTM or Transformer for NLP tasks.

Original: We use mean-pooling to aggregate all token’s word embeddings as s i .
Revised: We use mean-pooling to aggregate all token’s word embeddings as s i .

Original: 2. P ROPERTIES OF AL
Revised: 2. P ROPERTIES OF AL

Original: This section presents three properties of AL that do not exist in BP: forward shortcuts, dynamic layer accumulation, and pipelines.
Revised: This section presents three properties of AL that do not exist in BP: forward shortcuts, dynamic layer accumulation, and pipelines

Original:  2.2. F ORWARD S HORTCUTS
Revised: . 2.2. F ORWARD S HORTCUTS

Original: Forward shortcuts enable faster inference.
Revised: Forward shortcuts enable faster inference.

Original: However, we can also leverage “shortcut paths” for faster inference.
Revised: However, we can also leverage “shortcut paths” for faster inference.

Original: As shown in Algorithm 3, given an integer ℓ ( 1 ≤ ℓ ≤ L ), the bridge function b ℓ can serve as a shortcut to transform s ℓ to s ′ ℓ , which should be close to t ℓ when the network is well trained.
Revised: As shown in Algorithm 3, given an integer ℓ ( 1 ≤ ℓ ≤ L ), the bridge function b ℓ can serve as a shortcut to transform s ℓ to s ′ ℓ , which should be close to t ℓ when the network is well trained.

Original: As a result, we can skip f j , b j , and h j for all j > ℓ to reduce the length of the inference function.
Revised: As a result, we can skip f j , b j , and h j for all j > ℓ to reduce the length of the inference function.

Original: In other words, we have multiple inference functions based on a single model.
Revised: In other words, we have multiple inference functions based on a single model.

Original: When inference time is critical, we can select a shorter inference path, e.g., x f 1 −→ s 1 b 1 −→ t 1 h 1 −→ y .
Revised: When inference time is critical, we can select a shorter inference path, e.g., x f 1 −→ s 1 b 1 −→ t 1 h 1 −→ y .

Original: On the other hand, if inference time is unimportant, we can dynamically adjust the model complexity by modifying the number of AL layers used at the inference phase to reduce overfitting or underfitting.
Revised: On the other hand, if inference time is unimportant, we can dynamically adjust the model complexity by modifying the number of AL layers used at the inference phase to reduce overfitting or underfitting.

Original: 2.2. D YNAMIC L AYER A CCUMULATION
Revised: 2.2. D YNAMIC L AYER A CCUMULATION

Original: An AL-form network allows the dynamic creation of new layers during training.
Revised: An AL-form network allows the dynamic creation of new layers during training.

Original: Specifically, we initially create an AL-form network with k AL layers and train the network based on the strategy introduced in Section 2.1.
Revised: Specifically, we initially create an AL-form network with k AL layers and train the network based on the strategy introduced in Section 2.1.

Original: If the model still underfits, we simply add the ( k + 1) -th
Revised: If the model still underfits, we simply add the ( k + 1) -th

Original: AL layer on top of the first k AL layers.
Revised: AL layer on top of the first k AL layers.

Original: We may choose to fix the parameters in the first k AL layers and train only the parameters in the ( k +1) -th
Revised: We may choose to fix the parameters in the first k AL layers and train only the parameters in the ( k +1) -th

Original: AL layer.
Revised: AL layer.

Original: In contrast, when using a standard neural network, adding layers dynamically is more complicated.
Revised: In contrast, when using a standard neural network, adding layers dynamically is more complicated.

Original: 2.2. P IPELINE
Revised: 2.2. P IPELINE

Original: In an AL-form network, the network parameters in different AL layers can be trained simultaneously via pipelines because each AL layer i has a local objective function.
Revised: In an AL-form network, the network parameters in different AL layers can be trained simultaneously via pipelines because each AL layer i has a local objective function.

Original: In contrast, it is difficult to simultaneously update the parameters of different layers for a standard BP-based neural network.
Revised: In contrast, it is difficult to simultaneously update the parameters of different layers for a standard BP-based neural network.

Original: Given n training instances in a standard neural network with L hidden layers, if we apply BP for training, the time complexity is O ( nL ) because each instance must complete the entire forward ( L +1 transformations) and backward ( L +1 transformations) process.
Revised: Given n training instances in a standard neural network with L hidden layers, if we apply BP for training, the time complexity is O ( nL ) because each instance must complete the entire forward ( L +1 transformations) and backward ( L +1 transformations) process.

Original: Thus, for such an architecture, the time complexity increases linearly with the total number of layers L .
Revised: Thus, for such an architecture, the time complexity increases linearly with the total number of layers L .

Original: However, since each AL layer has its local objective, each layer can update its parameters without waiting for the gradient from preceding layers.
Revised: However, since each AL layer has its local objective, each layer can update its parameters without waiting for the gradient from preceding layers.

Original: As a result, for an AL-form network with L AL layers, using L computation units can make the parameter update process fully pipelined, as explained below.
Revised: As a result, for an AL-form network with L AL layers, using L computation units can make the parameter update process fully pipelined, as explained below.

Original: We denote task ji as the task of computing both forward and backward propagation for an AL layer i based on the j -th instance.
Revised: We denote task ji as the task of computing both forward and backward propagation for an AL layer i based on the j -th instance.

Original: If an AL layer i finishes task 1 i at time step t , at the next time step t + 1 ,
Revised: If an AL layer i finishes task 1 i at time step t , at the next time step t + 1 ,

Original: AL layers i can continue performing task 2 i since all parameters in this layer are updated based on task 1 i .
Revised: AL layers i can continue performing task 2 i since all parameters in this layer are updated based on task 1 i .

Original: Meanwhile, AL layer i + 1 performs task 1 i +1 .
Revised: Meanwhile, AL layer i + 1 performs task 1 i +1 .

Original: Consequently, training based on the first instance requires O ( L ) , and each of the following n − 1 instances requires extra O (1) time units.
Revised: Consequently, training based on the first instance requires O ( L ) , and each of the following n − 1 instances requires extra O (1) time units.

Original: The time complexity of one training epoch eventually reduces to O ( n + L ) .
Revised: The time complexity of one training epoch eventually reduces to O ( n + L ) .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To evaluate how AL works on different network architectures, we conducted experiments on image classification with vanilla CNN and VGG and on sentiment and topic classification with LSTM and
Revised: To evaluate how AL works on different network architectures, we conducted experiments on image classification with vanilla CNN and VGG and on sentiment and topic classification with LSTM and

Original: Transformer (using pre-trained word embeddings as input).
Revised: Transformer (using pre-trained word embeddings as input).

Original: In addition, we conducted a case study on AL’s generalization ability, evaluated by fitting AL models and BP models on different portions of noisy labels.
Revised: In addition, we conducted a case study on AL’s generalization ability, evaluated by fitting AL models and BP models on different portions of noisy labels.

Original: Finally, we discuss possible reasons for AL’s success based on ablation studies.
Revised: Finally, we discuss possible reasons for AL’s success based on ablation studies.

Original: We report more detailed settings in the Appendix.
Revised: We report more detailed settings in the Appendix.

Original: 3. D ATASETS
Revised: 3. D ATASETS

Original: We used four datasets (IMDB Review, AG’s News corpus, DBpedia Ontology, and the Stanford Sentiment Treebank) to compare AL and BP on LSTM and Transformer, and two datasets (CIFAR10 and Fashion-MNIST) to compare AL and BP onvanilla CNN and VGG.
Revised: We used four datasets (IMDB Review, AG’s News corpus, DBpedia Ontology, and the Stanford Sentiment Treebank) to compare AL and BP on LSTM and Transformer, and two datasets (CIFARand Fashion-MNIST) to compare AL and BP on CNN and VGG.

Original: Dataset details are in Appendix 6.
Revised: Dataset details are in Appendix 6.

Original: 3. T EXT C LASSIFICATION (LSTM AND T RANSFORMER )
Revised: 3. T EXT C LASSIFICATION (LSTM AND T RANSFORMER )

Original: We compared BP and AL on LSTM and Transformer based on the text classification task.
Revised: We compared BP and AL on LSTM and Transformer based on the text classification task.

Original: For LSTM, we used a 2-layer bidirectional LSTM with pre-trained GloVe word embeddings (Pennington et al., 2014) as input.
Revised: For LSTM, we used a 2-layer bidirectional LSTM with pre-trained GloVe word embeddings (Pennington et al., 2014) as input.

Original: For Transformer, we used a 2-layer Transformer encoder, again with pre-trained GloVe word embeddings.
Revised: For Transformer, we used a 2-layer Transformer encoder, again with pre-trained GloVe word embeddings.

Original: During training, we recorded the accuracies of the validation set for every epoch and saved that model with the best performance on the validation set for testing.
Revised: During training, we recorded the accuracies of the validation set for every epoch and saved that model with the best performance on the validation set for testing.

Original: For each model, we repeated the process five times and reported the average accuracy and standard deviation.
Revised: For each model, we repeated the process five times and reported the average accuracy and standard deviation.

Original: For a fair comparison, we forced the AL parameter counts to not exceed those of the compared standard neural network by reducing AL’s hidden dimension size.on
Revised: For a fair comparison, we forced the AL parameter counts to not exceed those of the compared standard neural network by reducing AL’s hidden dimension size.

Original: AL-form network <X>, and <X>-AL-SC i denotes shortcut inference through

Original: bridge function b i +1 .

Original: EMB-AL-SC indicates inference through the shortcut in the embedding layer,

Original: i

Original: [Table]
Revised: [Table]

Original: e., after computing mean-pooling on the word embeddings, the output is passed to a bridge function.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] We report the accuracies of LSTM and Transformer trained via BP and AL in Table 1.
Revised: We report the accuracies of LSTM and Transformer trained via BP and AL in Table 1.

Original: We also report the accuracies based on the predictions using shorter inference paths in the same Table.
Revised: We also report the accuracies based on the predictions using shorter inference paths in the same Table.

Original: Although
Revised: Although

Original: AL is trained by isolated gradient flows, AL-form LSTMs outperform their standard BP-trained counterparts on most datasets.
Revised: AL is trained by isolated gradient flows, AL-form LSTMs outperform their standard BP-trained counterparts on most datasets.

Original: Transformer-AL also outperforms Transformer on every dataset.
Revised: Transformer-AL also outperforms Transformer on every dataset.

Original: Also, we observe that standard Transformer easily overfits smaller datasets like SST and IMDB, resulting in poor test accuracies, but Transformer-AL does not seem to have this weakness.
Revised: Also, we observe that standard Transformer easily overfits smaller datasets like SST and IMDB, resulting in poor test accuracies, but Transformer-AL does not seem to have this weakness.

Original: In Table 2 and Table [Table] we report the epoch that yielded the best results.
Revised: In Table 2 and Table 3, we report the epoch that yielded the best results.

Original: We discover that LSTMAL converges much faster than BP, suggesting that training time will be improved if applying early  stopping.
Revised: We discover that LSTMAL converges much faster than BP, suggesting that training time will be improved if applying early [Table] [Table] stopping.

Original: For the Transformer-AL, although the best validation and test accuracy come at roughly the 20th epoch on AGNews, Transformer-AL reaches high validation and test accuracy within the first 5 epochs.
Revised: For the Transformer-AL, although the best validation and test accuracy come at roughly the 20th epoch on AGNews, Transformer-AL reaches high validation and test accuracy within the first 5 epochs.

Original: Since we report only the epoch that reaches the best validation and test accuracies, the required epochs look large.
Revised: Since we report only the epoch that reaches the best validation and test accuracies, the required epochs look large.

Original: Particularly, Transformer-AL achieved 99% of the best validation set accuracy before the 5th epoch.
Revised: Particularly, Transformer-AL achieved 99% of the best validation set accuracy before the 5th epoch.

Original: More details can be found in Appendix 6.6.
Revised: More details can be found in Appendix 6.7.

Original:  I MAGE C 
Revised: 3. I MAGE C LASSIFICATION (V ANILLA CNN AND VGG)

Original: [Table]

Original: We also reproduced the experiments in Kao & Chen (2021) to determine whether AL-form CNNs also converge faster.
Revised: We also reproduced the experiments in Kao & Chen (2021) to determine whether AL-form CNNs also converge faster.

Original: In image classification, we report performance on the CIFAR-10 and FashionMNIST benchmarks.
Revised: In image classification, we report performance on the CIFAR-10 and FashionMNIST benchmarks.

Original: We selected FashionMNIST instead of MNIST because MNIST is likely too simple to show the power of different models: CIFAR-10 is more challenging than FashionMNIST since it contains three channels, whereas FashionMNIST is limited to grayscale images.
Revised: We selected FashionMNIST instead of MNIST because MNIST is likely too simple to show the power of different models: CIFAR-10 is more challenging than FashionMNIST since it contains three channels, whereas FashionMNIST is limited to grayscale images.

Original: Tableshows the results: all AL models converge faster than BP.
Revised: Tableshows the results: all AL models converge faster than BP.

Original: Figure 4 in Appendix 6.5 shows the relationship between epochs and test accuracies on CNN and CNN-AL.
Revised: Figure 6 in Appendix 6.6 shows the relationship between epochs and test accuracies on CNN and CNN-AL.

Original: Also, AL models perform slightly better in terms of accuracy, consistent with the results demonstrated in Kao & Chen (2021)
Revised: Also, AL models perform slightly better in terms of accuracy, consistent with the results demonstrated in Kao & Chen (2021)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 3. G ENERALIZATION
Revised: G ENERALIZATION

Original: To assess the generalizability of a model, a common approach is to empirically compare the errors on the training data and test data.
Revised: To assess the generalizability of a model [Table] a common approach is to empirically compare the errors on the training data and test data.

Original: Garg et al.
Revised: Garg et al.

Original: (2021) show another way to evaluate generalizability: a model is prone to overfitting if it closely fits a dataset with random labels.
Revised: (2021) show another way to evaluate generalizability: a model is prone to overfitting if it closely fits a dataset with random labels.

Original: We use both methods to show that an AL-trained model may be more general than that trained by BP.
Revised: We use both methods to show that an AL-trained model may be more general than that trained by BP.

Original: We took θ proportion of data, in which we changed the labels randomly.
Revised: We took θ proportion of data, in which we changed the labels randomly.

Original: We used different proportions to train models.
Revised: We used different proportions to train models.

Original: We report both the training and test accuracies on clean test data.
Revised: We report both the training and test accuracies on clean test data.

Original:  shows the results.
Revised: Table 5 shows the results.

Original: First, as shown in the last column (for a label noise rate of θ = 0 ), the training accuracy of the CNN is larger than that of the CNN-AL, but CNN-AL outperforms CNN in terms of test accuracy.
Revised: First, as shown in the last column (for a label noise rate of θ = 0 ), the training accuracy of the CNN is larger than that of the CNN-AL, but CNN-AL outperforms CNN [Figure]

Original: Second, with θ > 0 , the vanilla CNN network with BP training fits the
Revised: i

Original: [Figure]
Revised: [Figure]

Original:  randomly labeled training data much better than the AL-form CNN.
Revised: in terms of test accuracy. Second, with θ > 0 , the vanilla CNN network with BP training fits the randomly labeled training data much better than the AL-form CNN.

Original: However, if we evaluate model accuracies based on the clean test data, the AL-form CNN performs better.
Revised: However, if we evaluate model accuracies based on the clean test data, the AL-form CNN performs better.

Original: Taken together, the above empirical evidence suggests that the new model may generalize better (Garg et al., 2021).
Revised: Taken together, the above empirical evidence suggests that the new model may generalize better (Garg et al., 2021).

Original: The relationship between accuracy and training epochs (for θ = 1 . 0 , [Table] 2 , and 0 . 6 ) is given in Figure [Table]
Revised: The relationship between accuracy and training epochs (for θ = 1 . 0 , 0 . 2 , and 0 . 6 ) is given in Figure 3.

Original: [Table]
Revised: [Table]

Original: 3. W HAT MAKES AL WORK – AN A BLATION S TUDY
Revised: 3. W HAT MAKES AL WORK – AN A BLATION S TUDY

Original: Wehave empirically validated that AL generates excellent models, converges fast, and generalizes well.
Revised: We empirically validates that AL generates great models, converges fast, and generalizes well.

Original: We further conducted ablation studies based on AG’s news dataset tobetter understand why AL has these properties.
Revised: Below we report ablation studies using AG’s news dataset to understand why AL performs excellently.

Original: First, we ensure the gradient flow in each AL layer does not influence the gradients in other layers via the gradient stopping mechanism.
Revised: First, we ensure the gradient flow in each AL layer does not influence the gradients in other layers via the gradient stopping mechanism.

Original: However, to optimize the objective function (Equation 3), either allowing or blocking the gradient flow between AL layers is acceptable.
Revised: However, to optimize the objective function (Equation 3), either allowing or blocking the gradient flow between AL layers is acceptable.

Original: We sought to determine whether allowing gradient flow between AL layers improves model accuracy (although such a design may make pipelining infeasible).
Revised: We sought to determine whether allowing gradient flow between AL layers improves model accuracy (although such a design may make pipelining infeasible).

Original: Hence, we disabled all the gradient stops (allowing gradients to flow through different AL layers).
Revised: Hence, we disabled all the gradient stops (allowing gradients to flow through different AL layers).

Original: The results are denoted as “LSTM-AL w/o grad.
Revised: The results are denoted as “LSTM-AL w/o grad.

Original: stops” in
Revised: stops” in

Original: Table 6: accuracy is nearly unchanged.
Revised: Table 6: accuracy is nearly unchanged.

Original: Therefore, given enough computing nodes, applying a pipelined AL increases training throughput without sacrificing model accuracy.
Revised: Therefore, given enough computing nodes, applying a pipelined AL increases training throughput without sacrificing model accuracy.

Original: Second, to test the effect of associated loss, we disabled the gradient stops and removed the associated loss (except the last AL layer) from the model.
Revised: Second, to test the effect of associated loss, we disabled the gradient stops and removed the associated loss (except the last AL layer) from the model.

Original: The second row of Table 6 shows the result: although the accuracy is similar to that with associated loss, far more epochs are needed to converge.
Revised: The second row of Table 6 shows the result: although the accuracy is similar to that with associated loss, far more epochs are needed to converge.

Original: So, the bridge functions likely facilitate model training by gradually bringing s ′ i and t i closer.
Revised: So, the bridge functions likely facilitate model training by gradually bringing s ′ i and t i closer.

Original: With the third experiment, we sought to investigate the effect of the autoencoder components in the
Revised: With the third experiment, we sought to investigate the effect of the autoencoder components in the

Original: AL model.
Revised: AL model.

Original: To do this, we passed the output of a normal Bi-LSTM to a bridge function, whose output t 1 was fed to the bottleneck layer of an autoencoder, which decodes t 1 back to the target y .
Revised: To do this, we passed the output of a normal Bi-LSTM to a bridge function, whose output t 1 was fed to the bottleneck layer of an autoencoder, which decodes t 1 back to the target y .

Original: Compared with the original Bi-LSTM classifier, the new design results in higher accuracy, as seen in the third (LSTM + AE*1) in Table 6.
Revised: Compared with the original Bi-LSTM classifier, the new design results in higher accuracy, as seen in the third (LSTM + AE*1) in Table 6.

Original: This suggests that replacing the standard classifier with a decoder of an AE and training it with AL objectives could produce a better model.
Revised: This suggests that replacing the standard classifier with a decoder of an AE and training it with AL objectives could produce a better model.

Original: Additionally, we experimented with removing all autoencoders and using b i as the classifier to predict y for each AL layer (similar to the early exit technique in Teerapittayanon et al.
Revised: Additionally, we experimented with removing all autoencoders and using b i as the classifier to predict y for each AL layer (similar to the early exit technique in Teerapittayanon et al.

Original: We use “LSTM-AL w/o
Revised: We use “LSTM-AL w/o

Original: AE (SC-1)” and “LSTM-AL w/o AE (full)” in Table 6 to denote “early exit at the first layer” and “using b L the last bridge as the classifier”, respectively.
Revised: AE (SC-1)” and “LSTM-AL w/o AE (full)” in Table 6 to denote “early exit at the first layer” and “using b L the last bridge as the classifier”, respectively.

Original: The results show that using autoencoders gives a better predicting model.
Revised: The results show that using autoencoders gives a better predicting model.

Original: We suspect that each autoencoder perhaps performs some kind of feature extraction and regularization.
Revised: We suspect that each autoencoder perhaps performs some kind of feature extraction and regularization.

Original: Particularly, although it is easy to transform one-hot encoded y to t 1 to decrease associated loss and transform t 1 to t ′ 0 to decrease the autoencoder loss, the network still needs to learn more challenging tasks: converting t i to t
Revised: Particularly, although it is easy to transform one-hot encoded y to t 1 to decrease associated loss and transform t 1 to t ′ 0 to decrease the autoencoder loss, the network still needs to learn more challenging tasks: converting t i to t

Original: i +1 and t i +1 to t ′
Revised: i +1 and t i +1 to t ′

Original: i for i > 1 to reduce the associated loss and autoencoder loss of layer i .
Revised: i for i > 1 to reduce the associated loss and autoencoder loss of layer i .

Original: Consequently, adding autoencoders may prevent overfitting for the components beyond layer 1.
Revised: Consequently, adding autoencoders may prevent overfitting for the components beyond layer 1.

Original: Finally, for LSTM-AL
Revised: Finally, for LSTM-AL

Original: [Table]
Revised: [Table]

Original: the inference path goes through 4 transformation functions ( b 3 −→ h 3 −→ h 2 −→ h 1 ) after the LSTM module.
Revised: the inference path goes through 4 transformation functions ( b 3 −→ h 3 −→ h 2 −→ h 1 ) after the LSTM module.

Original: To ensure a fair comparison
Revised: To ensure a fair comparison

Original: [Table]
Revised: [Table]

Original: we also tested a network with a 3-layer MLP classifier after the standard LSTM model.
Revised: we also tested a network with a 3-layer MLP classifier after the standard LSTM model.

Original: The result is denoted as LSTM + MLP* in Table 6.
Revised: The result is denoted as LSTM + MLP* in Table 6.

Original: As shown, replacing the MLP from one layer to three layers does not produce a better model,
Revised: As shown, replacing the MLP from one layer to three layers does not produce a better model,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e., the number of layers is not why LSTM-AL is a better model.
Revised: e., the number of layers is not why LSTM-AL is a better model.

Original: [Table]
Revised: [Table]

Original:  R ELATED W ORK 4. B ACKPROPAGATION A LTERNATIVES
Revised:  R ELATED W ORK 4. B ACKPROPAGATION A LTERNATIVES

Original: Backpropagation is fundamental in deep learning.
Revised: Backpropagation is fundamental in deep learning.

Original: However, backpropagation suffers from optimization and performance issues.
Revised: However, backpropagation suffers from optimization and performance issues.

Original: Related work usually concerns methodologies that better imitate the signal transmission process of biological neural networks, since biological neural networks are highly efficient in analyzing visual, audio, textual, and other signals (Lee et al., 2015; Kao & Chen, 2021; Nøkland, 2016; Ororbia et al., 2018; Ororbia & Mali, 2019).
Revised: Related work usually concerns methodologies that better imitate the signal transmission process of biological neural networks, since biological neural networks are highly efficient in analyzing visual, audio, textual, and other signals (Lee et al., 2015; Kao & Chen, 2021; Nøkland, 2016; Ororbia et al., 2018; Ororbia & Mali, 2019).

Original: Alternatives to end-to-end backpropagation can be categorized into three types: proxy objective, target propagation, and synthetic gradients (Duan & Principe, 2021).
Revised: Alternatives to end-to-end backpropagation can be categorized into three types: proxy objective, target propagation, and synthetic gradients (Duan & Principe, 2021).

Original: Below we introduce each kind and the representative works.
Revised: Below we introduce each kind and the representative works.

Original: The first type—proxy objective—uses a local objective function for one layer i to learn θ
Revised: The first type—proxy objective—uses a local objective function for one layer i to learn θ

Original: i , the parameters for the current layer i .
Revised: i , the parameters for the current layer i .

Original: The model then fixes θ i , adds one more layer
Revised: The model then fixes θ i , adds one more layer

Original: i + 1 with another objective function, and learns θ i +1 , the parameters for the new layer.
Revised: i + 1 with another objective function, and learns θ i +1 , the parameters for the new layer.

Original: Consequently, proxy objective decouples the end-to-end backpropagation since the parameters in each layer are trained via a local objective function (Nøkland & Eidnes, 2019; Belilovsky et al., 2019; 2020; Löwe et al., 2019).
Revised: Consequently, proxy objective decouples the end-to-end backpropagation since the parameters in each layer are trained via a local objective function (Nøkland & Eidnes, 2019; Belilovsky et al., 2019; 2020; Löwe et al., 2019).

Original: The second type—target propagation—approximates g i , the inverse of the downstream layers for every layer i .
Revised: The second type—target propagation—approximates g i , the inverse of the downstream layers for every layer i .

Original: To update θ i , the parameters of layer i , the model backpropagates the target (not gradient) through function g i .
Revised: To update θ i , the parameters of layer i , the model backpropagates the target (not gradient) through function g i .

Original: Consequently, target propagation handles the situation where the relationship between the parameters and the cost is highly non-linear (e.g., discrete), which is otherwise difficult to solve using backpropagation (Lee et al., 2015; Meulemans et al., 2020).
Revised: Consequently, target propagation handles the situation where the relationship between the parameters and the cost is highly non-linear (e.g., discrete), which is otherwise difficult to solve using backpropagation (Lee et al., 2015; Meulemans et al., 2020).

Original: The third type is synthetic gradients, which learns a function to approximate the gradient for each layer and utilizes the approximated gradients as the true gradients to update the parameters.
Revised: The third type is synthetic gradients, which learns a function to approximate the gradient for each layer and utilizes the approximated gradients as the true gradients to update the parameters.

Original: This method thus decouples the layer-wise parameters (Jaderberg et al., 2017; Czarnecki et al., 2017;
Revised: This method thus decouples the layer-wise parameters (Jaderberg et al., 2017; Czarnecki et al., 2017;

Original: Lansdell et al., 2019).
Revised: Lansdell et al., 2019).

Original: Associated learning is different from these methods in several ways.
Revised: Associated learning is different from these methods in several ways.

Original: First, both proxy gradient and target propagation propagate the signal (loss or target) from the output layer directly to each layer.
Revised: First, both proxy gradient and target propagation propagate the signal (loss or target) from the output layer directly to each layer.

Original: Biologically, this could be implausible because when in deep networks, hidden layers near the inputs are unlikely to receive direct signals from the output layer that are far away.
Revised: Biologically, this could be implausible because when in deep networks, hidden layers near the inputs are unlikely to receive direct signals from the output layer that are far away.

Original: Associated learning, on the other hand, does not require each component to receive signals directly from the output layer.
Revised: Associated learning, on the other hand, does not require each component to receive signals directly from the output layer.

Original: Additionally, although proxy gradient allows each layer to have a local objective, most models of this type still learn the parameters in a layer-wise fashion, so it may still be challenging to learn the parameters using a pipeline.
Revised: Additionally, although proxy gradient allows each layer to have a local objective, most models of this type still learn the parameters in a layer-wise fashion, so it may still be challenging to learn the parameters using a pipeline.

Original: One exception is Greedy InfoMax (GIM) (Löwe et al., 2019), which does not require a direct signal from the output layer.
Revised: One exception is Greedy InfoMax (GIM) (Löwe et al., 2019), which does not require a direct signal from the output layer.

Original: However, GIM uses contrastive loss as the loss function, so GIM can mainly be applied to self-supervised learning tasks.
Revised: However, GIM uses contrastive loss as the loss function, so GIM can mainly be applied to self-supervised learning tasks.

Original: As a result, for supervised learning tasks, AL is likely a more natural choice than GIM.
Revised: As a result, for supervised learning tasks, AL is likely a more natural choice than GIM.

Original: As for synthetic gradients, empirical experiments show that gradients are difficult to predict, so model training based on synthetic gradients usually yields much worse accuracies than those for models trained via backpropagation.
Revised: As for synthetic gradients, empirical experiments show that gradients are difficult to predict, so model training based on synthetic gradients usually yields much worse accuracies than those for models trained via backpropagation.

Original: 4. N EURAL N ETWORKS WITH S TRUCTURES S IMILAR TO AL
Revised: 4. N EURAL N ETWORKS WITH S TRUCTURES S IMILAR TO AL

Original: The structure of an AL-form model may look similar to a ResNet, as the bridge functions in AL appear analogous to the residual connections in ResNet.
Revised: The structure of an AL-form model may look similar to a ResNet, as the bridge functions in AL appear analogous to the residual connections in ResNet.

Original: Therefore, some may argue that AL’s superiority is the result of its resemblance to ResNet.
Revised: Therefore, some may argue that AL’s superiority is the result of its resemblance to ResNet.

Original: However, the residual connections in ResNet are implemented to mitigate accuracy saturation (He et al., 2016), and all the parameters are adjusted to fit a global objective function, whereas the bridge functions in AL are implemented to modularize the network such that the gradient flow in one AL layer does not pass to other layers.
Revised: However, the residual connections in ResNet are implemented to mitigate accuracy saturation (He et al., 2016), and all the parameters are adjusted to fit a global objective function, whereas the bridge functions in AL are implemented to modularize the network such that the gradient flow in one AL layer does not pass to other layers.

Original: Therefore, the bridge functions and the residual connections are only structurally similar; their purposes and functionalities are entirely different.
Revised: Therefore, the bridge functions and the residual connections are only structurally similar; their purposes and functionalities are entirely different.

Original: An AL-form network may also appear similar to contrastive learning (CL).
Revised: An AL-form network may also appear similar to contrastive learning (CL).

Original: Particularly, CL transforms a pair of images i and j (along with their augmented images i ′ and j ′ ) into vectors with large distances between (1) images i and j and (2) augmented images i ′ and j ′ , with small distances between (1) an image i and its augmented image i ′ and (2) an image j and its augmented image j ′ .
Revised: Particularly, CL transforms a pair of images i and j (along with their augmented images i ′ and j ′ ) into vectors with large distances between (1) images i and j and (2) augmented images i ′ and j ′ , with small distances between (1) an image i and its augmented image i ′ and (2) an image j and its augmented image j ′ .

Original: This process is related to AL because both models convert inputs to vectors and compare the distance between the transformed vectors.
Revised: This process is related to AL because both models convert inputs to vectors and compare the distance between the transformed vectors.

Original: Following this line, we are interested in studying the relationship between AL and CL in two ways.
Revised: Following this line, we are interested in studying the relationship between AL and CL in two ways.

Original: First, we could apply AL to CL such that CL is trained more effectively.
Revised: First, we could apply AL to CL such that CL is trained more effectively.

Original: Second, we could extend the AL model such that the inputs include a pair of samples x 1 and x 2 with different labels and design an objective function that maximizes the distance between the vectors transformed from x 1 and x 2 (since their labels are different) and simultaneously minimizes the current AL loss function (Equation 3).
Revised: Second, we could extend the AL model such that the inputs include a pair of samples x 1 and x 2 with different labels and design an objective function that maximizes the distance between the vectors transformed from x 1 and x 2 (since their labels are different) and simultaneously minimizes the current AL loss function (Equation 3).

Original: An AL network is also structurally similar to the models in (Lin et al., 2014;
Revised: An AL network is also structurally similar to the models in (Lin et al., 2014;

Original: Yeh et al., 2017), which also transform x into a latent space and convert it to y via an autoencoder.
Revised: Yeh et al., 2017), which also transform x into a latent space and convert it to y via an autoencoder.

Original: However, the design motivations are very different from AL.
Revised: However, the design motivations are very different from AL.

Original: The two works attempt to efficiently represent a multi-label target y by a low-dimensional latent representation c , so learning a function to convert x to c may be easier.
Revised: The two works attempt to efficiently represent a multi-label target y by a low-dimensional latent representation c , so learning a function to convert x to c may be easier.

Original: The motivation of AL is to design isolated loss functions so that the new design has favorable properties that BP does not have, such as short gradient flows, forward shortcuts, dynamic layer accumulation, and layer-wise pipeline learning.
Revised: The motivation of AL is to design isolated loss functions so that the new design has favorable properties that BP does not have, such as short gradient flows, forward shortcuts, dynamic layer accumulation, and layer-wise pipeline learning.

Original: Isolated objectives are not implemented in (Lin et al., 2014; Yeh et al., 2017).
Revised: Isolated objectives are not implemented in (Lin et al., 2014; Yeh et al., 2017).

Original: Ladder network is another structurally similar network (Rasmus et al., 2015), which adds skip connections between each encoder to its corresponding decoder.
Revised: Ladder network is another structurally similar network (Rasmus et al., 2015), which adds skip connections between each encoder to its corresponding decoder.

Original: While the ladder network may look similar to AL structurally, at least two manifest differences exist.
Revised: While the ladder network may look similar to AL structurally, at least two manifest differences exist.

Original: First, the residual between y and ˆ y in the ladder network is propagated through all layers to update the parameters on the inference path, but in AL, each layer has a local objective function, and most layers do not receive signals from the output layer.
Revised: First, the residual between y and ˆ y in the ladder network is propagated through all layers to update the parameters on the inference path, but in AL, each layer has a local objective function, and most layers do not receive signals from the output layer.

Original: Second, the skip connection in the ladder network connects an encoder layer to a decoder layer.
Revised: Second, the skip connection in the ladder network connects an encoder layer to a decoder layer.

Original: In contrast, the bridge function in AL connects a hidden representation of x to the bottleneck layer of an autoencoder to create the local loss.
Revised: In contrast, the bridge function in AL connects a hidden representation of x to the bottleneck layer of an autoencoder to create the local loss.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper studies associated learning, an alternative methodology to end-to-end backpropagation.
Revised: This paper studies associated learning, an alternative methodology to end-to-end backpropagation.

Original: Since backpropagation is the cornerstone of today’s deep learning but still suffers from optimization and training performance issues, it is appropriate to study alternatives to backpropagation.
Revised: Since backpropagation is the cornerstone of today’s deep learning but still suffers from optimization and training performance issues, it is appropriate to study alternatives to backpropagation.

Original: We discussed the unique properties of AL that BP does not have.
Revised: We discussed the unique properties of AL that BP does not have.

Original: AL provides multiple inference paths; it allows dynamic layer accumulation during training; it supports pipeline training, and, perhaps most importantly, its prediction power is comparable to and frequently better than backpropagation on many of the most successful neural network structures.
Revised: AL provides multiple inference paths; it allows dynamic layer accumulation during training; it supports pipeline training, and, perhaps most importantly, its prediction power is comparable to and frequently better than backpropagation on many of the most successful neural network structures.

Original: Ablation studies disclose why AL may work and demonstrate AL’s effectiveness and architecture flexibility in training and testing.
Revised: Ablation studies disclose why AL may work and demonstrate AL’s effectiveness and architecture flexibility in training and testing.

Original:  A PPENDIX 6. AL I NFERENCE A LGORITHM
Revised:  A PPENDIX 6. AL I NFERENCE A LGORITHM

Original: Algorithm 3 shows the inference algorithm for AL networks.
Revised: Algorithm 3 shows the inference algorithm for AL networks.

Original: It supports forward shortcuts by going through only ℓ of the L AL layers.
Revised: It supports forward shortcuts by going through only ℓ of the L AL layers.

Original: From Algorithm 3, it is clear that the functions g i -s are not used in inference, and only one bridge function out of the L b i -s are used in inference.
Revised: From Algorithm 3, it is clear that the functions g i -s are not used in inference, and only one bridge function out of the L b i -s are used in inference.

Original: Therefore, the number of parameters of inference is smaller than the number of training parameters.
Revised: Therefore, the number of parameters of inference is smaller than the number of training parameters.

Original: Algorithm
Revised: Algorithm

Original: Input AL network
Revised: Input AL network

Original: A =
Revised: A =

Original: ( a 1 , a 2 , ...a L ) , a i = { f i , b i , g i , h i } Li =1 , input data x , inference length ℓ, 1 ≤ ℓ ≤ L
Revised: ( a 1 , a 2 , ...a L ) , a i = { f i , b i , g i , h i } Li =1 , input data x , inference length ℓ, 1 ≤ ℓ ≤ L

Original: Output predicted label 1: s ← x 2: for i = 1 to ℓ do 3: s ← f i ( s ) 4: t ← b ℓ ( s ) 5: t ′ ← t ▷ Same as Figure6: for i =
Revised: Output predicted label 1: s ← x 2: for i = 1 to ℓ do 3: s ← f i ( s ) 4: t ← b ℓ ( s ) 5: t ′ ← t ▷ Same as Figure6: for i =

Original: ℓ to 1 do 7: t ′ ← h i ( t ′ ) 8: return t ′
Revised: ℓ to 1 do 7: t ′ ← h i ( t ′ ) 8: return t ′

Revised: 6. CNN, RNN, AND T RANSFORMER IN THEIR AL FORMS

Revised: Here we extend 2.1.1 for further explanation.

Revised: In AL, the process in bridge layers did not influence the actual forward path in AL.

Revised: Therefore, the process in 2.1.1 is only used to calculate associated loss and will not effect the original representation shape.

Original: 6. I MPLEMENTATION E NVIRONMENT D ETAILS
Revised: Figure 4 and Figure 5 show that when we take the last hidden state in the sequence input to calculate associated loss, the original sequence representation remains the same when propagate to the next layer. 𝑠 ! 𝑥 𝑠 " 𝑤 !

Revised: 𝑤 " 𝑦 𝑡 !

Revised: 𝑡 " 𝑓 ! 𝑓 " 𝑔 ! 𝑔 " 𝑠 "# 𝑠 !

Revised: # 𝑏 !

Revised: 𝑏 " ≈ ≈ unfold 𝑠 !!

Revised: 𝑥 !

Revised: 𝑠 "!

Revised: 𝑢 𝑣 𝑠 !"

Revised: 𝑥 " 𝑠 "" 𝑤 !

Revised: 𝑤 " 𝑤 !

Revised: 𝑤 " … 𝑠 !

Revised: $ 𝑥 $ 𝑠 " $ 𝑤 !

Revised: 𝑤 " 𝑏 ! 𝑏 " 𝑦 𝑡 !

Revised: 𝑡 " 𝑔 ! 𝑔 " 𝑠 "# 𝑠 !

Revised: ≈ ≈

Revised: 𝑢 𝑣 𝑢 𝑣

Revised: 6. I MPLEMENTATION E NVIRONMENT D ETAILS

Original: For LSTM and Transformer, we used the PyTorch package (Paszke et al., 2019).
Revised: For LSTM and Transformer, we used the PyTorch package (Paszke et al., 2019).

Original: For CNN, we used the Tensorflow Abadi et al.
Revised: For CNN, we used the Tensorflow Abadi et al.

Original: (2015) package to build the AL models (vanilla CNN and a visual geometry group (VGG) network).
Revised: (2015) package to build the AL models (vanilla CNN and a visual geometry group (VGG) network).

Original: We trained the models with GeForce 3090 GPUs on Ubuntu
Revised: We trained the models with GeForce 3090 GPUs on Ubuntu

Original: Linux 20.04.
Revised: Linux 20.04.

Original: Since current hardware development does not support actual pipeline training, we simulated pipeline training by blocking gradient flows between AL layers with Tensor.detach() in PyTorch and with tf.stop_gradient() in TensorFlow. 
Revised: Since current hardware development does not support actual pipeline training, we simulated pipeline training by blocking gradient flows between AL layers with Tensor.detach() in PyTorch and with tf.stop_gradient() in TensorFlow. 𝑠 ! 𝑥 𝑠 " 𝑦 𝑡 !

Revised: 𝑡 " 𝑓 ! 𝑓 " 𝑔 ! 𝑔 " 𝑠 "# 𝑠 !

Revised: # 𝑏 !

Revised: 𝑏 " ≈ ≈ unfold 𝑠 !!

Original: 6. D ATASET S TATISTICS
Revised: 𝑥 ! 𝑠 !" 𝑥 " 𝑠 ! $ 𝑥 $ 𝑦 𝑡 !

Revised: 𝑡 " 𝑔 ! 𝑔 "

Revised: Attention

Revised: Attention

Revised: Mean-pooling 𝑠 !

Revised: 𝑠 !

Revised: # 𝑏 !

Revised: ≈ 𝑠 "!

Revised: 𝑠 "" 𝑠 "$ … … …

Revised: Mean-pooling 𝑠 " 𝑠 "# 𝑏 " ≈

Revised: 6. D ATASET S TATISTICS

Original: The IMDB Review dataset (Maas et al., 2011) is a binary sentiment classification dataset containing 25k training data and 25k test data.
Revised: The IMDB Review dataset (Maas et al., 2011) is a binary sentiment classification dataset containing 25k training data and 25k test data.

Original: AG’s News corpus dataset (Zhang et al., 2015) includes four news topics: world, sports, business, and sci/tech.
Revised: AG’s News corpus dataset (Zhang et al., 2015) includes four news topics: world, sports, business, and sci/tech.

Original: It has 120k training data and 7.6k test data.
Revised: It has 120k training data and 7.6k test data.

Original: The DBpedia Ontology classification dataset (Zhang et al., 2016) was constructed by pickingnon-overlapping topic classes from DBpedia 2014, containing 560k training samples and 70k test samples.
Revised: The DBpedia Ontology classification dataset (Zhang et al., 2016) was constructed by pickingnon-overlapping topic classes from DBpedia 2014, containing 560k training samples and 70k test samples.

Original: The Stanford Sentiment Treebank (SST2) dataset (Socher et al., 2013) is a binary sentiment classification dataset from GLUE (Wang et al., 2018).
Revised: The Stanford Sentiment Treebank (SST2) dataset (Socher et al., 2013) is a binary sentiment classification dataset from GLUE (Wang et al., 2018).

Original: The dataset contains 67k training samples, 872 validation samples, and 1.8k test samples.
Revised: The dataset contains 67k training samples, 872 validation samples, and 1.8k test samples.

Original: The CIFAR-10 dataset contains color images from 10 classes; 50k are training samples, and 10k are test samples.
Revised: The CIFAR-10 dataset contains color images from 10 classes; 50k are training samples, and 10k are test samples.

Original: Finally, the Fashion-MNIST dataset (Xiao et al., 2017) is a grayscale image classification dataset with 10 classes.
Revised: Finally, the Fashion-MNIST dataset (Xiao et al., 2017) is a grayscale image classification dataset with 10 classes.

Original: It consists of 60k training samples and 10k test samples.
Revised: It consists of 60k training samples and 10k test samples.

Original: 6. T EXT C LASSIFICATION
Revised: 6. T EXT C LASSIFICATION

Original: During training, the most frequent 30k words in the training corpus were included in the word embedding.
Revised: During training, the most frequent 30k words in the training corpus were included in the word embedding.

Original: We replaced the remaining words with the [UNK] token.
Revised: We replaced the remaining words with the [UNK] token.

Original: We trained all text classification experiments with 50 epochs.
Revised: We trained all text classification experiments with 50 epochs.

Original: We used the Adam optimizer.
Revised: We used the Adam optimizer.

Original: 6.4. B I -LSTM
Revised: 6.5. B I -LSTM

Original: For Bi-LSTM, we applied gradient clipping with the value 5.
Revised: For Bi-LSTM, we applied gradient clipping with the value 5.

Original: For the standard version of Bi-LSTM, we experimented with learning rates of 0.0005 and 0.001 and reported that with better accuracy: we used 0.0005 for AG’s news and IMDB, and 0.001 for DBpedia and SST.
Revised: For the standard version of Bi-LSTM, we experimented with learning rates of 0.0005 and 0.0001 and reported that with better accuracy.

Original: We also adjusted the learning rate for AL, but the learning rate has little influence on the accuracy and the convergence speed.
Revised: We also adjusted the learning rate for AL, but the learning rate has little influence on the accuracy and the convergence speed.

Original: We used a batch size of 64 for all Bi-LSTM experiments.
Revised: We used a batch size of 64 for all Bi-LSTM experiments.

Original: For IMDB and DBpedia, we set the max sequence length to 500.
Revised: For IMDB and DBpedia, we set the max sequence length to 500.

Original: For AG’s News and SST, we set the max sequence length to the longest sequence length; the exact length settings were applied to the Transformer experiments.
Revised: For AG’s News and SST, we set the max sequence length to the longest sequence length; the exact length settings were applied to the Transformer experiments.

Original: The objective of the standard Bi-LSTM was a cross-entropy loss.
Revised: The objective of the standard Bi-LSTM was a cross-entropy loss.

Original: Details are given in Table 7.
Revised: Details are given in Table 7.

Original: [Table]
Revised: [Table]

Original: 6.4. T RANSFORMER
Revised: 6.5. T RANSFORMER

Original: For the standard Transformer model, we used negative log-likelihood as the objective with multihead attention (6 heads).
Revised: For the standard Transformer model, we used negative log-likelihood as the objective with multihead attention (6 heads).

Original: We set the batch size to 128 because the standard Transformer converges only with a large batch size in our experiment.
Revised: We set the batch size to 128 because the standard Transformer converges only with a large batch size in our experiment.

Original: Table 8 gives the details.
Revised: Table 8 gives the details.

Original: [Table]
Revised: [Table]

Original: 6. I MAGE C LASSIFICATION
Revised: 6. I MAGE C LASSIFICATION

Original: We normalized the image pixels for both FashionMNIST and CIFAR-10 datasets as a preprocessing step.
Revised: We normalized the image pixels for both FashionMNIST and CIFAR-10 datasets as a preprocessing step.

Original: Since CIFAR-10 is more challenging, we also performed data augmentation on this dataset by resizing, random cropping, random flipping, and adjusting the brightness of the images.
Revised: Since CIFAR-10 is more challenging, we also performed data augmentation on this dataset by resizing, random cropping, random flipping, and adjusting the brightness of the images.

Original: We set the batch size to 128.
Revised: We set the batch size to 128.

Original: We set the initial learning rate to 0.0001 and applied scheduled decay until converging.
Revised: We set the initial learning rate to 0.0001 and applied scheduled decay until converging.

Original: We used Adam as the optimizer and cross-entropy loss for the standard version.
Revised: We used Adam as the optimizer and cross-entropy loss for the standard version.

Original: Vanilla CNN Model We implemented the vanilla CNN by modifying the previous code.
Revised: Vanilla CNN Model We implemented the vanilla CNN by modifying the previous code.

Original: The
Revised: The

Original: CNN contains 13 hidden layers and an output layer.
Revised: CNN contains 13 hidden layers and an output layer.

Original: The first 8 hidden layers are convolution layers; the last is flattened to a fully connected layer with 1280 neurons.
Revised: The first 8 hidden layers are convolution layers; the last is flattened to a fully connected layer with 1280 neurons.

Original: The subsequent layers consist offeedforward layers, followed by an output layer with 10 classes.
Revised: The subsequent layers consist offeedforward layers, followed by an output layer with 10 classes.

Original: We set the initial learning rate to 0.0001 and applied scheduled decay.
Revised: We set the initial learning rate to 0.0001 and applied scheduled decay.

Original: [Figure]
Revised: [Figure]

Original: 6. T RANSFORMER -AL CONVERGENCE SPEED
Revised: 6. T RANSFORMER -AL CONVERGENCE SPEED

Original: Training an LSTM, a CNN, or a VGG based on AL requires much fewer epochs when compared with training them by BP; this trend is less clear when training a Transformer-AL model.
Revised: Training an LSTM, a CNN, or a VGG based on AL requires much fewer epochs when compared with training them by BP; this trend is less clear when training a Transformer-AL model.

Original: However, we found that after training the Transformer-AL for roughly 5 epochs, the validation set accuracy (90.95) can already achieve 99% of the best validation set accuracy (91.31), and the test accuracy (90.98) is very close to the best test accuracy (91.17, i.e., training Transformer-AL for 20 epochs for
Revised: However, we found that after training the Transformer-AL for roughly 5 epochs, the validation set accuracy (90.95) can already achieve 99% of the best validation set accuracy (91.31), and the test accuracy (90.98) is very close to the best test accuracy (91.17, i.e., training Transformer-AL for 20 epochs for

Original: AG News).
Revised: AG News).

Original: So, we could reduce the number of epochs by sacrificing slight accuracy.
Revised: So, we could reduce the number of epochs by sacrificing slight accuracy.

Original: TransformerAL on DBpedia has a similar trend: when training Transformer-AL on DBpedia for 5 epochs, its validation set accuracy achieves 96.0902, which is very close to the best validation set accuracy (96.6134).
Revised: TransformerAL on DBpedia has a similar trend: when training Transformer-AL on DBpedia for 5 epochs, its validation set accuracy achieves 96.0902, which is very close to the best validation set accuracy (96.6134).

Original: This result shows that Transformer-AL is efficient on convergence when training on
Revised: This result shows that Transformer-AL is efficient on convergence when training on

Original: DBpedia and AGnews.
Revised: DBpedia and AGnews.

Original: 6. A SSOCIATED L OSS VS . L AYER D EPTH
Revised: 6. A SSOCIATED L OSS VS . L AYER D EPTH

Original: This section shows that, under the AL network structure, the latent representations in the lower layer appear to help generate the latent representations in an upper layer.
Revised: This section shows that, under the AL network structure, the latent representations in the lower layer appear to help generate the latent representations in an upper layer.

Original: We experimented with the AL layer number i vs. the associated loss ∥
Revised: We experimented with the AL layer number i vs. the associated loss ∥

Original: s
Revised: s

Original: i − t i ∥ 22 using different datasets on different network structures.
Revised: i − t i ∥ 22 using different datasets on different network structures.

Original: The following tables show that the associated loss at an upper layer (i.e., the layers closer to the output) is smaller than at a lower layer (i.e., the layers closer to the input).
Revised: The following tables show that the associated loss at an upper layer (i [Table] e., the layers closer to the output) is smaller than at a lower layer (i

Revised: [Table]

Revised: e., the layers closer to the input).

Original: Consequently, the shortcuts are likely doing some kind of curriculum learning. 
Revised: Consequently, the shortcuts are likely doing some kind of curriculum learning. [Table] [Table] [Table]

Revised: T HE D ESIGN OF THE O VERCOMPLETE A UTOENCODERS IN AL

Revised: An autoencoder’s hidden layer usually consists of fewer neurons than the input and output layers.

Original: Layer
Revised: However, we use an overcomplete autoencoder (i.e., the hidden layer has more neurons than the input/output layers) (Vincent et al., 2010) in AL. An overcomplete autoencoder can reach zero reconstruction loss (i.e., the autoencoder loss in Equation 3) by copying the neurons from the input layer to the output layer. However, in AL, g i (the encoding function of the autoencoder at layer i ) needs to minimize both L iAE the autoencoder loss and L iA the associated loss. As a result, L iA can be regarded as a regularization term when minimizing L iAE .

Revised: Additionally, we hypothesize that mapping the one-hot encoded y to a high dimensional vector can accelerate the convergence speed.

Original: [Table]
Revised: According to Graf et al. (2021), the cross-entropy loss and supervised contrastive loss attain their minimum if and only if the features of each class collapse to the vertices of an origin-centered regular K − 1 simplex, where K is the number of classes. In AL, since the autoencoder in layer-1 simply maps y to a high dimensional space and back (i.e., letting h 1 ( g 1 ( y ))

Original: [Table]
Revised: ≈ y ), it is nearly guaranteed to reach minimum loss. Therefore, we can consider g 1 ( y ) as an ideal prototype for AL to fit in, such that, b 1 ( f 1 ( x )) ≈ g 1 ( y ) . Therefore, the associated loss directly optimizes for a K − 1 regular simplex in the representation space, which is the best configuration to minimize the supervised contrastive learning loss (Khosla et al., 2020).

Revised: Such an autoencoder setup probably helps AL find the best configuration in the early stage of training.

Revised: This may explain why AL converges so fast.

Revised: Besides, an overparameterized model may accelerate optimization (Arora et al., 2018; Chen & Chen, 2020).

Revised: Empirical studies in Section 3.5 support our claim: using overcomplete autoencoders to replace classifiers performs better.

Original: 6. P Y T ORCH P SEUDOCODE
Revised: 6. P Y T ORCH P SEUDOCODE

Original: Here we provide a PyTorch pseudocode of a 5-layer AL network.
Revised: Here we provide a PyTorch pseudocode of a 5-layer AL network.

Original: import torch import torch.nn as nn class MLP (nn.
Revised: import torch import torch.nn as nn class MLP (nn.

Original: Module): def __init__(self, d1, d2): super().__init__() self.f
Revised: Module): def __init__(self, d1, d2): super().__init__() self.f

Original: =nn.Linear(d1,d2) self.act=nn.
Revised: =nn. Sequential( nn.Linear(d1,d2), nn.

Original: Sigmoid() def forward(self, x): return self.act(self.f(x)) class AL (nn.
Revised: Sigmoid()) def forward(self, x): return self.f(x) class AL (nn.

Original: Module): def __init__(self, class_num): super().__init__()  self.f
Revised: Module): def __init__(self, class_num): super().__init__() # we choose 10 as hidden dimension size for simplicity. self.f

Original: = nn.
Revised: = nn.

Original: ModuleList([MLP(10, 10) for i in range(5)]) self.g
Revised: ModuleList([MLP(10, 10) for i in range(5)]) self.g

Original: = nn.
Revised: = nn.

Original: ModuleList([MLP(10, 10) for i in range(5)])
Revised: ModuleList([MLP(10, 10) for i in range(5)])

Original: self.b = nn.
Revised: self.b = nn.

Original: ModuleList([MLP(10, 10) for i in range(5)]) self.h
Revised: ModuleList([MLP(10, 10) for i in range(5)]) self.h

Original: = nn.
Revised: = nn.

Original: ModuleList([MLP(10, 10) for i in range(5)]) self.g[0], self.h[0] =
Revised: ModuleList([MLP(10, 10) for i in range(5)]) self.g[0], self.h[0] =

Original: MLP(class_num, 10), MLP(10, class_num)
Revised: MLP(class_num, 10), MLP(10, class_num)

Original: self.mse = nn.
Revised: self.mse, self.ce = nn.

Original: MSELoss() self.c =
Revised: MSELoss(), nn. CrossEntropyLoss() self.c =

Original: class_num def forward(self, x, y): loss = 0.
Revised: class_num def forward(self, x, y): loss = 0.

Original: s_i, t_i = x, nn.functional.one_hot(y, num_classes=self.c).float()
Revised: s_i, t_i = x, nn.functional.one_hot(y, num_classes=self.c).float()

Original: # convert y into one-hot vector for linear input for i in range(5):
Revised: # convert y into one-hot vector for linear input for i in range(5):

Original: last_t_i =
Revised: last_t_i, s_i

Original: t_i s_i =  self.f[i](s_i)
Revised: = t_i, self.f[i](s_i)

Original: # encode input  s_i_prime = self.b[i](s_i)
Revised: # encode input t_i, s_i_prime = self.b[i](s_i), self.g[i](t i)

Original: # bridge 
Revised: # bridge + encode target t

Original: t_i = self.g[i](t_i)
Revised: i .h[i i)

Original: # encode target t_i_prime = self.h[i](t_i)
Revised: # decode target if i = = 0:

Original: # decode target loss += self.mse(t_i.detach(), s_i_prime) + self.mse(last_t_i, t_i_prime)
Revised: = .mse i.detach( s i prime) self.ce(last_t_i, t_i_prime) else : loss += self.mse(t i.detach(), s_i_prime) + self.mse(last_t_i, t_i_prime)

Original: s_i, t_i = s_i.detach(), t_i.detach() # cut gradient return loss def inference(self, x): for layer in self.f:x = layer(x) y = self.b[-1](x) for layer in reversed(self.h):y = layer(y) return y model =
Revised: s_i, t_i = s_i.detach(), t_i.detach() # cut gradient return loss def inference(self, x): for layer in self.f: for layer in reversed(self.h): return y model =

Original: AL(class_num=5) x, y = torch.rand(30, 10, dtype=torch.float), torch.randint(high=4, size=(30, )) out, pred = model(x, y), model.inference(x)
Revised: AL(class_num=5) [Table]