Original: L EARNING THE C ONNECTIONS IN D IRECT F EEDBACK A LIGNMENT
Revised: L EARNING THE C ONNECTIONS IN D IRECT F EEDBACK A LIGNMENT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Feedback alignment was proposed to address the biological implausibility of the backpropagation algorithm which requires the transportation of the weight transpose during the backwards pass.
Revised: Feedback alignment was proposed to address the biological implausibility of the backpropagation algorithm which requires the transportation of the weight transpose during the backwards pass.

Original: The idea was later built upon with the proposal of direct feedback alignment (DFA), which propagates the error directly from the output layer to each hidden layer in the backward path using a ﬁxed random weight matrix.
Revised: The idea was later built upon with the proposal of direct feedback alignment (DFA), which propagates the error directly from the output layer to each hidden layer in the backward path using a ﬁxed random weight matrix.

Original: This contribution was signiﬁcant because it allowed for the parallelization of the backwards pass by the use of these feedback connections.
Revised: This contribution was signiﬁcant because it allowed for the parallelization of the backwards pass by the use of these feedback connections.

Original: However, just as feedback alignment, DFA does not perform well in deep convolutional networks.
Revised: However, just as feedback alignment, DFA does not perform well in deep convolutional networks.

Original: We propose to learn the backward weight matrices in DFA, adopting the methodology of Kolen-Pollack learning , to improve training and inference accuracy in deep convolutional neural networks by updating the direct feedback connections such that they come to estimate the forward path.
Revised: We propose to learn the backward weight matrices in DFA, adopting the methodology of Kolen-Pollack learning , to improve training and inference accuracy in deep convolutional neural networks by updating the direct feedback connections such that they come to estimate the forward path.

Original: The proposed method improves the accuracy of learning by direct feedback connections and reduces the gap between parallel training to serial training by means of backpropagation.
Revised: The proposed method improves the accuracy of learning by direct feedback connections and reduces the gap between parallel training to serial training by means of backpropagation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: When feedback alignment was proposed by Lillicrap et al.
Revised: When feedback alignment was proposed by Lillicrap et al.

Original: it was cited as being a biologically plausible alternative to the backpropagation algorithm, but not long after Nøkland (2016) showed that variants of this approach may show tangible beneﬁts during training such as mitigating the vanishing gradients issue or enabling parallelization of the backwards pass at the cost of additional memory requirements.
Revised: it was cited as being a biologically plausible alternative to the backpropagation algorithm, but not long after Nøkland (2016) showed that variants of this approach may show tangible beneﬁts during training such as mitigating the vanishing gradients issue or enabling parallelization of the backwards pass at the cost of additional memory requirements.

Original: Recently, interest in the latter has begun to grow as the memory capacity and compute capability of modern GPUs has continued to observe signiﬁcant leaps.
Revised: Recently, interest in the latter has begun to grow as the memory capacity and compute capability of modern GPUs has continued to observe signiﬁcant leaps.

Original: While many of these recently proposed alternatives have been shown to be just as capable as the backpropagation algorithm in terms of inference accuracy on deep convolutional networks, it should be noted that many of these approaches have not yet been shown to perform well outside of the image classiﬁcation task.
Revised: While many of these recently proposed alternatives have been shown to be just as capable as the backpropagation algorithm in terms of inference accuracy on deep convolutional networks, it should be noted that many of these approaches have not yet been shown to perform well outside of the image classiﬁcation task.

Original: Direct feedback alignment (DFA), an earlier approach proposed by Nøkland (2016), was shown to perform reasonably well on a number of natural language processing tasks with recurrent neural networks and transformers by Launay et al.
Revised: Direct feedback alignment (DFA), an earlier approach proposed by Nøkland (2016), was shown to perform reasonably well on a number of natural language processing tasks with recurrent neural networks and transformers by Launay et al.

Original: However, direct feedback alignment still shows poor performance on the image classiﬁcation task due to its inability to effectively train convolutional layers.
Revised: However, direct feedback alignment still shows poor performance on the image classiﬁcation task due to its inability to effectively train convolutional layers.

Original: We propose a modiﬁcation to the DFA algorithm to improve its ability in training deep convolutional neural networks.
Revised: We propose a modiﬁcation to the DFA algorithm to improve its ability in training deep convolutional neural networks.

Original: Due to its relationship with another approach(Akrout et al., 2019), we call our method Direct Kolen-Pollack learning or DKP.
Revised: Due to its relationship with another approach(Akrout et al., 2019), we call our method Direct Kolen-Pollack learning or DKP.

Original: We empirically show the mechanisms that allow the improvement in our approach over DFA by measuring DKP’s ability to better estimate the backpropagation algorithm.
Revised: We empirically show the mechanisms that allow the improvement in our approach over DFA by measuring DKP’s ability to better estimate the backpropagation algorithm.

Original: We also show this improvement directly by training two deep convolutional neural network architectures on the Fashion-MNIST, CIFAR10, CIFAR100, and TinyImageNet200(Le & Yang, 2015) datasets.
Revised: We also show this improvement directly by training two deep convolutional neural network architectures on the Fashion-MNIST, CIFAR10, CIFAR100, and TinyImageNet200(Le & Yang, 2015) datasets.

Original: More so, we recommend training procedures for training with DFA, pointing out the important role batch normalization plays in our experiments.
Revised: More so, we recommend training procedures for training with DFA, pointing out the important role batch normalization plays in our experiments.

Original: And while a couple of works have shown that direct feedback connections can be viable when connecting to only the output of a block of layers in a network (Ororbia et al., 2020; Han & Yoo, 2019), we show advances in the case of having feedback connections to all layers in deep convolutional neural networks.
Revised: And while a couple of works have shown that direct feedback connections can be viable when connecting to only the output of a block of layers in a network (Ororbia et al., 2020; Han & Yoo, 2019), we show advances in the case of having feedback connections to all layers in deep convolutional neural networks.

Original: While direct feedback connections to all layers for current PC hardware, and also from a software perspective, may not be practical, in the future it may be useful for edge devices, IoT, SOC design, etc .(Frenkel et al., 2019; Han & Yoo, 2019), especially those that involve learning vision tasks.
Revised: While direct feedback connections to all layers for current PC hardware, and also from a software perspective, may not be practical, in the future it may be useful for edge devices, IoT, SOC design, etc .(Frenkel et al., 2019; Han & Yoo, 2019), especially those that involve learning vision tasks.

Original: Thus, making advances in the training scenario of direct feedback connections to all layers in a neural network at a minimal computational cost for vision tasks is a primary motivation of this work.
Revised: Thus, making advances in the training scenario of direct feedback connections to all layers in a neural network at a minimal computational cost for vision tasks is a primary motivation of this work.

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: Alternatives to the backpropagation algorithm have been proposed for their heightened biologically plausibility, or often as a means of parallelizing the training process.
Revised: Alternatives to the backpropagation algorithm have been proposed for their heightened biologically plausibility, or often as a means of parallelizing the training process.

Original: More recently, a number of algorithms have shown impressive results on large classiﬁcation datasets such as ImageNet(Akrout et al., 2019;
Revised: More recently, a number of algorithms have shown impressive results on large classiﬁcation datasets such as ImageNet(Akrout et al., 2019;

Original: Kunin et al., 2020; Belilovsky et al., 2019; Xu et al., 2020).
Revised: Kunin et al., 2020; Belilovsky et al., 2019; Xu et al., 2020).

Original: To enable further parallelization of the training process, these works often focus on tackling three major deﬁciencies with the backpropagtion algorithm: forward locking , backward locking , and update locking .
Revised: To enable further parallelization of the training process, these works often focus on tackling three major deﬁciencies with the backpropagtion algorithm: forward locking , backward locking , and update locking .

Original: Forward locking prevents any calculation of gradients until the forward pass has been completed.
Revised: Forward locking prevents any calculation of gradients until the forward pass has been completed.

Original: Backward locking means that the gradients at some layer can not be calculated until the learning signals at all of the downstream layers have been calculated ﬁrst.
Revised: Backward locking means that the gradients at some layer can not be calculated until the learning signals at all of the downstream layers have been calculated ﬁrst.

Original: Update locking means that the parameters at some layer cannot be updated before the learning signal at the layer upstream of it has been calculated.
Revised: Update locking means that the parameters at some layer cannot be updated before the learning signal at the layer upstream of it has been calculated.

Original: Difference Target Propagation (DTP), proposed by Lee et al. (2015), is one such alternative to the backpropagation algorithm that instead of computing gradients at each layer computes targets that are propagated backwards through the network by means of layer-wise autoencoders.
Revised: Difference Target Propagation (DTP), proposed by Lee et al. (2015), is one such alternative to the backpropagation algorithm that instead of computing gradients at each layer computes targets that are propagated backwards through the network by means of layer-wise autoencoders.

Original: In a recent paper by Lillicrap et al.
Revised: In a recent paper by Lillicrap et al.

Original: (2020), DTP and methods that use layer-wise autoencoders in the backward path to propagate gradients are claimed to be more biologically plausible alternative to backpropagation and help to explain how biological neural networks might learn using a process similar to the backpropagation algorithm.
Revised: (2020), DTP and methods that use layer-wise autoencoders in the backward path to propagate gradients are claimed to be more biologically plausible alternative to backpropagation and help to explain how biological neural networks might learn using a process similar to the backpropagation algorithm.

Original: Around the time DTP was ﬁrst proposed, Lillicrap et al.
Revised: Around the time DTP was ﬁrst proposed, Lillicrap et al.

Original: (2016) demonstrated that artiﬁcial neural networks can learn using so-called feedback connections that are inspired by the biological feedback connections in the brain, and did so by using ﬁxed random weight matrices in place of the weight transpose when calculating each learning signal during the backwards pass.
Revised: (2016) demonstrated that artiﬁcial neural networks can learn using so-called feedback connections that are inspired by the biological feedback connections in the brain, and did so by using ﬁxed random weight matrices in place of the weight transpose when calculating each learning signal during the backwards pass.

Original: This approach, referred to as feedback alignment (FA), was claimed by the authors to be more biologically plausible than backpropagation as it addressed the implausibility of weight transportation in biological neural networks.
Revised: This approach, referred to as feedback alignment (FA), was claimed by the authors to be more biologically plausible than backpropagation as it addressed the implausibility of weight transportation in biological neural networks.

Original: FA and its derivatives would be further evaluated by Bartunov et al.
Revised: FA and its derivatives would be further evaluated by Bartunov et al.

Original: and be shown to be very limited in comparison to the backpropagation algorithm on difﬁcult tasks such as ImageNet.
Revised: and be shown to be very limited in comparison to the backpropagation algorithm on difﬁcult tasks such as ImageNet.

Original: However, Moskovitz et al.
Revised: However, Moskovitz et al.

Original: (2019) would concurrently propose their own variations of the feedback alignment algorithm and make great strides in bringing these biologically motivated algorithms closer to the performance of backpropagation in deep convolutional neural networks.
Revised: (2019) would concurrently propose their own variations of the feedback alignment algorithm and make great strides in bringing these biologically motivated algorithms closer to the performance of backpropagation in deep convolutional neural networks.

Original: Following this initial work on feedback alignment, Nøkland (2016) proposed an alternative approach that connected each layer directly to the error through a ﬁxed random weight matrix in the backward path.
Revised: Following this initial work on feedback alignment, Nøkland (2016) proposed an alternative approach that connected each layer directly to the error through a ﬁxed random weight matrix in the backward path.

Original: Called direct feedback alignment (DFA), this contribution was signiﬁcant as it leveraged feedback alignment to enable backwards unlocking meaning that during training the gradients for each layer can be calculated in parallel.
Revised: Called direct feedback alignment (DFA), this contribution was signiﬁcant as it leveraged feedback alignment to enable backwards unlocking meaning that during training the gradients for each layer can be calculated in parallel.

Original: Unfortunately, just as the original feedback alignment method, direct feedback alignment has difﬁculty scaling to more difﬁcult problems and training convolutional layers.
Revised: Unfortunately, just as the original feedback alignment method, direct feedback alignment has difﬁculty scaling to more difﬁcult problems and training convolutional layers.

Original: In a follow up paper on DFA, Launay et al.
Revised: In a follow up paper on DFA, Launay et al.

Original: (2019) showed that the approach simply failed to train convolutional layers.
Revised: (2019) showed that the approach simply failed to train convolutional layers.

Original: Later, Han & Yoo (2019) showed that
Revised: Later, Han & Yoo (2019) showed that

Original: VGG-16 could be trained with DFA if only the full connected layers are trained with DFA while the convolutional layers are trained with backpropagation, and Han et al.
Revised: VGG-16 could be trained with DFA if only the full connected layers are trained with DFA while the convolutional layers are trained with backpropagation, and Han et al.

Original: (2020) later showed that by only having direct connections to speciﬁc layers better performance in accuracy over DFA while training convolutional networks on the CIFAR10 dataset could be made.
Revised: (2020) later showed that by only having direct connections to speciﬁc layers better performance in accuracy over DFA while training convolutional networks on the CIFAR10 dataset could be made.

Original: Despite this shortcoming,
Revised: Despite this shortcoming,

Original: DFA shows fairly strong performance on various NLP tasks as shown by (Launay et al., 2020), and been used to enable higher power efﬁciency in SOC design (Han et al., 2019).
Revised: DFA shows fairly strong performance on various NLP tasks as shown by (Launay et al., 2020), and been used to enable higher power efﬁciency in SOC design (Han et al., 2019).

Original: Other follow up works to DFA helped to reduce the additional memory costs of DFA(Han et al., 2019; Crafton et al., 2019), and Frenkel et al.
Revised: Other follow up works to DFA helped to reduce the additional memory costs of DFA(Han et al., 2019; Crafton et al., 2019), and Frenkel et al.

Original: (2019) even showed that propagating targets in place of the gradient at the output can be just as effective.
Revised: (2019) even showed that propagating targets in place of the gradient at the output can be just as effective.

Original: Further recLRA, proposed by Ororbia et al.
Revised: Further recLRA, proposed by Ororbia et al.

Original: (2020), showed strong performance on the ResNet architectures with its own biologically inspired derivation of DFA that, similarly to our proposed approach, updates the backward feedback connections, but this performance was achieved by the more practical method of only have direct feedback connections to some layers.
Revised: (2020), showed strong performance on the ResNet architectures with its own biologically inspired derivation of DFA that, similarly to our proposed approach, updates the backward feedback connections, but this performance was achieved by the more practical method of only have direct feedback connections to some layers.

Original: More recently, Akrout et al.
Revised: More recently, Akrout et al.

Original: (2019) and Kunin et al.
Revised: (2019) and Kunin et al.

Original: (2020) have shown that credit assignment approaches similar to FA can scale to larger problems by training the backward weights and even come close to matching the performance of backpropagation on the ImageNet classiﬁcation task.
Revised: (2020) have shown that credit assignment approaches similar to FA can scale to larger problems by training the backward weights and even come close to matching the performance of backpropagation on the ImageNet classiﬁcation task.

Original: Akrout et al.
Revised: Akrout et al.

Original: (2019) proposed weight mirroring (WM) which trained the backward weights to mirror their forward counterparts using the transposing rule and proposed another method, referred to as Kolen-Pollack learning, based on the research of Kolen & Pollack (1994), that updates the backward matrices with the same gradient as the forward weights and uses weight decay on both the forward and backward matrices to encourage symmetry between the two.
Revised: (2019) proposed weight mirroring (WM) which trained the backward weights to mirror their forward counterparts using the transposing rule and proposed another method, referred to as Kolen-Pollack learning, based on the research of Kolen & Pollack (1994), that updates the backward matrices with the same gradient as the forward weights and uses weight decay on both the forward and backward matrices to encourage symmetry between the two.

Original: As a follow up to this,
Revised: As a follow up to this,

Original: Kunin et al.
Revised: Kunin et al.

Original: (2020) proposed a set of regularization primitives with which to update the backward weights and combined these primitives into various conﬁgurations that showed improved performance and stability over WM.
Revised: (2020) proposed a set of regularization primitives with which to update the backward weights and combined these primitives into various conﬁgurations that showed improved performance and stability over WM.

Original: One such conﬁguration, information alignment (IA), uses only local learning rules that follow what the authors consider key biological constraints to train various
Revised: One such conﬁguration, information alignment (IA), uses only local learning rules that follow what the authors consider key biological constraints to train various

Original: deep
Revised: deep

Original: ResNet architectures on the ImageNet data set.
Revised: ResNet architectures on the ImageNet data set.

Original: Since the introduction of DFA, a number of notable works have proposed their own unique approaches for enabling the parallelization of the forward and backward passes during training.
Revised: Since the introduction of DFA, a number of notable works have proposed their own unique approaches for enabling the parallelization of the forward and backward passes during training.

Original: Among these approaches, one of the most notable advances was made by Jaderberg et al.
Revised: Among these approaches, one of the most notable advances was made by Jaderberg et al.

Original: (2017) in their work on decoupled neural interfaces (DNI) which trains modules at each layer to predict synthetic gradients given the layer’s input and some context information, such as the global target, as the input into the module, and then waiting for the true gradients in order to update each module’s parameters.
Revised: (2017) in their work on decoupled neural interfaces (DNI) which trains modules at each layer to predict synthetic gradients given the layer’s input and some context information, such as the global target, as the input into the module, and then waiting for the true gradients in order to update each module’s parameters.

Original: Nøkland & Eidnes (2019) used sub-networks at each layer to produce local loss functions with which to update the weights of the primary network.
Revised: Nøkland & Eidnes (2019) used sub-networks at each layer to produce local loss functions with which to update the weights of the primary network.

Original: They also showed that a backprop free version of this approach can train with only a minor loss in accuracy.
Revised: They also showed that a backprop free version of this approach can train with only a minor loss in accuracy.

Original: More recently, based on the earlier greedy layer-wise learning (DGL) method(Bengio et al., 2007), Belilovsky et al.
Revised: More recently, based on the earlier greedy layer-wise learning (DGL) method(Bengio et al., 2007), Belilovsky et al.

Original: (2019) proposed synchronous and asynchronous variants of DGL for addressing the issues of update locking and forward locking, and showed that these variants can match the performance of DGL and are capable of training deep convolutional neural networks on the ImageNet classiﬁcation task.
Revised: (2019) proposed synchronous and asynchronous variants of DGL for addressing the issues of update locking and forward locking, and showed that these variants can match the performance of DGL and are capable of training deep convolutional neural networks on the ImageNet classiﬁcation task.

Original: In the same vein as prior work which decoupled the backpropagation algorithm using delayed gradients(Huo et al., 2018), Xu et al.
Revised: In the same vein as prior work which decoupled the backpropagation algorithm using delayed gradients(Huo et al., 2018), Xu et al.

Original: (2020) also recently introduced another novel training method, diversely stale parameters (DSP), which achieved comparable results to backpropagation on the ImageNet task.
Revised: (2020) also recently introduced another novel training method, diversely stale parameters (DSP), which achieved comparable results to backpropagation on the ImageNet task.

Original:  A PPROACH 2. B ACKPROPAGATION AND D IRECT F EEDBACK A LIGNMENT
Revised:  A PPROACH 2. B ACKPROPAGATION AND D IRECT F EEDBACK A LIGNMENT

Original: The back-propagation algorithm has long proven to be a robust, well performing approach for credit assignment in artiﬁcial neural networks.
Revised: The back-propagation algorithm has long proven to be a robust, well performing approach for credit assignment in artiﬁcial neural networks.

Original: However, in recent years, there has been a search for alternatives that overcome the constraints of backwards and forwards locking which are found in backpropagation.
Revised: However, in recent years, there has been a search for alternatives that overcome the constraints of backwards and forwards locking which are found in backpropagation.

Original: One such alternative is direct feedback alignment (DFA), a bio-inspired credit assignment algorithm that enables backwards unlocking and the foundational approach for the method we are proposing.
Revised: One such alternative is direct feedback alignment (DFA), a bio-inspired credit assignment algorithm that enables backwards unlocking and the foundational approach for the method we are proposing.

Original: For a simple comparison of BP to DFA and our approach, we deﬁne a simple feed-forward neural network with N layers, ignoring the biases.
Revised: For a simple comparison of BP to DFA and our approach, we deﬁne a simple feed-forward neural network with N layers, ignoring the biases.

Original: Then the learning signal δ k for the output layer k and the learning signal δ (cid:96) for some layer (cid:96) <
Revised: Then the learning signal δ k for the output layer k and the learning signal δ (cid:96) for some layer (cid:96) <

Original: k as prescribed by the backpropagation algorithm, where a (cid:96) are our activations and f (cid:48)
Revised: k as prescribed by the backpropagation algorithm, where a (cid:96) are our activations and f (cid:48)

Original: the derivative of some non-linearity such as a sigmoid function, are as follows.
Revised: the derivative of some non-linearity such as a sigmoid function, are as follows.

Original: [Equation]
Revised: [Equation]

Original: In the equations above and from here on out, · is the dot product operation between two matrices, and (cid:12) is the Hadamard product, or also known as the element-wise product.
Revised: In the equations above and from here on out, · is the dot product operation between two matrices, and (cid:12) is the Hadamard product, or also known as the element-wise product.

Original: In DFA, B (cid:96) is a ﬁxed random weight matrix that projects the gradient δ k at the output of a network to the output of layer (cid:96) − 1 .
Revised: In DFA, B (cid:96) is a ﬁxed random weight matrix that projects the gradient δ k at the output of a network to the output of layer (cid:96) − 1 .

Original: Thus, the learning signal δ (cid:96) for any layer (cid:96) upstream of output layer can be computed in the following way where a (cid:96) are the activations and f (cid:48)
Revised: Thus, the learning signal δ (cid:96) for any layer (cid:96) upstream of output layer can be computed in the following way where a (cid:96) are the activations and f (cid:48)

Original: the derivative of the non-linearity used at layer (cid:96) .
Revised: the derivative of the non-linearity used at layer (cid:96) .

Original: [Equation]
Revised: [Equation]

Original: Then, every other aspect of the network is calculated in the same way as backpropagation.
Revised: Then, every other aspect of the network is calculated in the same way as backpropagation.

Original: Thus, the weights W (cid:96) at layer (cid:96) would be updated in the following way, where η W is the learning rate for the forward parameters, just as it is with backpropagation.
Revised: Thus, the weights W (cid:96) at layer (cid:96) would be updated in the following way, where η W is the learning rate for the forward parameters, just as it is with backpropagation.

Original: [Equation]
Revised: [Equation]

Original: Because the backward weights are ﬁxed, it is thought that the forward weights are aligning themselves with the backward weights such that the backward weights become useful for providing
Revised: Because the backward weights are ﬁxed, it is thought that the forward weights are aligning themselves with the backward weights such that the backward weights become useful for providing

Original: [Figure]
Revised: [Figure]

Original: meaningful updates to the network during training.
Revised: meaningful updates to the network during training.

Original: However, Launay et al.
Revised: However, Launay et al.

Original: (2019) showed that
Revised: (2019) showed that

Original: DFA, while showing results that nearly match backpropagation on the MNIST dataset with fully connected (FC) layers, fails to scale to harder problems such as CIFAR100 and shows signiﬁcantly worse performance while training convolutional neural networks.
Revised: DFA, while showing results that nearly match backpropagation on the MNIST dataset with fully connected (FC) layers, fails to scale to harder problems such as CIFAR100 and shows signiﬁcantly worse performance while training convolutional neural networks.

Original: 2. K OLEN -P OLLACK L EARNING
Revised: 2. K OLEN -P OLLACK L EARNING

Original: For the purpose of overcoming the weight transportation problem, both Kolen-Pollack learning and the method it is based on, feedback alignment (FA), use a backward matrix at each layer, which shares the same dimensions as the transpose of the corresponding forward matrix, that takes the place of the weight transpose while calculating the learning signal at the upstream layer.
Revised: For the purpose of overcoming the weight transportation problem, both Kolen-Pollack learning and the method it is based on, feedback alignment (FA), use a backward matrix at each layer, which shares the same dimensions as the transpose of the corresponding forward matrix, that takes the place of the weight transpose while calculating the learning signal at the upstream layer.

Original: While the backward path consists of ﬁxed random weight matrices in feedback alignment, the backward matrices in Kolen-Pollack learning are updated with the transpose of the weight updates made to their forward counterparts, and weight decay is used on both the forward and backward weight matrices in Kolen-Pollack learning.
Revised: While the backward path consists of ﬁxed random weight matrices in feedback alignment, the backward matrices in Kolen-Pollack learning are updated with the transpose of the weight updates made to their forward counterparts, and weight decay is used on both the forward and backward weight matrices in Kolen-Pollack learning.

Original: Following our prior notation, the learning signal δ (cid:96) for some layer (cid:96) <
Revised: Following our prior notation, the learning signal δ (cid:96) for some layer (cid:96) <

Original: k as prescribed by Kolen-Pollack learning and feedback alignment are as follows:
Revised: k as prescribed by Kolen-Pollack learning and feedback alignment are as follows:

Original: [Equation]
Revised: [Equation]

Original: For Kolen-Pollack learning, the update directions for B (cid:96) and W (cid:96) with weight decay, where λ is a number between 0 and 1 , are as follows:
Revised: For Kolen-Pollack learning, the update directions for B (cid:96) and W (cid:96) with weight decay, where λ is a number between 0 and 1 , are as follows:

Original: [Equation]
Revised: [Equation]

Original: In later sections we will discuss the learning dynamic of KP and how it applies to our proposed method.
Revised: In later sections we will discuss the learning dynamic of KP and how it applies to our proposed method.

Original: 2. D IRECT K OLEN -P OLLACK
Revised: 2. D IRECT K OLEN -P OLLACK

Original: To address these issues with DFA, we propose a method for updating the backward matrices inspired by the work of Akrout et al.
Revised: To address these issues with DFA, we propose a method for updating the backward matrices inspired by the work of Akrout et al.

Original: and Kolen & Pollack (1994) which we call direct Kolen-Pollack (DKP) learning.
Revised: and Kolen & Pollack (1994) which we call direct Kolen-Pollack (DKP) learning.

Original: For DKP, the prior rules for DFA as stated in the previous section remain the same, however B (cid:96) is no longer a ﬁxed matrix.
Revised: For DKP, the prior rules for DFA as stated in the previous section remain the same, however B (cid:96) is no longer a ﬁxed matrix.

Original: Rather, we will adjust the backward matrices after each batch using the following update rule where η B is the learning rate for the backward parameters and δ k is the learning signal at the output layer k .
Revised: Rather, we will adjust the backward matrices after each batch using the following update rule where η B is the learning rate for the backward parameters and δ k is the learning signal at the output layer k .

Original: [Equation]
Revised: [Equation]

Original: Similarly to the Kolen-Pollack approach used by Akrout et al.
Revised: Similarly to the Kolen-Pollack approach used by Akrout et al.

Original: (2019), we also use weight decay on both the forward and backward parameters during training.
Revised: (2019), we also use weight decay on both the forward and backward parameters during training.

Original: Thus, the update directions for B (cid:96) and
Revised: Thus, the update directions for B (cid:96) and

Original: W (cid:96) with weight decay would be as follows where the hyper-parameter λ is ranged between 0 and 1 .
Revised: W (cid:96) with weight decay would be as follows where the hyper-parameter λ is ranged between 0 and 1 .

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In our experiments, we found that weight decay on both the forward and backward matrices was crucial for maintaining the stability of the network during training, but we also try to substantiate this mathematically in the following sections.
Revised: In our experiments, we found that weight decay on both the forward and backward matrices was crucial for maintaining the stability of the network during training, but we also try to substantiate this mathematically in the following sections.

Original: While clear parallels exist between DKP and KP, the underlying learning dynamics between the two approaches are different as we will discuss in section 2.5 though we assert that the end result of these approaches are the same.
Revised: While clear parallels exist between DKP and KP, the underlying learning dynamics between the two approaches are different as we will discuss in section 2.5 though we assert that the end result of these approaches are the same.

Original: This result being that the forward and backward connections align to estimate the same function.
Revised: This result being that the forward and backward connections align to estimate the same function.

Original: 2. K OLEN -P OLLACK L EARNING AND C ONVERGENCE THROUGH W EIGHT D ECAY
Revised: 2. K OLEN -P OLLACK L EARNING AND C ONVERGENCE THROUGH W EIGHT D ECAY

Original: Kolen-Pollack learning works on the principal of convergence through weight decay.
Revised: Kolen-Pollack learning works on the principal of convergence through weight decay.

Original: Akrout et al.
Revised: Akrout et al.

Original: (2019) show that two synapses receiving the same arbitrary updates with an equal amount of weight decay will eventually converge on the same value.
Revised: (2019) show that two synapses receiving the same arbitrary updates with an equal amount of weight decay will eventually converge on the same value.

Original: Before showing how this may apply to direct
Revised: Before showing how this may apply to direct

Original: Kolen-Pollack learning, let us ﬁrst reiterate how two synapses, and by extension two weight matrices that share the same dimensions, will converge in Kolen-Pollack learning.
Revised: Kolen-Pollack learning, let us ﬁrst reiterate how two synapses, and by extension two weight matrices that share the same dimensions, will converge in Kolen-Pollack learning.

Original: We ﬁrst consider a pair of discrete-valued forward and backward weight matrices, W and B , that share the same dimensions such that all i, j element pairs W i,j and B i,j are different at time step t = 0 .
Revised: We ﬁrst consider a pair of discrete-valued forward and backward weight matrices, W and B , that share the same dimensions such that all i, j element pairs W i,j and B i,j are different at time step t = 0 .

Original: Then, at every time step t we update each pair of elements W i,j ( t ) and B i,j ( t ) with some arbitrary adjustment value
Revised: Then, at every time step t we update each pair of elements W i,j ( t ) and B i,j ( t ) with some arbitrary adjustment value

Original: A i,j ( t ) along with an equal amount of weight decay on both elements where 0 < λ < 1 .
Revised: A i,j ( t ) along with an equal amount of weight decay on both elements where 0 < λ < 1 .

Original: Thus, at time step t the updates made to W i,j ( t ) and B i,j ( t ) are given as follows:
Revised: Thus, at time step t the updates made to W i,j ( t ) and B i,j ( t ) are given as follows:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: t )
Revised: t )

Original: [Equation]
Revised: [Equation]

Original: For the sake of algebraically showing why the forward and backward weight matrices in KolenPollack learning come to mirror one another, let us now consider the difference between W i,j ( t +1) and B i,j ( t + 1) as
Revised: For the sake of algebraically showing why the forward and backward weight matrices in KolenPollack learning come to mirror one another, let us now consider the difference between W i,j ( t +1) and B i,j ( t + 1) as

Original: [Equation]
Revised: [Equation]

Original: Since 0 < λ < 1 , we also ﬁnd that 0 < 1 − λ < 1 .
Revised: Since 0 < λ < 1 , we also ﬁnd that 0 < 1 − λ < 1 .

Original: Thus, lim t →∞ (1 − λ ) t +1 = 0 , and by extension lim t →∞ (1 − λ ) t +1 [ W i,j (0) − B i,j (0)]
Revised: Thus, lim t →∞ (1 − λ ) t +1 = 0 , and by extension lim t →∞ (1 − λ ) t +1 [ W i,j (0) − B i,j (0)]

Original: = 0 since W i,j and B i,j are initialized as discrete values.
Revised: = 0 since W i,j and B i,j are initialized as discrete values.

Original: Therefore as t increases in value, the difference between W i,j ( t ) and B i,j ( t ) will approach zero.
Revised: Therefore as t increases in value, the difference between W i,j ( t ) and B i,j ( t ) will approach zero.

Original: Since this is true for any pair of corresponding i, j elements between W and B , it follows that the same is true for the two weight matrices themselves, and thus, the following also should hold:
Revised: Since this is true for any pair of corresponding i, j elements between W and B , it follows that the same is true for the two weight matrices themselves, and thus, the following also should hold:

Original: [Equation]
Revised: [Equation]

Original: Therefore, after enough time steps, W and B will come to approximately equal to each other.
Revised: Therefore, after enough time steps, W and B will come to approximately equal to each other.

Original: Note that while training a neural network with Kolen-Pollack learning, the updates made to the forward and backward weight matrices at each layer are not arbitrary.
Revised: Note that while training a neural network with Kolen-Pollack learning, the updates made to the forward and backward weight matrices at each layer are not arbitrary.

Original: Thus, each pair of forward and backward matrices will come to equal each other while also minimizing the global error of the network.
Revised: Thus, each pair of forward and backward matrices will come to equal each other while also minimizing the global error of the network.

Original: Because the forward and backward weight matrices do eventually come to approximately mirror one another, i.e . B (cid:96) (cid:39) W T (cid:96) , we expect that the gradients made by Kolen-Pollack learning will eventually closely align with the gradients made by the backpropagation algorithm.
Revised: Because the forward and backward weight matrices do eventually come to approximately mirror one another, i.e . B (cid:96) (cid:39) W T (cid:96) , we expect that the gradients made by Kolen-Pollack learning will eventually closely align with the gradients made by the backpropagation algorithm.

Original: 2. A LIGNMENT IN D IRECT K OLEN -P OLLACK L EARNING
Revised: 2. A LIGNMENT IN D IRECT K OLEN -P OLLACK L EARNING

Original: In direct Kolen-Pollack (DKP) learning, it is not immediately clear how the forward and backward paths are aligning since the forward and backward weight matrices at each layer do not share the
Revised: In direct Kolen-Pollack (DKP) learning, it is not immediately clear how the forward and backward paths are aligning since the forward and backward weight matrices at each layer do not share the

Original: [Figure]
Revised: [Figure]

Original: same dimensions.
Revised: same dimensions.

Original: The only exception is the second to last layer ( k − 1) where the forward and backward weight matrices do share the same dimensions, and thus, we expect this layer to behave as it would in Kolen-Pollack learning.
Revised: The only exception is the second to last layer ( k − 1) where the forward and backward weight matrices do share the same dimensions, and thus, we expect this layer to behave as it would in Kolen-Pollack learning.

Original: In Kolen-Pollack learning, we can say that some backward matrix
Revised: In Kolen-Pollack learning, we can say that some backward matrix

Original: B (cid:96) is converging on the function f (cid:96) that maps the input a (cid:96) − 1 of layer (cid:96) to the preactivation hidden state h (cid:96) i.e f (cid:96) : a (cid:96) − 1 (cid:55)→ h (cid:96) .
Revised: B (cid:96) is converging on the function f (cid:96) that maps the input a (cid:96) − 1 of layer (cid:96) to the preactivation hidden state h (cid:96) i.e f (cid:96) : a (cid:96) − 1 (cid:55)→ h (cid:96) .

Original: As we showed in the previous section, this method of convergence works with Kolen-Pollack learning because f (cid:96) is a linear mapping that uses the same dimensions as B T(cid:96) .
Revised: As we showed in the previous section, this method of convergence works with Kolen-Pollack learning because f (cid:96) is a linear mapping that uses the same dimensions as B T(cid:96) .

Original: However, with direct Kolen-Pollack learning the function f (cid:96) is no longer a linear function for all layers (cid:96) <
Revised: However, with direct Kolen-Pollack learning the function f (cid:96) is no longer a linear function for all layers (cid:96) <

Original: k − 1 and instead is the nonlinear function f (cid:96) : a (cid:96) − 1 (cid:55)→ h k which is comprised of layers (cid:96) through k .
Revised: k − 1 and instead is the nonlinear function f (cid:96) : a (cid:96) − 1 (cid:55)→ h k which is comprised of layers (cid:96) through k .

Original: Due to the highly nonlinear nature of f (cid:96) alone it would not be reasonably possible for any direct linear function in the backward path to converge with this function.
Revised: Due to the highly nonlinear nature of f (cid:96) alone it would not be reasonably possible for any direct linear function in the backward path to converge with this function.

Original: However, we might expect these direct feedback connections in DKP to converge with a linear function that estimates the nonlinear forward connections.
Revised: However, we might expect these direct feedback connections in DKP to converge with a linear function that estimates the nonlinear forward connections.

Original: That is to say B (cid:96) converges on some linear function g (cid:96) that estimates the nonlinear function f (cid:96) : a (cid:96) − 1 (cid:55)→ h k .
Revised: That is to say B (cid:96) converges on some linear function g (cid:96) that estimates the nonlinear function f (cid:96) : a (cid:96) − 1 (cid:55)→ h k .

Original: To substantiate these claims we experimentally show that as a network trains with DKP the cosine alignment angle between the preactivation output of the network h k , to which all functions f (cid:96) map to, and the dot product of the activations a (cid:96) at any layer (cid:96) <
Revised: To substantiate these claims we experimentally show that as a network trains with DKP the cosine alignment angle between the preactivation output of the network h k , to which all functions f (cid:96) map to, and the dot product of the activations a (cid:96) at any layer (cid:96) <

Original: k − 1 and the transpose of the corresponding direct backward weight matrix B (cid:96) will closely align after a sufﬁcient number of training steps has passed.
Revised: k − 1 and the transpose of the corresponding direct backward weight matrix B (cid:96) will closely align after a sufﬁcient number of training steps has passed.

Original: The network we will be using to examine this property of DKP is a ﬁve layer fully connected neural network trained on MNIST with cross entropy used for the loss function.
Revised: The network we will be using to examine this property of DKP is a ﬁve layer fully connected neural network trained on MNIST with cross entropy used for the loss function.

Original: After each hidden layer, batch normalization is used and followed with a rectiﬁed linear unit activation function.
Revised: After each hidden layer, batch normalization is used and followed with a rectiﬁed linear unit activation function.

Original: In ﬁgure 2b, we see that with DKP the cosine alignment angle between a (cid:96) − 1 · B T(cid:96) for each layer (cid:96) <
Revised: In ﬁgure 2b, we see that with DKP the cosine alignment angle between a (cid:96) − 1 · B T(cid:96) for each layer (cid:96) <

Original: k and the output of the network h k is much lower than it is with DFA in ﬁgure 2a with the exception of the angle measured at the ﬁrst layer which does not seem to improve much from DFA to DKP.
Revised: k and the output of the network h k is much lower than it is with DFA in ﬁgure 2a with the exception of the angle measured at the ﬁrst layer which does not seem to improve much from DFA to DKP.

Original: Thus, we see that both DFA and DKP seem to exhibit this behavior of estimating the nonlinear forward path with some linear function, however DKP clearly does this to greater effect.
Revised: Thus, we see that both DFA and DKP seem to exhibit this behavior of estimating the nonlinear forward path with some linear function, however DKP clearly does this to greater effect.

Original: We also notice that generally the direct feedback connections further downstream align much more closely with the forward connections than those further upstream.
Revised: We also notice that generally the direct feedback connections further downstream align much more closely with the forward connections than those further upstream.

Original: This result likely occurs because the upstream feedback connections are converging to some linear function that has to estimate a forward function that is much more complex and highly nonlinear relative to the downstream functions.
Revised: This result likely occurs because the upstream feedback connections are converging to some linear function that has to estimate a forward function that is much more complex and highly nonlinear relative to the downstream functions.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: All of our experiments were coded with Python using the PyTorch library.
Revised: All of our experiments were coded with Python using the PyTorch library.

Original: Code will be available on GitHub soon.
Revised: Code will be available on GitHub soon.

Original: 3. E XPERIMENTAL S ET - UP
Revised: 3. E XPERIMENTAL S ET - UP

Original: In our experiments, using batch normalization was necessary to gain stable training for both DFA and DKP, and both beneﬁt signiﬁcantly from its usage depending on the weight initialization method.
Revised: In our experiments, using batch normalization was necessary to gain stable training for both DFA and DKP, and both beneﬁt signiﬁcantly from its usage depending on the weight initialization method.

Original: This was especially an important for DKP as without batch normalization we would often run into an issue of exploding gradients.
Revised: This was especially an important for DKP as without batch normalization we would often run into an issue of exploding gradients.

Original: Additionally, for DFA and DKP we use Kaiming weight initialization(He et al., 2015) on the backward weight matrices as suggested by Launay et al.
Revised: Additionally, for DFA and DKP we use Kaiming weight initialization(He et al., 2015) on the backward weight matrices as suggested by Launay et al.

Original: We also found that the optimal hyperparameters and optimizers for the backward weight matrices in DKP seem to vary a some from one architecture to the next.
Revised: We also found that the optimal hyperparameters and optimizers for the backward weight matrices in DKP seem to vary a some from one architecture to the next.

Original: For our ﬁrst set of experiments with convolutional networks we employ a smaller network with only two convolutional layers to train on the CIFAR10 and Fashion-MNIST datasets, and for this network, training on CIFAR10, we found that the best optimizer for the backward weights was simply the same as that used on the forward weights, SGD with Nesterov acceleration and momentum at 0.9, with the only difference being a reduced learning rate.
Revised: For our ﬁrst set of experiments with convolutional networks we employ a smaller network with only two convolutional layers to train on the CIFAR10 and Fashion-MNIST datasets, and for this network, training on CIFAR10, we found that the best optimizer for the backward weights was simply the same as that used on the forward weights, SGD with Nesterov acceleration and momentum at 0.9, with the only difference being a reduced learning rate.

Original: On the forward parameters a learning rate of 10 − 2 was used, and on the backward parameters a learning rate of 10 − 4 was instead used.
Revised: On the forward parameters a learning rate of 10 − 2 was used, and on the backward parameters a learning rate of 10 − 4 was instead used.

Original: Both forward and backward parameters used weight decay with λ = 10 − 6 .
Revised: Both forward and backward parameters used weight decay with λ = 10 − 6 .

Original: Additionally, we used a step-wise learning rate decay on the backward parameters with γ = 0 . 85 which further improved the training stability.
Revised: Additionally, we used a step-wise learning rate decay on the backward parameters with γ = 0 . 85 which further improved the training stability.

Original: However, in our second experiment training the AlexNet architecture on the CIFAR100 dataset and TinyImageNet200 dataset, neither of these previous strategies involving the optimizer and learning rate scheduler were optimal.
Revised: However, in our second experiment training the AlexNet architecture on the CIFAR100 dataset and TinyImageNet200 dataset, neither of these previous strategies involving the optimizer and learning rate scheduler were optimal.

Original: For training with AlexNet on CIFAR100, the Adam optimizer (Kingma & Ba, 2014) was used on the backward weight matrices with a learning rate of 5 × 10 − 4 , while SGD with Nesterov acceleration, a learning rate of 0 . 01 , and a momentum of 0.9 was again used on the forward weight matrices.
Revised: For training with AlexNet on CIFAR100, the Adam optimizer (Kingma & Ba, 2014) was used on the backward weight matrices with a learning rate of 5 × 10 − 4 , while SGD with Nesterov acceleration, a learning rate of 0 . 01 , and a momentum of 0.9 was again used on the forward weight matrices.

Original: In the case of the AlexNet tests, both the forward and backward parameters used weight decay where λ = 10 − 4 .
Revised: In the case of the AlexNet tests, both the forward and backward parameters used weight decay where λ = 10 − 4 .

Original: The only difference moving to the TinyImageNet200 dataset was a slight decrease of the forward learning rate to 5 /cross 10 − 3 for all approaches except the backpropagation tests.
Revised: The only difference moving to the TinyImageNet dataset was a slight decrease of the forward learning rate to 5 × 10 − 3 for all approaches except the backpropagation tests.

Original: We believe that the differences in what was optimal for training with DKP between the two networks were the result of the increased difﬁculty in the problems being solved and potentially the size of the networks themselves.
Revised: We believe that the differences in what was optimal for training with DKP between the two networks were the result of the increased difﬁculty in the problems being solved and potentially the size of the networks themselves.

Original: However, the weight initializations, hyperparameters, and optimizers used in our experiments were not found through a rigorous search, and thus further improvements to the performance of these learning algorithms could potentially be found.
Revised: However, the weight initializations, hyperparameters, and optimizers used in our experiments were not found through a rigorous search, and thus further improvements to the performance of these learning algorithms could potentially be found.

Original: We also note that in all of our experiments, the learning rates seen are about the highest values that one would want to use before running the risk of exploding gradients, and that for all methods it is perfectly acceptable to use lower values.
Revised: We also note that in all of our experiments, the learning rates seen are about the highest values that one would want to use before running the risk of exploding gradients, and that for all methods it is perfectly acceptable to use lower values.

Original: However there is a slight exception when training with KP and DKP: if the learning rate on the backward weights are too low, then these methods begin to behave as their static counterparts FA and DFA respectively which leads to reduced performance.
Revised: However there is a slight exception when training with KP and DKP: if the learning rate on the backward weights are too low, then these methods begin to behave as their static counterparts FA and DFA respectively which leads to reduced performance.

Original: Because the change from DFA to DKP is analogous to the change from FA to KP, we will be including FA and KP in our experiments for comparison to their direct counterparts.
Revised: Because the change from DFA to DKP is analogous to the change from FA to KP, we will be including FA and KP in our experiments for comparison to their direct counterparts.

Original: These additional experiments will help to illustrate the difﬁculty that arises from training with direct feedback connections and that the application of Kolen-Pollack learning alone is not enough to overcome their shortcomings.
Revised: These additional experiments will help to illustrate the difﬁculty that arises from training with direct feedback connections and that the application of Kolen-Pollack learning alone is not enough to overcome their shortcomings.

Original: 3. T RAINING C ONVOLUTIONAL N ETWORKS
Revised: 3. T RAINING C ONVOLUTIONAL N ETWORKS

Original: Because DFA has a difﬁcult time training convolutional layers, achieving a more meaningful result with direct feedback connections in these deep convolutional networks is an important step and the primary improvement DKP makes over its predecessor.
Revised: Because DFA has a difﬁcult time training convolutional layers, achieving a more meaningful result with direct feedback connections in these deep convolutional networks is an important step and the primary improvement DKP makes over its predecessor.

Original: To demonstrate DKP’s capability in training convolutional layers we not only compare it to backpropagation and DFA but also we test each network’s performance with the training on the convolutional layers frozen such that only the fully connected layers are trained with backpropagation; anything that performs less than or equal to this in terms of accuracy or loss is likely not training the convolutional layers effectively.
Revised: To demonstrate DKP’s capability in training convolutional layers we not only compare it to backpropagation and DFA but also we test each network’s performance with the training on the convolutional layers frozen such that only the fully connected layers are trained with backpropagation; anything that performs less than or equal to this in terms of accuracy or loss is likely not training the convolutional layers effectively.

Original: We also compare it to direct random target projection(DRTP)
Revised: We also compare it to direct random target projection(DRTP)

Original: (Frenkel et al., 2019) which projects the one hot encoding of the target directly to each layer.
Revised: (Frenkel et al., 2019) which projects the one hot encoding of the target directly to each layer.

Original: The loss function used in all of our experiments is the cross entropy loss.
Revised: The loss function used in all of our experiments is the cross entropy loss.

Original: All reported numbers are an average of 10 trials.
Revised: All reported numbers are an average of 10 trials.

Original: For our ﬁrst experiment, we train on the CIFAR10 dataset for 50 epochs using a network that consists of just two convolutional layers followed by two fully connected layers, the second being the output layer.
Revised: For our ﬁrst experiment, we train on the CIFAR10 dataset for 50 epochs using a network that consists of just two convolutional layers followed by two fully connected layers, the second being the output layer.

Original: The results in Table 1 show that DFA does considerably better than BP with the convolutional layers frozen, and that DKP performs even better than DFA.
Revised: The results in Table 1 show that DFA does considerably better than BP with the convolutional layers frozen, and that DKP performs even better than DFA.

Original: Of course, we still see that BP is out performing both DFA and DKP by a much larger margin.
Revised: Of course, we still see that BP is out performing both DFA and DKP by a much larger margin.

Original: In our second experiment, we train on the CIFAR100 and TinyImageNet 200 datasets using the
Revised: In our second experiment, we train on the CIFAR100 and TinyImageNet 200 datasets using the

Original: AlexNet architecture for 90 epochs, and again, batch normalization is used before the activation
Revised: AlexNet architecture for 90 epochs, and again, batch normalization is used before the activation

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: CIFAR

Original: BP (FC Only)
Revised: BP (FC Only)

Original: Serial 47 .
Revised: Serial 47 .

Original: 72% ± 0 .39 .
Revised: 72% ± 0 .39 .

Original: 03% ± 0 .
Revised: 03% ± 0 .

Original: BP (Upperbound)
Revised: BP (Upperbound)

Original: Serial 65 . 88% ± 1 .64 .
Revised: Serial 65 . 88% ± 1 .64 .

Original: 09% ± 0 .
Revised: 09% ± 0 .

Original: KP
Revised: KP

Original: Serial 66 .
Revised: Serial 66 .

Original: 78% ± 

Revised: 78% ±

Original: 70% ± 1 .
Revised: 70% ± 1 .

Original: FA
Revised: FA

Original: Serial 19 .
Revised: Serial 19 .

Original: 49% ± 0 .12 .
Revised: 49% ± 0 .12 .

Original: 90% ± 0 .
Revised: 90% ± 0 .

Original: DRTP
Revised: DRTP

Original: Parallel+ 5 .
Revised: Parallel+ 5 .

Original: 84% ± 0 .5 .
Revised: 84% ± 0 .5 .

Original: 49% ± 0 .
Revised: 49% ± 0 .

Original: DFA
Revised: DFA

Original: Parallel 48 .
Revised: Parallel 48 .

Original: 03% ± 0 .35 .
Revised: 03% ± 0 .35 .

Original: 18% ± 0 .
Revised: 18% ± 0 .

Original: DKP (Ours)
Revised: DKP (Ours)

Original: Parallel 52 .
Revised: Parallel 52 .

Original: 62 % ± 0 .45 .
Revised: 62 % ± 0 .45 .

Original: 17 % ± 0 .
Revised: 17 % ± 0 .

Original: TinyImageNet
Revised: TinyImageNet

Original: BP (FC Only)
Revised: BP (FC Only)

Original: Serial 29 .
Revised: Serial 29 .

Original: 78% ± 0 .24 . 89% ± 0 .
Revised: 78% ± 0 .24 . 89% ± 0 .

Original: BP (Upperbound)
Revised: BP (Upperbound)

Original: Serial 49 .
Revised: Serial 49 .

Original: 53% ± 

Revised: 53% ±

Original: 25% ± 0 .
Revised: 25% ± 0 .

Original: KP
Revised: KP

Original: Serial 51 .
Revised: Serial 51 .

Original: 36% ± 1 .60 .
Revised: 36% ± 1 .60 .

Original: 44% ± 1 .
Revised: 44% ± 1 .

Original: FA
Revised: FA

Original: Serial 9 .
Revised: Serial 9 .

Original: 98% ± 1 .7 .
Revised: 98% ± 1 .7 .

Original: 12% ± 0 .
Revised: 12% ± 0 .

Original: DRTP
Revised: DRTP

Original: Parallel+ 2 .
Revised: Parallel+ 2 .

Original: 86% ± 0 .2 .
Revised: 86% ± 0 .2 .

Original: 86% ± 0 .
Revised: 86% ± 0 .

Original: DFA
Revised: DFA

Original: Parallel 32 . 116% ± 0 .24 .
Revised: Parallel 32 . 116% ± 0 .24 .

Original: 75% ± 0 .
Revised: 75% ± 0 .

Original: DKP (Ours)
Revised: DKP (Ours)

Original: Parallel 35 .
Revised: Parallel 35 .

Original: 78 % ± 1 .35 .
Revised: 78 % ± 1 .35 .

Original: 97 % ± 0 .
Revised: 97 % ± 0 .

Original: Layer No.
Revised: Layer No.

Original: Layer Conﬁguration
Revised: Layer Conﬁguration

Original: Conv(Channels: 3-32, Kernel: 3, Padding: 1), BN., ReLU
Revised: Conv(Channels: 3-32, Kernel: 3, Padding: 1), BN., ReLU

Revised: [Table]

Original: Conv(Channels: 32-32, Kernel: 3, Padding: 1), BN, ReLU
Revised: (Channels: 32-32, Kernel: 3, Padding: 1), BN, ReLU

Original: Max Pool(Kernel: 2), Linear(6272, 128), BN, ReLU
Revised: Max Pool(Kernel: 2), Linear(6272, 128), BN, ReLU

Original: Linear(128, 10)
Revised: Linear [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: functions on all layers except the output layer.
Revised: functions on all layers except the output layer.

Original: For CIFAR100, while we worked to get as close to a direct comparison as we could between BP, DFA, and DKP in terms of hyperparameters and weight initializations, we did have to lower the learning rate from 0 .
Revised: For CIFAR100, while we worked to get as close to a direct comparison as we could between BP, DFA, and DKP in terms of hyperparameters and weight initializations, we did have to lower the learning rate from 0 .

Original: 1 to 0 .
Revised: 1 to 0 .

Original: 01 on the forward parameters with DFA and DKP to achieve more stable training; lowering these values for the backpropagation tests resulted in slightly lower accuracies.
Revised: 01 on the forward parameters with DFA and DKP to achieve more stable training; lowering these values for the backpropagation tests resulted in slightly lower accuracies.

Revised: Similarly, the TinyImageNet200 results required slightly lower learning rates.

Original: The results in table 2 are mostly consistent with what is seen in the CIFAR10 test.
Revised: The results in table 2 are mostly consistent with what is seen in the CIFAR10 test.

Original: DKP, while showing a solid jump in performance over DFA, still fails to match the performance of BP, and DFA only performs marginally better than BP when only the fully connected layers trained.
Revised: DKP, while showing a solid jump in performance over DFA, still fails to match the performance of BP, and DFA only performs marginally better than BP when only the fully connected layers trained.

Original: We see that as the contribution to the results in accuracy made by the fully connected layers diminishes, so too does the results of DFA.
Revised: We see that as the contribution to the results in accuracy made by the fully connected layers diminishes, so too does the results of DFA.

Original: Also, we see that as the networks become more reliant on the convolutional layers to perform well, the gap between DFA and DKP in terms of accuracy widens in favor of our approach.
Revised: Also, we see that as the networks become more reliant on the convolutional layers to perform well, the gap between DFA and DKP in terms of accuracy widens in favor of our approach.

Original: We also note that the direct random target projection(DRTP) approach suffers terribly from the usage of the cross entropy loss despite performing well on networks similar to our experiments in table 1 when using mean squared error.
Revised: We also note that the direct random target projection(DRTP) approach suffers terribly from the usage of the cross entropy loss despite performing well on networks similar to our experiments in table 1 when using mean squared error.

Original: So another beneﬁt of DKP is that it is compatible with a more robust loss function, cross entropy.
Revised: So another beneﬁt of DKP is that it is compatible with a more robust loss function, cross entropy.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Direct feedback alignment (DFA) enables the parallelization of the backwards pass, called backwards unlocking, and has shown promising results in NLP tasks(Launay et al., 2020).
Revised: Direct feedback alignment (DFA) enables the parallelization of the backwards pass, called backwards unlocking, and has shown promising results in NLP tasks(Launay et al., 2020).

Original: Despite these clear advantages, DFA fails to effectively train deep convolutional networks on difﬁcult image classiﬁcations tasks.
Revised: Despite these clear advantages, DFA fails to effectively train deep convolutional networks on difﬁcult image classiﬁcations tasks.

Original: We propose direct Kolen-Pollack (DKP) learning by incorporating Kolen-Pollack learning into DFA to more effectively train deep convolutional neural networks with direct feedback connections by updating the backward weight matrices.
Revised: We propose direct Kolen-Pollack (DKP) learning by incorporating Kolen-Pollack learning into DFA to more effectively train deep convolutional neural networks with direct feedback connections by updating the backward weight matrices.

Original: We empirically show that our approach produces gradients that better align with the backpropagation algorithm.
Revised: We empirically show that our approach produces gradients that better align with the backpropagation algorithm.

Original: We further show that DKP outperforms DFA while training convolutional neural networks on the Fashion-MNIST, CIFAR10,
Revised: We further show that DKP outperforms DFA while training convolutional neural networks on the Fashion-MNIST, CIFAR10,

Original: CIFAR100, and TinyImageNet200 datasets.
Revised: CIFAR