Original: E FFICIENT C ONVOLUTIONAL N EURAL N ETWORK T RAINING WITH D IRECT F EEDBACK A LIGNMENT
Revised: E FFICIENT C ONVOLUTIONAL N EURAL N ETWORK T RAINING WITH D IRECT F EEDBACK A LIGNMENT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: There were many algorithms to substitute the back-propagation (BP) in the deep neural network (DNN) training.
Revised: There were many algorithms to substitute the back-propagation (BP) in the deep neural network (DNN) training.

Original: However, they could not become popular because their training accuracy and the computational efﬁciency were worse than
Revised: However, they could not become popular because their training accuracy and the computational efﬁciency were worse than

Original: BP.
Revised: BP.

Original: One of them was direct feedback alignment (DFA), but it showed low training performance especially for the convolutional neural network (CNN).
Revised: One of them was direct feedback alignment (DFA), but it showed low training performance especially for the convolutional neural network (CNN).

Original: In this paper, we overcome the limitation of the DFA algorithm by combining with the conventional BP during the CNN training.
Revised: In this paper, we overcome the limitation of the DFA algorithm by combining with the conventional BP during the CNN training.

Original: To improve the training stability, we also suggest the feedback weight initialization method by analyzing the patterns of the ﬁxed random matrices in the DFA.
Revised: To improve the training stability, we also suggest the feedback weight initialization method by analyzing the patterns of the ﬁxed random matrices in the DFA.

Original: Finally, we propose the new training algorithm, binary direct feedback alignment (BDFA) to minimize the computational cost while maintaining the training accuracy compared with the DFA.
Revised: Finally, we propose the new training algorithm, binary direct feedback alignment (BDFA) to minimize the computational cost while maintaining the training accuracy compared with the DFA.

Original: In our experiments, we use the CIFAR-10 and CIFAR-100 dataset to simulate the CNN learning from the scratch and apply the BDFA to the online learning based object tracking application to examine the fully-connected layer (FC) ﬁne-tuning task.
Revised: In our experiments, we use the CIFAR-10 and CIFAR-100 dataset to simulate the CNN learning from the scratch and apply the BDFA to the online learning based object tracking application to examine the training in the limited dataset environment.

Original: Our proposed algorithms show better performance than conventional BP in both two different training tasks, but they have lighter computations for the error propagation.
Revised: Our proposed algorithms show better performance than conventional BP in both two different training tasks especially when the dataset is small.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning becomes the core of the machine learning and it has been utilized for many applications such as machine translation (Singh et al.
Revised: Deep learning becomes the core of the machine learning and it has been utilized for many applications such as machine translation (Singh et al.

Original: speech recognition (Wang et al.
Revised: speech recognition (Wang et al.

Original: (2017)), and object classiﬁcation (Rawat & Wang (2017)).
Revised: (2017)), and object classiﬁcation (Rawat & Wang (2017)).

Original: Training the deep neural network (DNN) is the important portion of the deep learning because we need different pre-trained models to cover the various deep learning tasks.
Revised: Training the deep neural network (DNN) is an important portion of the deep learning because we need different pre-trained models to cover the various deep learning tasks.

Original: One well-known method ofthe DNN training is the algorithm called backpropagation (BP) (Rumelhart et al.
Revised: One well-known method of DNN training is the algorithm called backpropagation (BP) (Rumelhart et al.

Original: The BP is the gradient descent based training method which follows the steepest gradient to ﬁnd the optimum weights.
Revised: The BP is the gradient descent based training method which follows the steepest gradient to ﬁnd the optimum weights.

Original: Therefore, BP based training can be applied to any DNN conﬁgurations if the network consists of any differentiable operations.
Revised: Therefore, BP based training can be applied to any DNN conﬁgurations if the network consists of any differentiable operations.

Original: For instance, not only multi-layer perceptron (MLP) but also both convolutional neural network (CNN,
Revised: For instance, not only multi-layer perceptron (MLP) but also both convolutional neural network (CNN,

Original: LeCun et al.
Revised: LeCun et al.

Original: and recurrent neural network (RNN, Hopﬁeld (1982)) can be trained by using the BP.
Revised: and recurrent neural network (RNN, Hopﬁeld (1982)) can be trained by using the BP.

Original: Even though the BP shows the outstanding performance in DNN training, BP based training suffers from gradient vanishing problem.
Revised: Even though the BP shows the outstanding performance in DNN training, BP based training suffers from the overﬁtting problem.

Original: At the beginning of the deep learning research, people used activation functions such as sigmoid and tanh.

Original: However, the magnitude of the propagated error has signiﬁcantly shrunk when the error is far from the last layer.
Revised: Since the BP easily sinks into a local minimum, we need a large scale of the dataset to avoid the overﬁtting.

Original: That’s because the BP uses chainrule based computation, and  the derivative of the activation function is less than 1. Hence, the gradient becomes much smaller if the network consists of multiple layers.
Revised: In addition, we use various data augmentation techniques such as ﬂipping and cropping. If the DNN training is done in the limited resources and dataset, BP based DNN training is too slow to be converged and shows low accuracy.

Original: To solve this problem, new activation function, ReLU was proposed by Nair & Hinton (2010), but it was not enough to substitute all conventional sigmoid or tanh activation functions. Another solution was adopting a residual network introduced by He et al.
Revised: One example is MDNet (Nam & Han (2016)) which introduces an object tracking algorithm with the online learning concept.

Original: The residual network can prevent the problem of gradient vanishing by creating short path passing through the multiple layers. Therefore, the residual connection makes the DNN become much deeper by changing the network structure.
Revised: Realtime implementation is an important issue in object tracking, so it has the limitation to utilize various data augmentation and large dataset for online learning.

Original: To break the BP based DNN training paradigm, a lot of new training methods have been developed.
Revised: To break the BP based DNN training paradigm, a lot of new training methods have been developed.

Original: Salimans et al.
Revised: Salimans et al.

Original: (2017) proposed the evolution strategy which searches various weight without the gradient descent.
Revised: (2017) proposed the evolution strategy which searches various weight without the gradient descent.

Original: It has a chance to be computed in parallel, but it needs a lot of seeds to ﬁnd the global optimal solution.
Revised: It has a chance to be computed in parallel, but it needs a lot of seeds to ﬁnd the global optimal solution.

Original: Moreover, evolution strategy has the slow convergence problem which can be the obstacle for fast online learning application.
Revised: Moreover, evolution strategy has a slow convergence problem which can be an obstacle for fast online learning application.

Original: Another algorithm, feedback alignment (FA, Lillicrap et al.
Revised: Another algorithm, feedback alignment (FA, Lillicrap et al.

Original: (2014)), was proposed based on the gradient descent methodology, but without following the steepest gradient.
Revised: (2014)), was proposed based on the gradient descent methodology, but without following the steepest gradient.

Original: The FA pre-deﬁnes a feedback weight before starting the training, and the weight is determined by the random values.
Revised: The FA pre-deﬁnes a feedback weight before starting the training, and the weight is determined by the random values.

Original: 1 It is also known to converge slowly, and shows worse performance compared with the previous BP.
Revised: 1 It is also known to converge slowly, and shows worse performance compared with the previous BP.

Original: Direct feedback alignment (DFA, Nokland (2016)) was developed by getting the idea from the FA.
Revised: Direct feedback alignment (DFA, Nokland (2016)) was developed by getting the idea from the FA.

Original: Although the FA propagates errors from the last layer back to the ﬁrst layer step-by-step, the DFA propagates errors from the last layer directly to each layer.
Revised: Although the FA propagates errors from the last layer back to the ﬁrst layer step-by-step, the DFA propagates errors from the last layer directly to each layer.

Original: This approachhelps the DNN become free from gradient vanishing problem without changing the network structure by the residual connection. Nokland (2016) showed that DFA makes the multilayer perceptron (MLP) become learnable even when the network is too large. One big advantage of the DFA is that it gives the opportunity of the parallel processing.
Revised: This approach gives the opportunity of the parallel processing.

Original: Since the errors of every layer are generated independently in DFA, we can immediately calculate each layer’s gradient if the DNN inference is ﬁnished.
Revised: Since the errors of every layer are generated independently in DFA, we can immediately calculate each layer’s gradient if the DNN inference is ﬁnished.

Original: Moreover, the number of required computation is reduced in the DFA.
Revised: Moreover, the number of required computation is reduced in the DFA.

Original: This is because the number of neurons in the last layer is usually fewer than the prior layer, so the size of the feedback weight becomes smaller than the BP.
Revised: This is because the number of neurons in the last layer is usually fewer than the prior layer, so the size of the feedback weight becomes smaller than the BP.

Original: In spite of these advantages, the DFA suffers from accuracy degradation problem.
Revised: In spite of these advantages, the DFA suffers from accuracy degradation problem.

Original: The accuracy degradation problem becomes more serious when the DFA is applied to the CNN case.
Revised: The accuracy degradation problem becomes more serious when the DFA is applied to the CNN case.

Original: It is known that the DNN is not learnable with the DFA if the DNN becomes much deeper than AlexNet (Krizhevsky et al.
Revised: It is known that the DNN is not learnable with the DFA if the DNN becomes much deeper than AlexNet (Krizhevsky et al.

Original: In this paper, we explain the new DFA algorithm to improve the training accuracy in CNN, and suggest a feedback weight initialization method for the fast convergence.
Revised: In this paper, we explain the new DFA algorithm to improve the training accuracy in CNN, and suggest a feedback weight initialization method for the fast convergence.

Original: Moreover, we propose the binary direct feedback alignment (BDFA) to maximize the computational efﬁciency.
Revised: Moreover, we propose the binary direct feedback alignment (BDFA) to maximize the computational efﬁciency.

Original: We veriﬁed that the VGG-16 (Simonyan & Zisserman (2015)) trained by DFA shows higher accuracy compared with the BP.
Revised: We veriﬁed the training performance in the VGG-16 (Simonyan & Zisserman (2015)) without any data augmentation, and DFA shows higher accuracy compared with the BP.

Original: And then, the performance of the fully-connected layer (FC) ﬁne-tuning was proved through the online learning based object tracking application.
Revised: And then, the training with small dataset was proved through the online learning based object tracking application.

Original: Our proposed algorithm shows better performance than the conventional BP approach in both the learning from  scratch and the FC ﬁnetuning tasks.
Revised: Our proposed algorithm shows better performance than the conventional BP approach in both the learning from the scratch and the object tracking tasks.

Original: The remaining part of the paper is organized as follows.
Revised: The remaining part of the paper is organized as follows.

Original: The mathematical notation of the BP,
Revised: The mathematical notation of the BP,

Original: FA, and DFA will be introduced in Section 2.
Revised: FA, and DFA will be introduced in Section 2.

Original: Then, the details of the proposed algorithms will be explained in Section 3.
Revised: Then, the details of the proposed algorithms will be explained in Section 3.

Original: The experiment will be followed in Section 4 and the paper will be concluded in Section 5.
Revised: The experiment will be followed in Section 4 and the paper will be concluded in Section 5.

Original:  P RELIMINARIES 2. B ACK - PROPAGATION
Revised:  P RELIMINARIES 2. B ACK - PROPAGATION

Original: Back-propagation is a general algorithm for the DNN training, suggested by Rumelhart et al.
Revised: Back-propagation is a general algorithm for the DNN training, suggested by Rumelhart et al.

Original: Let o i be the i th layer’s feature map, when W i +1 ,i be the weight and bias between the i th layer and the i +

Revised: Let o i be the i th layer’s feature map, when W i +1 ,i be the weight and bias between the i th layer and the i +

Original: th layer.
Revised: th layer.

Original: If the activation function is represented as f ( · ) , and L is the total number of layers, the feature map of the i + 1 th layer can be calculated as
Revised: If the activation function is represented as f ( · ) , and L is the total number of layers, the feature map of the i + 1 th layer can be calculated as

Original: [Equation]
Revised: [Equation]

Original: Various activation functions such as sigmoid, tanh, and ReLU (Nair & Hinton (2010)) can be one of candidates of the f ( · ) .
Revised: Various activation functions such as sigmoid, tanh, and ReLU (Nair & Hinton (2010)) can be one of candidates of the f ( · ) .

Original: Some DNNs such as generative adversarial networks (Goodfellow et al.
Revised: Some DNNs such as generative adversarial networks (Goodfellow et al.

Original: (2014)) use more advanced activation functions like Leaky ReLU and PReLU (Xu et al.
Revised: (2014)) use more advanced activation functions like Leaky ReLU and PReLU (Xu et al.

Original: Once the inference is over, the inference result is compared with the pre-deﬁned labels, and the error map e L is calculated by a loss function such as cross-entropy.
Revised: Once the inference is over, the inference result is compared with the pre-deﬁned labels, and the error map e L is calculated by a loss function such as cross-entropy.

Original: The error is propagated gradually from the last layer to the ﬁrst layer, and the i th layer’s error map can be calculated as
Revised: The error is propagated gradually from the last layer to the ﬁrst layer, and the i th layer’s error map can be calculated as

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) is an element-wise multiplication operator and f (cid:48) () is the derivative of the non-linear function.
Revised: where (cid:12) is an element-wise multiplication operator and f (cid:48) () is the derivative of the non-linear function.

Original: We need the transposed weight matrix, W Ti +1 ,i , to propagate the errors in MLP case.
Revised: We need the transposed weight matrix, W Ti +1 ,i , to propagate the errors in MLP case.

Original: After the BP, the gradient of each layer is computed by using both the i th layer’s feature map and i + 1 th layer’s error map.
Revised: After the BP, the gradient of each layer is computed by using both the i th layer’s feature map and i + 1 th layer’s error map.

Original: The i th layer’s gradient, G i is calculated as following.
Revised: The i th layer’s gradient, G i is calculated as following.

Original: [Equation]
Revised: [Equation]

Original: If we use the mini-batch gradient descent with the batch size B , total B gradients, G i,b are averaged and get the G i .
Revised: If we use the mini-batch gradient descent with the batch size B , total B gradients, G i,b are averaged and get the G i .

Original: G i is used to update the W
Revised: G i is used to update the W

Original: i +1 ,
Revised: i +1 ,

Original: i by multiplying learning rate, η .
Revised: i by multiplying learning rate, η .

Original: In the CNN case, the matrix multiplication operations in every step described in MLP is substituted with the convolution operations.
Revised: In the CNN case, the matrix multiplication operations in every step described in MLP is substituted with the convolution operations.

Original: One more different thing is that it uses 180°ﬂipped kernel instead of the transposed weight used in the BP.
Revised: One more different thing is that it uses 180°ﬂipped kernel instead of the transposed weight used in the BP.

Original: Other operations are as same as in the MLP case.
Revised: Other operations are as same as in the MLP case.

Original: 2. F EEDBACK A LIGNMENT
Revised: 2. F EEDBACK A LIGNMENT

Original: Feedback alignment (FA) which was introduced by Lillicrap et al.
Revised: Feedback alignment (FA) which was introduced by Lillicrap et al.

Original: (2016) substituted the transposed weight with the ﬁxed random matrix in error propagation operation.
Revised: (2016) substituted the transposed weight with the ﬁxed random matrix in error propagation operation.

Original: In other words, they pre-deﬁned the feedback weight which has the same size as the transposed weight, but the values of the feedback weight are determined randomly.
Revised: In other words, they pre-deﬁned the feedback weight which has the same size as the transposed weight, but the values of the feedback weight are determined randomly.

Original: Although the weights used in the training are updated for every iteration, the pre-deﬁned random matrix is maintained until the training is ﬁnished.
Revised: Although the weights used in the training are updated for every iteration, the pre-deﬁned random matrix is maintained until the training is ﬁnished.

Original: Let R i +1 ,i be the feedback weight of the i th layer, the propagated error in the FA is calculated as
Revised: Let R i +1 ,i be the feedback weight of the i th layer, the propagated error in the FA is calculated as

Original: [Equation]
Revised: [Equation]

Original: , and the other procedures such as gradient generation and weight updating are same as BP does.
Revised: , and the other procedures such as gradient generation and weight updating are same as BP does.

Original: 2. D IRECT F EEDBACK A LIGNMENT
Revised: 2. D IRECT F EEDBACK A LIGNMENT

Original: Even though, both the BP and the FA propagate errors from the last layer to the ﬁrst layer in order, the DFA (Nokland (2016)) directly propagates errors from the last layer to other layers.
Revised: Even though, both the BP and the FA propagate errors from the last layer to the ﬁrst layer in order, the DFA (Nokland (2016)) directly propagates errors from the last layer to other layers.

Original: If the number of neurons in the last layer and the i th layer, are represented by N L and N i respectively, the size of the feedback weight is determined as N i × N L .
Revised: If the number of neurons in the last layer and the i th layer, are represented by N L and N i respectively, the size of the feedback weight is determined as N i × N L .

Original: Let, i th layer’s feedback weight in the DFA be D Ti , then the error is calculated as
Revised: Let, i th layer’s feedback weight in the DFA be D Ti , then the error is calculated as

Original: [Equation]
Revised: [Equation]

Original: One of the interesting characteristics is that there is no data dependency between different errors because the DFA propagates the errors directly from the last layer.
Revised: One of the interesting characteristics is that there is no data dependency between different errors because the DFA propagates the errors directly from the last layer.

Original: This characteristic gives the opportunity of parallel processing in the DFA based error propagation operation.
Revised: This characteristic gives the opportunity of parallel processing in the DFA based error propagation operation.

Original: Compared with the BP, DFA showed similar training performance for the multi-layer perceptron (MLP) structure.
Revised: Compared with the BP, DFA showed similar training performance for the multi-layer perceptron (MLP) structure.

Original: Moreover, DFA requires fewer computations because N L is generally smaller than the number of neurons in the intermediate layers.
Revised: Moreover, DFA requires fewer computations because N L is generally smaller than the number of neurons in the intermediate layers.

Original: However, the DFA dramatically degrades the accuracy when it applied in the CNN training as shown in Figure 1.
Revised: However, the DFA dramatically degrades the accuracy when it applied in the CNN training as shown in Figure 1.

Original: Furthermore, the elements of the intermediate feature map are all connected with the last layer neurons, so it requires much more computations compared with BP based CNN training.
Revised: Furthermore, the elements of the intermediate feature map are all connected with the last layer neurons, so it requires much more computations compared with BP based CNN training.

Original: In summary, the DFA’s computational efﬁciency can be induced in the MLP training, but not in the CNN.
Revised: In summary, the DFA’s computational efﬁciency can be induced in the MLP training, but not in the CNN.

Original:  O UR A PPROACH
Revised:  O UR A PPROACH

Original: As mentioned in Section 2, the DFA is efﬁcient for MLP training, because it can be computed in parallel without accuracy degradation.
Revised: As mentioned in Section 2, the DFA is efﬁcient for MLP training, because it can be computed in parallel without accuracy degradation.

Original: However, this advantage is diminished when the DFA is applied in the CNN.
Revised: However, this advantage is diminished when the DFA is applied in the CNN.

Original: To solve this problem, we propose the new training method to make the DFA applied to the CNN.
Revised: To solve this problem, we propose the new training method to make the DFA applied to the CNN.

Original: In addition, the initialization method of the feedback weight is suggested for the fast and stable learning curve.
Revised: In addition, the initialization method of the feedback weight is suggested for the fast and stable learning curve.

Original: At last, we propose the advanced training algorithm, binary direct feedback alignment (BDFA), which shows high computing efﬁciency andthe robust training performance in various conditions.
Revised: At last, we propose the advanced training algorithm, binary direct feedback alignment (BDFA), which shows high computing efﬁciency and robust training performance in various conditions.

Original: 3. CDFA: CNN T RAINING BY C OMBINING BOTH BP AND DFA
Revised: 3. CDFA: CNN T RAINING BY C OMBINING BOTH BP AND DFA

Original: Generally, CNN consists of the convolutional layers and fully-connected (FC) layers.
Revised: Generally, CNN consists of the convolutional layers and fully-connected (FC) layers.

Original: The two different kinds of layers have different roles.
Revised: The two different kinds of layers have different roles.

Original: For example in object classiﬁcation, convolutional layers are considered as the feature extractor by using the characteristics of the convolution computation.
Revised: For example in object classiﬁcation, convolutional layers are considered as the feature extractor by using the characteristics of the convolution computation.

Original: In contrast, FC layers receive the result of the feature extractor and judge what the object is.
Revised: In contrast, FC layers receive the result of the feature extractor and judge what the object is.

Original: However, in the initial iterations of the BP based CNN training, training the convolutional layers can be ambiguous because the FC layers cannot be considered as a good object classiﬁer.
Revised: However, in the initial iterations of the BP based CNN training, training the convolutional layers can be ambiguous because the FC layers cannot be considered as a good object classiﬁer.

Original: During some iterations of BP, the weight of the FC layers will be changed and the convolutional layers should be adaptive to the changed FC layers.
Revised: During some iterations of BP, the weight of the FC layers will be changed and the convolutional layers should be adaptive to the changed FC layers.

Original: In other words, convolutional layers can be confused if the propagated error’s domain is changed for every iteration.
Revised: In other words, convolutional layers can be confused if the propagated error’s domain is changed for every iteration.

Original: If the error is propagated from the last FC layer to convolutional layer through the constant domain shifting, the training of the convolutional layer can be more stable than conventional BP.
Revised: If the error is propagated from the last FC layer to convolutional layer through the constant domain shifting, the training of the convolutional layer can be more stable than conventional BP.

Original: From this motivation, we use DFA instead of BP in the FC layers.
Revised: From this motivation, we use DFA instead of BP in the FC layers.

Original: As shown in Figure 2, the network maintains the BP in the convolutional layers but adopts the DFA for the FC layers.
Revised: As shown in Figure 2, the network maintains the BP in the convolutional layers but adopts the DFA for the FC layers.

Original: Since there is no data dependency among the FC layers, the DFA can directly propagate the error to the FC1.
Revised: Since there is no data dependency among the FC layers, the DFA can directly propagate the error to the FC1.

Original: Since the DFA uses ﬁxed feedback weight for error propagation, convolutional layers do not have to be adaptive to various errors which are derived from the different domain.
Revised: Since the DFA uses ﬁxed feedback weight for error propagation, convolutional layers do not have to be adaptive to various errors which are derived from the different domain.

Original: In this method, the error propagation in the convolutional layers can be started even though the errors are not propagated for the remained FC layers.
Revised: In this method, the error propagation in the convolutional layers can be started even though the errors are not propagated for the remained FC layers.

Original: Therefore, the error propagation of both the convolutional layers and the
Revised: Therefore, the error propagation of both the convolutional layers and the

Original: FC layers canbe computedin parallelright after DFA is done for the ﬁrst FC layer.
Revised: FC layers can computed parallel

Original: Nokland (2016) shows that the randomly initialized feedback weight can be used for DFA based
Revised: Nokland (2016) shows that the randomly initialized feedback weight can be used for DFA based

Original: DNN training.
Revised: DNN training.

Original: However, both the FA and the DFA are sensitive to the initialization method of the feedback weight because it affects the training accuracy and the convergence speed signiﬁcantly.
Revised: However, both the FA and the DFA are sensitive to the initialization method of the feedback weight because it affects the training accuracy and the convergence speed signiﬁcantly.

Original: As shown in Liao et al. (2015), it is observed that the batch-normalization (BN) seems to make the
Revised: As shown in Liao et al. (2015), it is observed that the batch-normalization (BN) seems to make the

Original: FA become not sensitive to the initialization method, but it is still a problem because of the slow convergence.
Revised: FA become not sensitive to the initialization method, but it is still a problem because of the slow convergence.

Original: To make the DFA robust to the initialization method, we ﬁxed the feedback weight as the multiplication of the transposed weights in multiple layers.
Revised: To make the DFA robust to the initialization method, we ﬁxed the feedback weight as the multiplication of the transposed weights in multiple layers.

Original: To sum up, the feedback weight of the i th layer, D i , can be calculated as
Revised: To sum up, the feedback weight of the i th layer, D i , can be calculated as

Original: [Equation]
Revised: [Equation]

Original: , and ﬁnally, the error propagation operation can be summarized as follow.
Revised: , and ﬁnally, the error propagation operation can be summarized as follow.

Original: [Equation]
Revised: [Equation]

Original: The suggested initialization method is suitable for other various functions such as sigmoid, tanh and
Revised: The suggested initialization method is suitable for other various functions such as sigmoid, tanh and

Original: ReLU.
Revised: ReLU.

Original: Moreover, other normalization or optimization methods such as BN and dropout (Srivastava et al.
Revised: Moreover, other normalization or optimization methods such as BN and dropout (Srivastava et al.

Original: (2014)) are also applicable with the proposed initialization with the equation (7).
Revised: (2014)) are also applicable with the proposed initialization with the equation (7).

Original: 3. B INARY D IRECT F EEDBACK A LIGNMENT (BDFA)
Revised: 3. B INARY D IRECT F EEDBACK A LIGNMENT (BDFA)

Original: The DFA needs the feedback weight addition to the forward weight matrix, and it occupies a larger memory to store both two matrices.
Revised: The DFA needs the feedback weight addition to the forward weight matrix, and it occupies a larger memory to store both two matrices.

Original: Moreover, the DFA has the chance to be computed in parallel, but it requires much larger memory bandwidth.
Revised: Moreover, the DFA has the chance to be computed in parallel, but it requires much larger memory bandwidth.

Original: Since the throughput in the FC computing is vulnerable to the memory bandwidth, loading the additional feedback weight degrades the throughput compared with the unlimited bandwidth case.
Revised: Since the throughput in the FC computing is vulnerable to the memory bandwidth, loading the additional feedback weight degrades the throughput compared with the unlimited bandwidth case.

Original: To solve the throughput bottleneck problem caused limited bandwidth, we propose the binary direct feedback alignment (BDFA) algorithm.
Revised: To solve the throughput bottleneck problem caused limited bandwidth, we propose the binary direct feedback alignment (BDFA) algorithm.

Original: BDFA uses the binarized feedback weight, B i , whose values are determined as either +1 or -1.
Revised: BDFA uses the binarized feedback weight, B i , whose values are determined as either +1 or -1.

Original: In other words, the B i can be stored as a single bit to represent only the sign value of the feedback weight’s element.
Revised: In other words, the B i can be stored as a single bit to represent only the sign value of the feedback weight’s element.

Original: As a result, required memory to store the B i is reduced by 96.9% compared with the 32-bit ﬂoating point representation which now becomes the general numeric representation in CPU or GPU.
Revised: As a result, required memory to store the B i is reduced by 96.9% compared with the 32-bit ﬂoating point representation which now becomes the general numeric representation in CPU or GPU.

Original: As we determined in the DFA, BDFA’s feedback weight, B i can be similarly deﬁned by a modiﬁcation of the equation (7).
Revised: As we determined in the DFA, BDFA’s feedback weight, B i can be similarly deﬁned by a modiﬁcation of the equation (7).

Original: B i is determined as
Revised: B i is determined as

Original: [Equation]
Revised: [Equation]

Original: , when the sign ( · ) is the function which indicates the sign value of each element.
Revised: , when the sign ( · ) is the function which indicates the sign value of each element.

Original: The difference between equation (7) and (9) is only whether the sign ( · ) is applied or not.
Revised: The difference between equation (7) and (9) is only whether the sign ( · ) is applied or not.

Original: By applying equation (9),
Revised: By applying equation (9),

Original: BDFA shows faster and stable training convergence compared with the random initialization case.
Revised: BDFA shows faster and stable training convergence compared with the random initialization case.

Original: The effect of the binarization and initialization will be discussed in section 4.
Revised: The effect of the binarization and initialization will be discussed in section 4.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we compared the training accuracy of the conventional BP and suggested training algorithm.
Revised: In this section, we compared the training accuracy of the conventional BP and suggested training algorithm.

Original: We measured the relative accuracy by training CNN from the scratch in CIFAR-10 and
Revised: We measured the relative accuracy by training CNN from the scratch in CIFAR-10 and

Original: CIFAR-100 dataset (Krizhevsky (2009)).
Revised: CIFAR-100 dataset (Krizhevsky (2009)).

Original: We use thesame network conﬁguration as described in 
Revised: We used the network conﬁguration as described in Figure 3.

Original: Figure 3 and it follows the VGG-16 (Simonyan & Zisserman (2015)) conﬁguration.
Revised: The base network follows the VGG-16 (Simonyan & Zisserman (2015)) conﬁguration, but has one additional FC layer.

Original: It consists of 13 convolutional layers with BN and ReLU activation functions, followed by three FC layers without BN.
Revised: To sum up, it consists of 13 convolutional layers with BN and ReLU activation functions, followed by three FC layers without BN.

Original: The number of neurons in the last FC layer is determined by the number of classiﬁcation categories in a different dataset.
Revised: The number of neurons in the last FC layer is determined by the number of classiﬁcation categories in each different dataset.

Original: In the BP based approach, both the convolutional layers and the FC layers are trained by using BP.
Revised: In the BP based approach, both the convolutional layers and the FC layers are trained by using BP.

Original: In contrast, the training method of the last three FC layers is substituted with the DFA or the BDFA to measure the performance of the proposed training algorithm.
Revised: In contrast, the training method of the last three FC layers is substituted with the DFA or the BDFA to measure the performance of the proposed training algorithm.

Original: The simulation was based on mini-batch gradient descent with the batch size 100.
Revised: The simulation was based on minibatch gradient descent with the batch size 100 and uses momentum(Sutskever et al.

Original: There were many optimization algorithms such as RMSprop (Tieleman & Hinton (2012)), Adagrad (Duchi et al. and Adam (Kingma & Ba (2014)).
Revised: for the optimization method.

Original: We adopted the momentum (Sutskever et al.
Revised: The parameters of the network is initialized as introduced by He et al.

Original: (2013)), which is the simplest optimization method compared with other optimization algorithms. [Table]
Revised: (2015), and the learning rate decay and the weight decay method is adopted. Other hyper parameters are not changed for fair comparison.

Original: 4. CNN T RAINING FROM THE S CRATCH
Revised: 4. CNN T RAINING FROM THE S CRATCH

Original: The CNN training with the DFA and BDFA, are renamed as CDFA and CBDFA respectively for the simple explanation.
Revised: The CNN training with the DFA and BDFA, are renamed as CDFA and CBDFA respectively for the simple explanation.

Original: Table 1 shows Top5 and Top1 test accuracy after the CNN training is done in the two different CIFAR dataset.
Revised: Table 1 shows Top5 and Top1 test accuracy after the CNN training is done in the two different CIFAR datasets.

Revised: In this simulation, there is no data augmentation to make an environment which has a limited dataset.

Revised: This condition can examine whether the algorithm is robust to the training in the small dataset.

Original:  As a result, both the CDFA and the CBDFA show higher test accuracy compared with theconventional BP even though the feedback weight is randomly initialized. In CIFAR-
Revised: To sum up, only 50,000 images in the CIFAR-10 and CIFAR-100 is only used for DNN training and the other 10,000 images are tested to evaluate the test accuracy. As a result, both the CDFA and the CBDFA show higher test accuracy compared with the

Original: [Table]
Revised: [Table]

Revised: conventional BP

Revised: even though the feedback weight is randomly initialized.

Original:  the CDFA and CBDFA are 7.5% and 8.3% higher in Top1 test accuracy than the BP respectively.
Revised: In CIFAR-10, the CDFA and CBDFA are 7.5% and 8.3% higher in Top1 test accuracy than the BP respectively.

Original: The accuracy improvement by the CDFA and the CBDFA seems much more remarkable in CIFAR-100.
Revised: The accuracy improvement by the CDFA and the CBDFA seems much more remarkable in CIFAR-100.

Original: As shown in Figure 4, the training curve of the CDFA and CBDFA is much slower, but they achieve 21.3% and 19.6% better performance respectively compared with the BP.
Revised: As shown in Figure 4, the training curve of the CDFA and CBDFA is much slower, but they achieve 21.3% and 19.6% better performance respectively compared with the BP.

Original: However, the feedback weight with the random initialization has critical problems for training.
Revised: However, the feedback weight with the random initialization has critical problems for training.

Original: One of the problems is the slow training curve described in Figure 4.
Revised: One of the problems is the slow training curve described in Figure 4.

Original: DFA requires time to be adaptive to the randomly initialized feedback weights, so it takes a long latency to be converged.
Revised: DFA requires time to be adaptive to the randomly initialized feedback weights, so it takes a long latency to be converged.

Original: In the BP approach, we generally take the larger learning rate to make the training faster.
Revised: In the BP approach, we generally take the larger learning rate to make the training faster.

Original: However, the test accuracy of the DFA and BDFA is swung up and down dramatically when the large learning rate is applied.
Revised: However, the test accuracy of the DFA and BDFA is swung up and down dramatically when the large learning rate is applied.

Original: Moreover, it still spends the long time to converge.
Revised: Moreover, it still spends a long time to converge.

Original: In this problem, the initialization with the equation (7) and (9) can be useful to solve the learning speed and stability problem.
Revised: In this problem, the initialization with the equation (7) and (9) can be useful to solve the learning speed and stability problem.

Original: After the feedback weight is initialized by the proposed equations, it shows faster and more stable convergence characteristic as shown in Figure 5.
Revised: After the feedback weight is initialized by the proposed equations, it shows faster and more stable convergence characteristic as shown in Figure 5.

Original: When the proposed initialization method is combined with the large learning rate, it shows the best training performance compared with the other results.
Revised: When the proposed initialization method is combined with the large learning rate, it shows the best training performance compared with the other results.

Revised: When the dataset is augmented with the ﬂipping and cropping, the training performance of the BP and CBDFA becomes higher than before.

Revised: The simulation uses the large learning rate, and CBDFA

Revised: [Table]

Revised: takes the initialization with equation (9).

Revised: In table 2, the performance of the CBDFA shows the highest accuracy compared with not only BP but also the training suggested by Hoffer et al.

Original:  There are some interesting observations in our simulation.
Revised: It trains only the convolutional layers, and the parameters of the FC layers are ﬁxed. Even considering the data augmentation, CBDFA still shows higher training accuracy compared with the other two methods. As a result, CBDFA seems robust to the size of the dataset. There are some interesting observations in our simulation.

Original: First of all, the CBDFA shows negligible accuracy degradation compared with the CDFA based training.
Revised: First of all, the CBDFA shows negligible accuracy degradation compared with the CDFA based training.

Original: Sometimes, the CBDFA has a rather better performance than the CDFA case.
Revised: Sometimes, the CBDFA has a rather better performance than the CDFA case.

Original: Refer to the learning curve described in both Figureand Figure 5, the CBDFA’s learning speed is slightly degraded, but the ﬁnal training results are approximately the same.
Revised: Refer to the learning curve described in both Figureand Figure 5, the CBDFA’s learning speed is slightly degraded, but the ﬁnal training results are approximately the same.

Original: Therefore, CBDFA can improve the training performance and take the hardware beneﬁts such as smaller memory bandwidth by adopting binarized feedback weights.
Revised: Therefore, CBDFA can improve the training performance and take the hardware beneﬁts such as smaller memory bandwidth by adopting binarized feedback weights.

Original: The second one is the effect of the BN.
Revised: The second one is the effect of the BN.

Original: Even though the suggested initialization method achieves the fast and stable training, it has a little accuracy degradation compared with the random initialization method.
Revised: Even though the suggested initialization method achieves the fast and stable training, it has a little accuracy degradation compared with the random initialization method.

Original: This accuracy degradation can be reduced when the BN layer is added after the FC layer.
Revised: This accuracy degradation can be reduced when the BN layer is added after the FC layer.

Original: This result is counter characteristic compared with the BP case because the BP shows the worse training result when the BN layer is followed right after the FC layer.
Revised: This result is counter characteristic compared with the BP case because the BP shows the worse training result when the BN layer is followed right after the FC layer.

Original: To sum up, the equation (7) and (9) are much more powerful when the BN is followed after the convolutional layer.
Revised: To sum up, the equation (7) and (9) are much more powerful when the BN is followed after the convolutional layer.

Original: 4. FC F INE - TUNING : O BJECT T RACKINGWITH O NLINE L EARNING
Revised: 4. E XAMPLE OF O NLINE L EARNING WITH S MALL D ATASET : O BJECT T RACKING

Original: Nam & Han (2016) suggested online FC learning based object tracking algorithm, MDNet.
Revised: Nam & Han (2016) suggested online FC learning based object tracking algorithm, MDNet.

Original: The online learning concept in MDNet has derived many other algorithms such as BranchOut (Han et al.
Revised: The online learning concept in MDNet has derived many other algorithms such as BranchOut (Han et al.

Original: and ADNet (Yun et al.
Revised: and ADNet (Yun et al.

Original: In the MDNet, both the convolutional layers and FC layers are pre-trained with the VOT (Kristan et al. , Kristan et al. and
Revised: In the MDNet, both the convolutional layers and FC layers object tracking dataset.

Original: OTB (Wu et al.

Original: object tracking dataset.

Original: However, the last layer of the FC layers is randomly initialized for the new tracking task.
Revised: However, the last layer of the FC layers is randomly initialized for the new tracking task.

Original: The convolutional layers do not need to be trained during the tracking but the FC 
Revised: The convolutional layers do not need to be trained during the tracking but the FC layers are ﬁne-tuned by using the BP.

Revised: To apply BDFA to FC online learning with the small dataset, we replace the BP by the BDFA for FC layers in the MDNet.

Original: We compared the object tracking performance by drawing the precision and the success plots of the one-pass evaluation (OPE) in the OTB dataset.
Revised: We compared the object tracking performance by drawing the precision and the success plots of the one-pass evaluation (OPE) in the OTB dataset.

Original: As shown in Figure 6, the object tracking with the
Revised: As shown in Figure 6, the object tracking with the

Original: BDFA based online learning shows similar performance compared with the BP case.
Revised: BDFA based online learning shows similar performance compared with the BP case.

Original: However, the
Revised: However, the

Original: BDFA shows better performance than BP when the batch size becomes smaller.
Revised: BDFA shows better performance than BP when the batch size becomes smaller.

Original: Since the object tracking application is very sensitive to the online learning speed and the BDFA has a chance to be computed in parallel, BDFA based online learning becomes much more beneﬁcial than conventional
Revised: Since the object tracking application is very sensitive to the online learning speed and the BDFA has a chance to be computed in parallel, BDFA based online learning becomes much more beneﬁcial than conventional

Original: BP.
Revised: BP.

Original: Moreover, the BDFA can dramatically reduce the required data transaction in error propagation because of the fewer neuron interconnections and the binarized feedback weights.
Revised: Moreover, the BDFA can dramatically reduce the required data transaction in error propagation because of the fewer neuron interconnections and the binarization.

Revised: As pointed out in the paper, Han et al.

Original: Therefore, the effect of the BDFA can be maximized when the online learning is done in the devices which havethe limited memory bandwidth and less computing resources.
Revised: (2018), BP based online learning is inefﬁcient for online learning in the devices which have limited memory bandwidth, computing resources, and small dataset. In this case, the effect of the BDFA can be maximized because of its proﬁts.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we propose the new DNN learning algorithms to maximize the computation efﬁciency without accuracy degradation.
Revised: In this work, we propose the new DNN learning algorithms to maximize the computation efﬁciency without accuracy degradation.

Original: We adopt one of the training method, DFA and combine it with the conventional BP.
Revised: We adopt one of the training method, DFA and combine it with the conventional BP.

Original: The combination of the DFA and the BP shows much better test accuracy in CNN training through the simulation in the CIFAFR-10 and CIFAR-100 dataset.
Revised: The combination of the DFA and the BP shows much better test accuracy in CNN training through the simulation in the CIFAFR-10 and CIFAR-100 dataset.

Original: BDFA takes one step further, binarizing the feedback weight while maintaining the similar performance compared with the full-precision DFA.
Revised: BDFA takes one step further, binarizing the feedback weight while maintaining a similar performance compared with the full-precision DFA.

Original: The stability problem induced in the DFA and BDFA simulation (Figure 5) can be solved by the new feedback weight initialization method, equation (7) and (9).
Revised: The stability problem induced in the DFA and BDFA simulation (Figure 5) can be solved by the new feedback weight initialization method, equation (7) and (9).

Original: The BDFA is also simulated in the object tracking application, and it shows better tracking results compared with the conventional BP based online FC tuning.
Revised: The BDFA is also simulated in the object tracking application, and it shows better tracking results compared with the conventional BP based online FC tuning.

Original: In this research, we can see that the DFA can reduce the computational complexity and achieve better training performance than conventional training method.
Revised: In this research, we can see that the DFA can reduce the computational complexity and achieve better training performance than conventional training method.

Original: However, the feedback path of the
Revised: However, the feedback path of the

Original: DFA cannot still be applied directly into convolutional layer because of the signiﬁcant accuracy degradation.
Revised: DFA cannot still be applied directly into convolutional layer because of the signiﬁcant accuracy degradation.

Original: To break the limitation of the current research, We will continue the research about expanding the usage of the DFA to not only convolutional layers but also other RNN networks such as LSTM (Hochreiter & Schmidhubers (1997)) and GRU (Chung et al.
Revised: To break the limitation of the current research, We will continue the research about expanding the usage of the DFA to not only convolutional layers but also other RNN networks such as LSTM (Hochreiter & Schmidhubers (1997)) and GRU (Chung et al.