Original: A CCELERATING DNN T RAINING THROUGH S ELECTIVE L OCALIZED L
Revised: A CCELERATING DNN T RAINING THROUGH S ELECTIVE L OCALIZED L

Original: EARNING
Revised: EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Training Deep Neural Networks (DNNs) places immense compute requirements on the underlying hardware platforms, expending large amounts of time and energy.
Revised: Training Deep Neural Networks (DNNs) places immense compute requirements on the underlying hardware platforms, expending large amounts of time and energy.

Original: We propose LoCal+SGD , a new algorithmic approach to accelerate DNN training by selectively combining localized or Hebbian learning within a Stochastic
Revised: We propose LoCal+SGD , a new algorithmic approach to accelerate DNN training by selectively combining localized or Hebbian learning within a Stochastic

Original: Gradient Descent (SGD) based training framework .
Revised: Gradient Descent (SGD) based training framework .

Original: Back-propagation is a computationally expensive process that requires 2 Generalized Matrix Multiply (GEMM) operations to compute the error and weight gradients for each layer.
Revised: Back-propagation is a computationally expensive process that requires 2 Generalized Matrix Multiply (GEMM) operations to compute the error and weight gradients for each layer.

Original: We alleviate this by selectively updating some layers’ weights using localized learning rules that require only 1 GEMM operation per layer.
Revised: We alleviate this by selectively updating some layers’ weights using localized learning rules that require only 1 GEMM operation per layer.

Original: Further, since the weight update is performed during the forward pass itself, the layer activations for the mini-batch do not need to be stored until the backward pass, resulting in a reduced memory footprint.
Revised: Further, since the weight update is performed during the forward pass itself, the layer activations for the mini-batch do not need to be stored until the backward pass, resulting in a reduced memory footprint.

Original: Localized updates can substantially boost training speed, but need to be used selectively and judiciously in order to preserve accuracy and convergence.
Revised: Localized updates can substantially boost training speed, but need to be used selectively and judiciously in order to preserve accuracy and convergence.

Original: We address this challenge through the design of a Learning Mode Selection Algorithm , where all layers start with SGD, and as epochs progress, layers gradually transition to localized learning.
Revised: We address this challenge through the design of a Learning Mode Selection Algorithm , where all layers start with SGD, and as epochs progress, layers gradually transition to localized learning.

Original: Speciﬁcally, for each epoch, the algorithm identiﬁes a
Revised: Speciﬁcally, for each epoch, the algorithm identiﬁes a

Original: Localized → SGD transition layer, which delineates the network into two regions.
Revised: Localized → SGD transition layer, which delineates the network into two regions.

Original: Layers before the transition layer use localized updates, while the transition layer and later layers use gradient-based updates.
Revised: Layers before the transition layer use localized updates, while the transition layer and later layers use gradient-based updates.

Original: The trend in the weight updates made to the transition layer across epochs is used to determine how the boundary between
Revised: The trend in the weight updates made to the transition layer across epochs is used to determine how the boundary between

Original: SGD and localized updates is shifted in future epochs.
Revised: SGD and localized updates is shifted in future epochs.

Original: We also propose a low-cost weak supervision mechanism by controlling the learning rate of localized updates based on the overall training loss.
Revised: We also propose a low-cost weak supervision mechanism by controlling the learning rate of localized updates based on the overall training loss.

Original: We applied LoCal+SGD to 8 image recognition
Revised: We applied LoCal+SGD to 8 image recognition

Original: CNNs (including ResNet50 and MobileNetV2) across 3 datasets (Cifar10, Cifar and ImageNet).
Revised: CNNs (including ResNet50 and MobileNetV2) across 3 datasets (Cifar10, Cifar and ImageNet).

Original: Our measurements on a Nvidia GTX 1080Ti
Revised: Our measurements on a Nvidia GTX 1080Ti

Original: GPU demonstrate upto 1.5 × improvement in end-to-end training time with ∼
Revised: GPU demonstrate upto 1.5 × improvement in end-to-end training time with ∼

Original: 0.5% loss in Topclassiﬁcation accuracy
Revised: 0.5% loss in Topclassiﬁcation accuracy

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep Neural Networks (DNNs) have achieved continued success in many application domains involving images (Krizhevsky et al., 2017), videos (Ng et al., 2015), text (Zhou et al., 2015) and natural language (Goldberg & Hirst, 2017).
Revised: Deep Neural Networks (DNNs) have achieved continued success in many application domains involving images (Krizhevsky et al., 2017), videos (Ng et al., 2015), text (Zhou et al., 2015) and natural language (Goldberg & Hirst, 2017).

Original: However training state-of-the-art DNN models is computationally quite challenging, often requiring exa-FLOPs of compute as the models are quite complex and need to be trained using large datasets.
Revised: However training state-of-the-art DNN models is computationally quite challenging, often requiring exa-FLOPs of compute as the models are quite complex and need to be trained using large datasets.

Original: Despite rapid improvements in the capabilities of GPUs and the advent of specialized accelerators, training large models using current platforms is still quite expensive and often takes days to even weeks.
Revised: Despite rapid improvements in the capabilities of GPUs and the advent of specialized accelerators, training large models using current platforms is still quite expensive and often takes days to even weeks.

Original: In this work, we aim to reduce the computational complexity of DNN training through a new algorithmic approach called LoCal+SGD 1 , which alleviates the key performance bottlenecks in Stochastic Gradient Descent (SGD) through selective use of localized or Hebbian learning .
Revised: In this work, we aim to reduce the computational complexity of DNN training through a new algorithmic approach called LoCal+SGD 1 , which alleviates the key performance bottlenecks in Stochastic Gradient Descent (SGD) through selective use of localized or Hebbian learning .

Original: Computational Bottlenecks in DNN Training.
Revised: Computational Bottlenecks in DNN Training.

Original: DNNs are trained in a supervised manner using gradient-descent based cost minimization techniques such as SGD (Bottou, 2010) or Adam (Kingma &
Revised: DNNs are trained in a supervised manner using gradient-descent based cost minimization techniques such as SGD (Bottou, 2010) or Adam (Kingma &

Original: Ba, 2015).
Revised: Ba, 2015).

Original: The training inputs (typically grouped into minibatches) are iteratively forward propagated ( FP ) and back propagated ( BP ) through the DNN layers to compute weight updates that push the network parameters in the direction that decreases the overall classiﬁcation loss.
Revised: The training inputs (typically grouped into minibatches) are iteratively forward propagated ( FP ) and back propagated ( BP ) through the DNN layers to compute weight updates that push the network parameters in the direction that decreases the overall classiﬁcation loss.

Original: Back-propagation is computationally expensive, accounting for 65-75% of the total training time on
Revised: Back-propagation is computationally expensive, accounting for 65-75% of the total training time on

Original: GPUs.
Revised: GPUs.

Original: This is attributed to two key factors: (i) BP involves 2 Generalized Matrix Multiply (GEMM) operations, one to propagate the error across layers and the other to compute the weight gradients, and (ii) when training on distributed systems using data/model parallelism(Dean et al., 2012b; Krizhevsky et al., 2012), aggregation of weight gradients/errors across devices incurs signiﬁcant communication overhead.
Revised: This is attributed to two key factors: (i) BP involves 2 Generalized Matrix Multiply (GEMM) operations, one to propagate the error across layers and the other to compute the weight gradients, and (ii) when training on distributed systems using data/model parallelism(Dean et al., 2012b; Krizhevsky et al., 2012), aggregation of weight gradients/errors across devices incurs signiﬁcant communication overhead.

Original: Further, BP through auxiliary ops such as batch normalization are also more expensive than FP .
Revised: Further, BP through auxiliary ops such as batch normalization are also more expensive than FP .

Original: Prior Efforts on Efﬁcient DNN Training.
Revised: Prior Efforts on Efﬁcient DNN Training.

Original: Prior research efforts to improve DNN training time can be grouped into a few directions.
Revised: Prior research efforts to improve DNN training time can be grouped into a few directions.

Original: One group of efforts enable larger scales of parallelism in
Revised: One group of efforts enable larger scales of parallelism in

Original: DNN training through learning rate tuning (You et al., 2017a; Goyal et al., 2017; You et al., 2017b) and asynchronous weight updates (Dean et al., 2012a).
Revised: DNN training through learning rate tuning (You et al., 2017a; Goyal et al., 2017; You et al., 2017b) and asynchronous weight updates (Dean et al., 2012a).

Original: Another class of efforts employ importancebased sample selection during training, wherein ‘easier’ training samples are selectively discarded to improve runtime (Jiang et al., 2019; Zhang et al., 2019).
Revised: Another class of efforts employ importancebased sample selection during training, wherein ‘easier’ training samples are selectively discarded to improve runtime (Jiang et al., 2019; Zhang et al., 2019).

Original: Finally, model quantization (Sun et al., 2019) and pruning (Lym et al., 2019) can lead to signiﬁcant runtime beneﬁts during training by enabling the use of reduced-bitwidth processing elements.
Revised: Finally, model quantization (Sun et al., 2019) and pruning (Lym et al., 2019) can lead to signiﬁcant runtime beneﬁts during training by enabling the use of reduced-bitwidth processing elements.

Original: LoCal+SGD :
Revised: LoCal+SGD :

Original: Combining SGD with Localized Learning.
Revised: Combining SGD with Localized Learning.

Original: Complementary to the aforementioned efforts, we propose a new approach, LoCal+SGD , to alleviate the performance bottlenecks in DNN training, while preserving model accuracy.
Revised: Complementary to the aforementioned efforts, we propose a new approach, LoCal+SGD , to alleviate the performance bottlenecks in DNN training, while preserving model accuracy.

Original: Our hybrid approach combines Hebbian or localized learning (Hebb) with SGD by selectively applying it in speciﬁc layers and epochs.
Revised: Our hybrid approach combines Hebbian or localized learning (Hebb) with SGD by selectively applying it in speciﬁc layers and epochs.

Original: Localized learning rules (Hebb; Oja, 1982; Zhong, 2005) utilize a single feed-forward weight update to learn the feature representations, eschewing BP .
Revised: Localized learning rules (Hebb; Oja, 1982; Zhong, 2005) utilize a single feed-forward weight update to learn the feature representations, eschewing BP .

Original: Careful formulation of the localized learning rule can result in ∼ 2 × computation savings compared to SGD and also signiﬁcantly reduces memory footprint as activations from FP needed not be retained until BP .
Revised: Careful formulation of the localized learning rule can result in ∼ 2 × computation savings compared to SGD and also signiﬁcantly reduces memory footprint as activations from FP needed not be retained until BP .

Original: The reduction in memory footprint can in turn allow increasing the batch size during training, which leads to further runtime savings due to better compute utilization and reduced communication costs.
Revised: The reduction in memory footprint can in turn allow increasing the batch size during training, which leads to further runtime savings due to better compute utilization and reduced communication costs.

Original: It is worth noting that localized learning has been actively explored in the context of unsupervised learning (Chen et al., 2020; van den Oord et al., 2018; Hénaff et al., 2019). 
Revised: It is worth noting that localized learning has been actively explored in the context of unsupervised learning (Chen et al., 2020; van den Oord et al., 2018; Hénaff et al., 2019). Further, there has been active research efforts on neuro-scientiﬁc learning rules (Lee et al., 2015; Nøkland, 2016).

Original: Our work is orthogonal to such efforts and represents a new application of localized learning in a fully supervised context, wherein we selectively combine it within an SGD framework to achieve computational savings.
Revised: Our work is orthogonal to such efforts and represents a new application of localized learning in a fully supervised context, wherein we selectively combine it within an SGD framework to achieve computational savings.

Original: Preserving model accuracy and convergence with LoCal+SGD requires localized updates to be applied judiciously i.e., only to selected layers in certain epochs.
Revised: Preserving model accuracy and convergence with LoCal+SGD requires localized updates to be applied judiciously i.e., only to selected layers in certain epochs.

Original: We address this challenge through the design of a learning mode selection algorithm .
Revised: We address this challenge through the design of a learning mode selection algorithm .

Original: At the start training, the selection algorithm initializes the learning mode of all layers to SGD, and as training progresses determines the layers that transition to localized learning.
Revised: At the start training, the selection algorithm initializes the learning mode of all layers to SGD, and as training progresses determines the layers that transition to localized learning.

Original: Speciﬁcally, for each epoch, the algorithm identiﬁes a Localized → SGD transition layer, which delineates the network into two regions.
Revised: Speciﬁcally, for each epoch, the algorithm identiﬁes a Localized → SGD transition layer, which delineates the network into two regions.

Original: Layers before the transition layer use localized updates, while subsequent layers use gradient-based updates.
Revised: Layers before the transition layer use localized updates, while subsequent layers use gradient-based updates.

Original: This allows BP to stop at the transition layer, as layers before it have no use for the back-propagated errors.
Revised: This allows BP to stop at the transition layer, as layers before it have no use for the back-propagated errors.

Original: The algorithm takes advantage of the magnitude of the weight updates of the Localized → SGD transition layer in deciding the new position of the boundary every epoch.
Revised: The algorithm takes advantage of the magnitude of the weight updates of the Localized → SGD transition layer in deciding the new position of the boundary every epoch.

Original: Further, we provide weak supervision by tweaking the learning rate of locally updated layers based on overall training loss.
Revised: Further, we provide weak supervision by tweaking the learning rate of locally updated layers based on overall training loss.

Original: Contributions : To the best of our knowledge, LoCal+SGD is the ﬁrst effort that combines localized learning (an unsupervised learning technique) within a supervised SGD context to reduced computational costs while maintaining classiﬁcation accuracy.
Revised: Contributions : To the best of our knowledge, LoCal+SGD is the ﬁrst effort that combines localized learning (an unsupervised learning technique) within a supervised SGD context to reduced computational costs while maintaining classiﬁcation accuracy.

Original: This favorable tradeoff is achieved by LoCal+SGD through a Learning Mode Selection Algorithm that applies localized learning to selected layers and epochs.
Revised: This favorable tradeoff is achieved by LoCal+SGD through a Learning Mode Selection Algorithm that applies localized learning to selected layers and epochs.

Original: Further improvement is achieved through the use of weak supervision by modulating the learning rate of locally updated layers based on the overall training loss.
Revised: Further improvement is achieved through the use of weak supervision by modulating the learning rate of locally updated layers based on the overall training loss.

Original: Across 8 image recognition CNNs (including ResNet50 and MobileNet) and 3 datasets (Cifar10, Cifar and ImageNet), we demonstrate that LoCal+SGD achieves up to 1.5 × improvement in training time with ∼ 0.5% Top-1 accuracy loss on a Nvidia GTX 1080Ti GPU.
Revised: Across 8 image recognition CNNs (including ResNet50 and MobileNet) and 3 datasets (Cifar10, Cifar and ImageNet), we demonstrate that LoCal+SGD achieves up to 1.5 × improvement in training time with ∼ 0.5% Top-1 accuracy loss on a Nvidia GTX 1080Ti GPU.

Original:  LoCal+SGD : C OMBINING SGD WITH S ELECTIVE L OCALIZED L EARNING
Revised:  LoCal+SGD : C OMBINING SGD WITH S ELECTIVE L OCALIZED L EARNING

Original: The key idea in LoCal+SGD is to apply localized learning to selected layers and epochs during DNN training to improve the overall execution time, without incurring loss in accuracy.
Revised: The key idea in LoCal+SGD is to apply localized learning to selected layers and epochs during DNN training to improve the overall execution time, without incurring loss in accuracy.

Original: The following components are critical to the effectiveness of LoCal+SGD :
Revised: The following components are critical to the effectiveness of LoCal+SGD :

Original: Stage
Revised: Stage

Original: Localized Updates
Revised: Localized Updates

Original: SGD-Based Updates
Revised: SGD-Based Updates

Original: FP 𝑧 𝑙 = 𝐺𝐸𝑀𝑀(𝑎 𝑙−1 , 𝑊 𝑙 ) (2)
Revised: FP 𝑧 𝑙 = 𝐺𝐸𝑀𝑀(𝑎 𝑙−1 , 𝑊 𝑙 ) (2)

Original: Update Stage ∆𝑊 𝑙 = 𝐺𝐸𝑀𝑀 𝑎 𝑙−1 , 𝑧 𝑙 (3)
Revised: Update Stage ∆𝑊 𝑙 = 𝐺𝐸𝑀𝑀 𝑎 𝑙−1 , 𝑧 𝑙 (3)

Original: 𝑊 𝑙 =
Revised: 𝑊 𝑙 =

Original: 𝑊 𝑙 + η ∆𝑊 𝑙 | ∆𝑊 𝑙 | (4) 𝛿 𝑙 = 𝐺𝐸𝑀𝑀 𝛿 𝑙+1 ,𝑊 𝑙 (5) ∆𝑊 𝑙 = 𝐺𝐸𝑀𝑀 𝑎 𝑙−1 , 𝛿 𝑙 (6) 𝑊 𝑙 = 𝑊 𝑙 +η∆𝑊 𝑙 (7)
Revised: 𝑊 𝑙 + η ∆𝑊 𝑙 | ∆𝑊 𝑙 | (4) 𝛿 𝑙 = 𝐺𝐸𝑀𝑀 𝛿 𝑙+1 ,𝑊 𝑙 (5) ∆𝑊 𝑙 = 𝐺𝐸𝑀𝑀 𝑎 𝑙−1 , 𝛿 𝑙 (6) 𝑊 𝑙 = 𝑊 𝑙 +η∆𝑊 𝑙 (7)

Original: [Figure]
Revised: [Figure]

Original: 𝐥 : Filter Weight of Layer L
Revised: 𝐥 : Filter Weight of Layer L

Original: [Figure]
Revised: [Figure]

Original: 𝐥 ∶ Pre-Activation o/p of Layer L
Revised: 𝐥 ∶ Pre-Activation o/p of Layer L

Original: [Figure]
Revised: [Figure]

Original: 𝐥 : Post-Activation o/p of Layer
Revised: 𝐥 : Post-Activation o/p of Layer

Original: [Figure]
Revised: [Figure]

Original: 𝐥 :
Revised: 𝐥 :

Original: [Figure]
Revised: [Figure]

Original: at
Revised: at

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: NOTE
Revised: NOTE

Original: [Figure]
Revised: [Figure]

Original: GEMM includes operations in convolutional
Revised: GEMM includes operations in convolutional

Original: [Figure]
Revised: [Figure]

Original: fully-connected layers
Revised: fully-connected layers

Original: • Localized Learning Rule Formulation.
Revised: • Localized Learning Rule Formulation.

Original: We formulate a computationally efﬁcient localized learning rule and highlight the clear runtime beneﬁts when compared to SGD.
Revised: We formulate a computationally efﬁcient localized learning rule and highlight the clear runtime beneﬁts when compared to SGD.

Original: • Learning Mode Selection Algorithm.
Revised: • Learning Mode Selection Algorithm.

Original: We propose a learning mode selection algorithm that chooses between localized learning and SGD-based learning for each layer in every epoch, based on the potential impact on accuracy and computational beneﬁts.
Revised: We propose a learning mode selection algorithm that chooses between localized learning and SGD-based learning for each layer in every epoch, based on the potential impact on accuracy and computational beneﬁts.

Original: • Weak Supervision.
Revised: • Weak Supervision.

Original: We propose a weak supervision technique, which comprises of a lowcost supervision signal communicated to the localized learning layers in each epoch.
Revised: We propose a weak supervision technique, which comprises of a lowcost supervision signal communicated to the localized learning layers in each epoch.

Original: The signal modulates the learning rates of these layers based on the rate of change of the overall classiﬁcation loss.
Revised: The signal modulates the learning rates of these layers based on the rate of change of the overall classiﬁcation loss.

Original: In the following sub-sections, we describe the salient aspects of these components in greater detail.
Revised: In the following sub-sections, we describe the salient aspects of these components in greater detail.

Original: 2. E FFICIENT L OCALIZED L EARNING
Revised: 2. E FFICIENT L OCALIZED L EARNING

Original: Localized learning has been extensively explored in the context of unsupervised learning, demonstrating success on small (<= 3 layer) networks using relatively simpler datasets ( e.g. MNIST,
Revised: Localized learning has been extensively explored in the context of unsupervised learning, demonstrating success on small (<= 3 layer) networks using relatively simpler datasets ( e.g. MNIST,

Original: Cifar-10) (LeCun & Cortes, 2010; Krizhevsky et al., a)) with an accuracy gap that is yet to be bridged on larger datasets ( e.g. ResNet50 or MobileNetV2 on ImageNet (Deng et al., 2009)).
Revised: Cifar-10) (LeCun & Cortes, 2010; Krizhevsky et al., a)) with an accuracy gap that is yet to be bridged on larger datasets ( e.g. ResNet50 or MobileNetV2 on ImageNet (Deng et al., 2009)).

Original: First proposed in (Hebb), the key intuition behind localized learning rules is to encourage correlations between neurons that have similar activation patterns.
Revised: First proposed in (Hebb), the key intuition behind localized learning rules is to encourage correlations between neurons that have similar activation patterns.

Original: Equation 1 depicts the Hebbian weight update proposed in (Hebb), for a synapse with weight W , connecting a pair of input and output neurons whose activation values are represented by x and y respectively, with η as the learning rate.
Revised: Equation 1 depicts the Hebbian weight update proposed in (Hebb), for a synapse with weight W , connecting a pair of input and output neurons whose activation values are represented by x and y respectively, with η as the learning rate.

Original: [Equation]
Revised: [Equation]

Original: Considerable research has gone into evolving this equation over the years to improve the performance of localized learning (Oja, 1982; Zhong, 2005).
Revised: Considerable research has gone into evolving this equation over the years to improve the performance of localized learning (Oja, 1982; Zhong, 2005).

Original: However, many of the proposed rules are computationally complex, or are difﬁcult to parallelize on modern hardware platforms such as GPUs and TPUs.
Revised: However, many of the proposed rules are computationally complex, or are difﬁcult to parallelize on modern hardware platforms such as GPUs and TPUs.

Original: Since our primarily goal is improving DNN training time, we adopt the computationally simple localized learning rule presented in Equation 1.
Revised: Since our primarily goal is improving DNN training time, we adopt the computationally simple localized learning rule presented in Equation 1.

Original: The learning rule in Equation 1 assumes a distinct synapse between each input and output neuron pair.
Revised: The learning rule in Equation 1 assumes a distinct synapse between each input and output neuron pair.

Original: While its application to fully-connected (fc) layers is straightforward, we need to consider the sharing of weights between neuron pairs in convolutional (conv) layers.
Revised: While its application to fully-connected (fc) layers is straightforward, we need to consider the sharing of weights between neuron pairs in convolutional (conv) layers.

Original: For updating a shared weight of a conv layer, we calculate the individual updates due to each pair of pre- and post-synaptic neurons sharing the weight and sum all such updates.
Revised: For updating a shared weight of a conv layer, we calculate the individual updates due to each pair of pre- and post-synaptic neurons sharing the weight and sum all such updates.

Original: This essentially reduces to a convolution operation between the input and output activations of the layer and can be expressed by Equationin Figure 1.
Revised: This essentially reduces to a convolution operation between the input and output activations of the layer and can be expressed by Equationin Figure 1.

Original: For further computational efﬁciency improvement, unlike Equation 1, we consider the pre-activation-function values of the outputs i.e., z l instead of their post activation value a l .
Revised: For further computational efﬁciency improvement, unlike Equation 1, we consider the pre-activation-function values of the outputs i.e., z l instead of their post activation value a l .

Original: Further, we normalize the localized update values as shown in Equation 4 of Figure 1, as it was observed to achieve better convergence in practice.
Revised: Further, we normalize the localized update values as shown in Equation 4 of Figure 1, as it was observed to achieve better convergence in practice.

Original: Overall, we utilize Equations 3 and 4 from Figure 1 to perform the weight updates in all layers that are earlier than the Localized → SGD transition layer during a certain epoch.
Revised: Overall, we utilize Equations 3 and 4 from Figure 1 to perform the weight updates in all layers that are earlier than the Localized → SGD transition layer during a certain epoch.

Original: All other layers continue to be updated using SGD-based BP , expressed by Equations 5-7 in Figure 1.
Revised: All other layers continue to be updated using SGD-based BP , expressed by Equations 5-7 in Figure 1.

Original: SGD updates are applied to batch-normalization layers present after the Localized → SGD transition layer, and are otherwise skipped.
Revised: SGD updates are applied to batch-normalization layers present after the Localized → SGD transition layer, and are otherwise skipped.

Original: Clearly, Equation 3 has the same computational complexity as Equation 6 of SGD-based
Revised: Clearly, Equation 3 has the same computational complexity as Equation 6 of SGD-based

Original: BP for conv and fc layers.
Revised: BP for conv and fc layers.

Original: Thus, from Figure 1, we can directly infer that our localized learning rule will be considerable faster than SGD-based BP .
Revised: Thus, from Figure 1, we can directly infer that our localized learning rule will be considerable faster than SGD-based BP .

Original: In practice, we measured this improvement to be more than 2 × on a NVIDIA GTX 1080Ti
Revised: In practice, we measured this improvement to be more than 2 × on a NVIDIA GTX 1080Ti

Original: GPU for the ImageNet-ResNet50 benchmark, across all conv and fc layers.
Revised: GPU for the ImageNet-ResNet50 benchmark, across all conv and fc layers.

Original: In addition to the computational complexity, the memory footprint of SGD-based
Revised: In addition to the computational complexity, the memory footprint of SGD-based

Original: BP is also higher.
Revised: BP is also higher.

Original: This is because DNN software frameworks commonly store all activation values computed during FP to avoid recomputing a l − 1 , the input activations to the layers, used in Equation 6 of SGD-based BP .
Revised: This is because DNN software frameworks commonly store all activation values computed during FP to avoid recomputing a l − 1 , the input activations to the layers, used in Equation 6 of SGD-based BP .

Original: In contrast, the localized update for a layer can be performed as soon as the
Revised: In contrast, the localized update for a layer can be performed as soon as the

Original: FP through the layer is complete.
Revised: FP through the layer is complete.

Original: The activation tensor a l of layer L can be discarded or over-written as soon as FP proceeds to the next layer in the network, thereby freeing up a signiﬁcant portion of on-device memory during training.
Revised: The activation tensor a l of layer L can be discarded or over-written as soon as FP proceeds to the next layer in the network, thereby freeing up a signiﬁcant portion of on-device memory during training.

Original: In turn, this can allow larger minibatch sizes to be accommodated on a given hardware platform, when the localized updates are applied on a sufﬁcient number of layers.
Revised: In turn, this can allow larger minibatch sizes to be accommodated on a given hardware platform, when the localized updates are applied on a sufﬁcient number of layers.

Original: 2. L EARNING M ODE S ELECTION A LGORITHM
Revised: 2. L EARNING M ODE S ELECTION A LGORITHM

Original: The compute beneﬁts of localized learning come at the cost of potential loss in classiﬁcation accuracy with respect to SGD training.
Revised: The compute beneﬁts of localized learning come at the cost of potential loss in classiﬁcation accuracy with respect to SGD training.

Original: Thus, we utilize a learning mode selection algorithm to judiciously choose when and where to apply localized learning.
Revised: Thus, we utilize a learning mode selection algorithm to judiciously choose when and where to apply localized learning.

Original: The proposed algorithm identiﬁes the learning mode of each layer at every epoch to maximize the runtime beneﬁts, while incurring minimal losses in classiﬁcation accuracy.
Revised: The proposed algorithm identiﬁes the learning mode of each layer at every epoch to maximize the runtime beneﬁts, while incurring minimal losses in classiﬁcation accuracy.

Original: To design an efﬁcient learning mode selection algorithm, we ﬁrst study the effects of different spatiotemporal patterns of localized learning on the computational efﬁciency and classiﬁcation accuracy of a network.
Revised: To design an efﬁcient learning mode selection algorithm, we ﬁrst study the effects of different spatiotemporal patterns of localized learning on the computational efﬁciency and classiﬁcation accuracy of a network.

Original: We speciﬁcally investigate whether localized learning is more suitable for speciﬁc layers in the network and speciﬁc phases in the training process.
Revised: We speciﬁcally investigate whether localized learning is more suitable for speciﬁc layers in the network and speciﬁc phases in the training process.

Original: Impact on runtime efﬁciency : We ﬁrst analyze the spatial trends, i.e. , if locally updating speciﬁc layers in the network results in better runtime efﬁciency.
Revised: Impact on runtime efﬁciency : We ﬁrst analyze the spatial trends, i.e. , if locally updating speciﬁc layers in the network results in better runtime efﬁciency.

Original: In a particular epoch, if a convolutional layer L , updated with SGD precedes a convolutional layer K , that is updated locally, calculating the SGD-based error gradients of Layer L , i.e. δ L , requires error propagation through the locally updated layer K .
Revised: In a particular epoch, if a convolutional layer L , updated with SGD precedes a convolutional layer K , that is updated locally, calculating the SGD-based error gradients of Layer L , i.e. δ L , requires error propagation through the locally updated layer K .

Original: From a compute efﬁciency perspective, the beneﬁts of using localized-updates in layer K completely vanish.
Revised: From a compute efﬁciency perspective, the beneﬁts of using localized-updates in layer K completely vanish.

Original: Thus, it makes sense to partition the network into two regions - a preﬁx (set of initial layers) that are updated using localized learning, followed by layers that are updated with SGD.
Revised: Thus, it makes sense to partition the network into two regions - a preﬁx (set of initial layers) that are updated using localized learning, followed by layers that are updated with SGD.

Original: SGD-based BP is stopped at the junction of the two regions.
Revised: SGD-based BP is stopped at the junction of the two regions.

Original: Naturally, the compute beneﬁts increase when the number of locally updated layers are higher and thus the boundary i.e., the
Revised: Naturally, the compute beneﬁts increase when the number of locally updated layers are higher and thus the boundary i.e., the

Original: Localized → SGD transition layer is moved deeper into the network.
Revised: Localized → SGD transition layer is moved deeper into the network.

Original: The impact of different temporal patterns on runtime efﬁciency is quite straightforward, with higher number of locally updated epochs leading to higher beneﬁts.
Revised: The impact of different temporal patterns on runtime efﬁciency is quite straightforward, with higher number of locally updated epochs leading to higher beneﬁts.

Original: Further, as the compute complexity of localized updates is constant across different epochs, these beneﬁts are agnostic of which particular epoch involves localized learning.
Revised: Further, as the compute complexity of localized updates is constant across different epochs, these beneﬁts are agnostic of which particular epoch involves localized learning.

Original: Impact on accuracy : To analyze the impact on accuracy, we ﬁrst examine the nature of features learnt by different layers trained by SGD.
Revised: Impact on accuracy : To analyze the impact on accuracy, we ﬁrst examine the nature of features learnt by different layers trained by SGD.

Original: It is commonly accepted that the initial layers of a network (Agrawal et al., 2014) perform feature extraction, while later layers aid in the classiﬁcation process.
Revised: It is commonly accepted that the initial layers of a network (Agrawal et al., 2014) perform feature extraction, while later layers aid in the classiﬁcation process.

Original: As localized learning demonstrates better performance for feature extraction, applying it more aggressively, i.e for higher number of epochs, in the initial layers has a much smaller impact accuracy.
Revised: As localized learning demonstrates better performance for feature extraction, applying it more aggressively, i.e for higher number of epochs, in the initial layers has a much smaller impact accuracy.

Original: However, for later layers in the network, the number of localized learning epochs should be progressively reduced to preserve accuracy.
Revised: However, for later layers in the network, the number of localized learning epochs should be progressively reduced to preserve accuracy.

Original: [Figure]
Revised: [Figure]

Original: Overall, based on the impact of localized learning on both runtime and accuracy, we ﬁnd that a good learning mode selection algorithm should favor application of localized learning to a contiguous group of initial layers, while ensuring fewer or no localized learning epochs in later layers.
Revised: Overall, based on the impact of localized learning on both runtime and accuracy, we ﬁnd that a good learning mode selection algorithm should favor application of localized learning to a contiguous group of initial layers, while ensuring fewer or no localized learning epochs in later layers.

Original: We further impose an additional constraint on top of this spatio-temporal pattern.
Revised: We further impose an additional constraint on top of this spatio-temporal pattern.

Original: Speciﬁcally, we allow each layer to transition from one learning mode to another at most once during the entire training process.
Revised: Speciﬁcally, we allow each layer to transition from one learning mode to another at most once during the entire training process.

Original: We empirically observe that utilizing SGD as the initial learning mode allows the network to achieve a higher accuracy than utilizing localized learning as the initial mode.
Revised: We empirically observe that utilizing SGD as the initial learning mode allows the network to achieve a higher accuracy than utilizing localized learning as the initial mode.

Original: SGD essentially provides a better initialization point for all layers, and the subsequent use of localized updates enables the training to converge with good accuracy.
Revised: SGD essentially provides a better initialization point for all layers, and the subsequent use of localized updates enables the training to converge with good accuracy.

Original: Input: T E (Index of the transition layer at epoch
Revised: Input: T E (Index of the transition layer at epoch

Original: E), e k (epochs since last transition), || (cid:52) W E
Revised: E), e k (epochs since last transition), || (cid:52) W E

Original: ||
Revised: ||

Original: ( L 2 norm of the weight update of the transition layer at epoch E), K (minimum interval between transitions), t shift (number of layers to shift boundary)
Revised: ( L 2 norm of the weight update of the transition layer at epoch E), K (minimum interval between transitions), t shift (number of layers to shift boundary)

Original: Output: T E +1 (Index of the transition layer at epoch E+1) 1: W Avg = 1 K (cid:80) e
Revised: Output: T E +1 (Index of the transition layer at epoch E+1) 1: W Avg = 1 K (cid:80) e

Original: =
Revised: =

Original: E − 1 e =
Revised: E − 1 e =

Original: E − K || (cid:52) W e || 2: if || (cid:52) W E
Revised: E − K || (cid:52) W e || 2: if || (cid:52) W E

Original: || <
Revised: || <

Original: = α · W Avg and e k >
Revised: = α · W Avg and e k >

Original: =
Revised: =

Original: K 3:
Revised: K 3:

Original: T E +1 =
Revised: T E +1 =

Original: T E + t shift 4: e k =5: else 6:
Revised: T E + t shift 4: e k =5: else 6:

Original: T E +1 =
Revised: T E +1 =

Original: T E 7:
Revised: T E 7:

Original: e k = e k +
Revised: e k = e k +

Original: In accordance with the above considerations, we propose a learning mode selection algorithm, described in Algorithm 1, that identiﬁes the position of the boundary or the Localized → SGD transition layer every epoch.
Revised: In accordance with the above considerations, we propose a learning mode selection algorithm, described in Algorithm 1, that identiﬁes the position of the boundary or the Localized → SGD transition layer every epoch.

Original: To that end, the algorithm analyzes the L 2 norm of the SGD weight updates made to the Localized → SGD transition layer across epochs and determines whether the boundary can be shifted deeper into the network for the next epoch.
Revised: To that end, the algorithm analyzes the L 2 norm of the SGD weight updates made to the Localized → SGD transition layer across epochs and determines whether the boundary can be shifted deeper into the network for the next epoch.

Original: In order to ensure stability in the training process, the algorithm moves the boundary at most once in every K epochs.
Revised: In order to ensure stability in the training process, the algorithm moves the boundary at most once in every K epochs.

Original: It calculates the running average of the norm of the updates, W avg , over the last K epochs (line 1).
Revised: It calculates the running average of the norm of the updates, W avg , over the last K epochs (line 1).

Original: The boundary is shifted to the right only if the weight update in epoch E is within a fraction α of W avg , and K epochs have transpired since the last transition (line 2).
Revised: The boundary is shifted to the right only if the weight update in epoch E is within a fraction α of W avg , and K epochs have transpired since the last transition (line 2).

Original: The rationale for this criterion is that sustained high magnitudes of weight updates in the transition layer indicate that they are potentially critical to accuracy, in which case the transition layer must continue being updated with SGD.
Revised: The rationale for this criterion is that sustained high magnitudes of weight updates in the transition layer indicate that they are potentially critical to accuracy, in which case the transition layer must continue being updated with SGD.

Original: If the criterion is not satisﬁed, the boundary remains stationary (line 5).
Revised: If the criterion is not satisﬁed, the boundary remains stationary (line 5).

Original: [Figure]
Revised: [Figure]

Original: The value of α is set by analyzing the trends in the weight update magnitudes across the training process for different networks.
Revised: The value of α is set by analyzing the trends in the weight update magnitudes across the training process for different networks.

Original: The hyper-parameter t shift is set to the size of a recurring block, such as the residual blocks in ResNets and MobileNetV2.
Revised: The hyper-parameter t shift is set to the size of a recurring block, such as the residual blocks in ResNets and MobileNetV2.

Original: The hyper-parameter K is selected in a manner that ensures that localized updates are never applied beyond the initial 75% of the layers of the network.
Revised: The hyper-parameter K is selected in a manner that ensures that localized updates are never applied beyond some fraction of the initial network layers. We denote this fraction as L max , and is set to 0.75 in all our experiments.

Original: Equation 2 is used to compute K for a network of L layers and a total training period of E max epochs.
Revised: Equation 2 is used to compute K for a network of L layers and a total training period of E max epochs.

Original: [Equation]
Revised: [Equation]

Original: In Figure 3, we plot the progression of the transition layer across the ResNet-34 and -50 benchmarks trained on the ImageNet dataset using LoCal+SGD .
Revised: In Figure 3, we plot the progression of the transition layer across the ResNet-34 and -50 benchmarks trained on the ImageNet dataset using LoCal+SGD .

Original: Interestingly, the weight update norm metric automatically modulates the rate at which the boundary progresses, as the boundary traverses the deeper layers at a slower rate.
Revised: Interestingly, the weight update norm metric automatically modulates the rate at which the boundary progresses, as the boundary traverses the deeper layers at a slower rate.

Original: 2. W EAK S UPERVISION
Revised: 2. W EAK S UPERVISION

Original: To further bridge the accuracy gap between our approach and end-to-end SGD training, we introduce weak supervision in the locally updated layers.
Revised: To further bridge the accuracy gap between our approach and end-to-end SGD training, we introduce weak supervision in the locally updated layers.

Original: Unlike the SGD-updated layers, the locally updated layers in our approach cannot take advantage of the information provided by supervision, i.e., the classiﬁcation error evaluated at the output.
Revised: Unlike the SGD-updated layers, the locally updated layers in our approach cannot take advantage of the information provided by supervision, i.e., the classiﬁcation error evaluated at the output.

Original: We utilize this supervised information through a low-cost weak supervision scheme that consists of a single signal sent to all layers updated locally in a particular epoch, and is derived from the classiﬁcation loss observed over past few epochs.
Revised: We utilize this supervised information through a low-cost weak supervision scheme that consists of a single signal sent to all layers updated locally in a particular epoch, and is derived from the classiﬁcation loss observed over past few epochs.

Original: The weak supervision scheme is described in Algorithm 2.
Revised: The weak supervision scheme is described in Algorithm 2.

Original: The key principle behind the weak supervision scheme is to control the learning rates of the locally updated layers based on the rate at which the overall classiﬁcation loss changes.
Revised: The key principle behind the weak supervision scheme is to control the learning rates of the locally updated layers based on the rate at which the overall classiﬁcation loss changes.

Original: For example, if the overall classiﬁcation loss has increased across consecutive epochs, we reverse the direction of the updates (line 3) in the next epoch.
Revised: For example, if the overall classiﬁcation loss has increased across consecutive epochs, we reverse the direction of the updates (line 3) in the next epoch.

Original: In contrast, the update direction is maintained if the overall loss is decreasing (line 5).
Revised: In contrast, the update direction is maintained if the overall loss is decreasing (line 5).

Original: We ﬁnd that this weak supervision provides better accuracy results than other learning rate modulation techniques for the locally updated layers such as Adam or momentum-based updates.
Revised: We ﬁnd that this weak supervision provides better accuracy results than other learning rate modulation techniques for the locally updated layers such as Adam or momentum-based updates.

Original: Algorithm
Revised: Algorithm

Original: Input: L i (Overall classiﬁcation loss at epoch i ), lr L (original learning rate of layer L )
Revised: Input: L i (Overall classiﬁcation loss at epoch i ), lr L (original learning rate of layer L )

Original: Output: W L (Weight update of layer L ) 1: (cid:52) W L = conv ( a l − 1 , z l ) 2: if L i − 1 < L i 3:
Revised: Output: W L (Weight update of layer L ) 1: (cid:52) W L = conv ( a l − 1 , z l ) 2: if L i − 1 < L i 3:

Original: W L = W L - lr L · (cid:52) W L ||(cid:52)
Revised: W L = W L - lr L · (cid:52) W L ||(cid:52)

Original: W L || 4: else 5:
Revised: W L || 4: else 5:

Original: W L = W L + lr L · (cid:52) W L ||(cid:52)
Revised: W L = W L + lr L · (cid:52) W L ||(cid:52)

Original: W L ||
Revised: W L ||

Original: We would like to highlight that traditional SGD provides ﬁne-grained supervision and involves evaluating the error gradients for every neuron in the network.
Revised: We would like to highlight that traditional SGD provides ﬁne-grained supervision and involves evaluating the error gradients for every neuron in the network.

Original: In contrast, the proposed weak supervision scheme provides coarse-grained supervision by forcing all weights to re-use the same loss information.
Revised: In contrast, the proposed weak supervision scheme provides coarse-grained supervision by forcing all weights to re-use the same loss information.

Original: Overall, our weak supervision scheme is not developed with the intent to compete with SGD updates, but is rather a simple, approximate and low-cost technique that brings the ﬁnal accuracy of
Revised: Overall, our weak supervision scheme is not developed with the intent to compete with SGD updates, but is rather a simple, approximate and low-cost technique that brings the ﬁnal accuracy of

Original: LoCal+SGD at par with end-to-end SGD training performance.
Revised: LoCal+SGD at par with end-to-end SGD training performance.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In this section, we present the results of our experiments highlighting the compute beneﬁts achieved by LoCal+SGD .
Revised: In this section, we present the results of our experiments highlighting the compute beneﬁts achieved by LoCal+SGD .

Original: We evaluate the beneﬁts across a suite of 8 image-recognition DNNs acrossdatasets.
Revised: We evaluate the beneﬁts across a suite of 8 image-recognition DNNs acrossdatasets.

Original: We consider the ResNet18 (He et al., 2015) and VGG13 (Simonyan & Zisserman, 2015) networks for the Cifar10 (Krizhevsky et al., a) and Cifar

Revised: We consider the ResNet18 (He et al., 2015) and VGG13 (Simonyan & Zisserman, 2015) networks for the Cifar10 (Krizhevsky et al., a) and Cifar

Original: (Krizhevsky et al.
Revised: (Krizhevsky et al.

Original: , b) datasets; and the ResNet34, ResNet50 (He et al., 2015) and MobileNetV2 (Sandler et al., 2018) networks for the ImageNet dataset (Deng et al., 2009).
Revised: , b) datasets; and the ResNet34, ResNet50 (He et al., 2015) and MobileNetV2 (Sandler et al., 2018) networks for the ImageNet dataset (Deng et al., 2009).

Original: All experiments are conducted on Nvidia GTX 1080Ti
Revised: All experiments are conducted on Nvidia GTX 1080Ti

Original: GPUs with the batch size set to 64 per GPU, unless otherwise mentioned.
Revised: GPUs with the batch size set to 64 per GPU, unless otherwise mentioned.

Original: Further experimental methodology details for the baseline and proposed approach are provided in the Appendix.
Revised: Further experimental methodology details for the baseline and proposed approach are provided in the Appendix.

Original: 3. S INGLE GPU E XECUTION T IME B ENEFITS
Revised: 3. S INGLE GPU E XECUTION T IME B ENEFITS

Original: ImageNet : Table 1 presents the performance of the baseline (end-to-end SGD training) and the proposed LoCal+SGD algorithm on the ImageNet benchmarks in terms of the Top-1 classiﬁcation error and runtime observed on a single GPU.
Revised: ImageNet : Table 1 presents the performance of the baseline (end-to-end SGD training) and the proposed LoCal+SGD algorithm on the ImageNet benchmarks in terms of the Top-1 classiﬁcation error and runtime observed on a single GPU.

Original: For all benchmarks listed here, LoCal+SGD applies localized updates for nearly 50-60% of the layers.
Revised: For all benchmarks listed here, LoCal+SGD applies localized updates for nearly 50-60% of the layers.

Original: As can be seen, LoCal+SGD achieves upto ∼ 1.4 × reduction in runtime compared to to the baseline, while sacriﬁcing < 0.5% loss in Top-1 accuracy.
Revised: As can be seen, LoCal+SGD achieves upto ∼ 1.4 × reduction in runtime compared to to the baseline, while sacriﬁcing < 0.5% loss in Top-1 accuracy.

Original: Table 1 also compares the performance of LoCal+SGD against existing research efforts designed to improve training efﬁciency.
Revised: Table 1 also compares the performance of LoCal+SGD against existing research efforts designed to improve training efﬁciency.

Original: We perform this analysis against two efforts, namely (i) Training with stochastic depth (Huang et al., 2016) and (ii) Structured Pruning during Training (Lym et al., 2019).
Revised: We perform this analysis against two efforts, namely (i) Training with stochastic depth (Huang et al., 2016) and (ii) Structured Pruning during Training (Lym et al., 2019).

Original: Training with stochastic depth, as the name suggests, stochastically bypasses residual blocks by propagating input activations/error gradients via identity or downsampling transformations, resulting in improved training time.
Revised: Training with stochastic depth, as the name suggests, stochastically bypasses residual blocks by propagating input activations/error gradients via identity or downsampling transformations, resulting in improved training time.

Original: However, the approach is targeted towards extremely deep networks and
Revised: However, the approach is targeted towards extremely deep networks and

Original: [Table]
Revised: [Table]

Original: as seen in Table 1, it incurs a noticeable accuracy loss on networks such as ResNet34, ResNet and MobileNetV2.
Revised: as seen in Table 1, it incurs a noticeable accuracy loss on networks such as ResNet34, ResNet and MobileNetV2.

Original: Compared to training with stochastic depth, our proposal clearly achieves better accuracy as well as training runtime beneﬁts.
Revised: Compared to training with stochastic depth, our proposal clearly achieves better accuracy as well as training runtime beneﬁts.

Original: The key principle behind the pruning during training approach is to reduce the size of the weight and activation tensors in a structured manner during training, thereby providing speed-ups on GPU/TPU platforms.
Revised: The key principle behind the pruning during training approach is to reduce the size of the weight and activation tensors in a structured manner during training, thereby providing speed-ups on GPU/TPU platforms.

Original: However, on complex benchmarks such as ResNet50, such techniques achieve speed-ups at the cost of signiﬁcant drop in accuracy ( ∼ 1 . 5% ).
Revised: However, on complex benchmarks such as ResNet50, such techniques achieve speed-ups at the cost of signiﬁcant drop in accuracy ( ∼ 1 . 5% ).

Original: To further demonstrate the utility of localized updates in our approach, we consider a third technique, wherein layers selected to be updated locally for a given epoch are instead frozen, i.e., the parameters are held ﬁxed during that epoch.
Revised: To further demonstrate the utility of localized updates in our approach, we consider a third technique, wherein layers selected to be updated locally for a given epoch are instead frozen, i.e., the parameters are held ﬁxed during that epoch.

Original: While this achieves better runtime savings, it incurs considerably higher loss ( ∼ 1%) in accuracy, further underscoring the beneﬁts of LoCal+SGD .
Revised: While this achieves better runtime savings, it incurs considerably higher loss ( ∼ 1%) in accuracy, further underscoring the beneﬁts of LoCal+SGD .

Original: CIFAR-10 and CIFAR-100 : Table 2 presents the accuracy and corresponding compute beneﬁts of the baseline and the proposed technique, as well as training with stochastic depth and layer freezing, for the CIFAR-10 and CIFAR-100 datasets.
Revised: CIFAR-10 and CIFAR-100 : Table 2 presents the accuracy and corresponding compute beneﬁts of the baseline and the proposed technique, as well as training with stochastic depth and layer freezing, for the CIFAR-10 and CIFAR-100 datasets.

Original: Stochastic depth is applicable only to residual blocks and is hence not considered for the VGG-13 network.
Revised: Stochastic depth is applicable only to residual blocks and is hence not considered for the VGG-13 network.

Original: Across benchmarks, we observe upto a 1.51 × improvement in training runtime.
Revised: Across benchmarks, we observe upto a 1.51 × improvement in training runtime.

Original: Compared to the ImageNet benchmarks, LoCal+SGD applies localized updates more aggressively in the CIFAR-10 and CIFAR-100 benchmarks i.e., for more layers are updated locally for a higher number of epochs.
Revised: Compared to the ImageNet benchmarks, LoCal+SGD applies localized updates more aggressively in the CIFAR-10 and CIFAR-100 benchmarks i.e., for more layers are updated locally for a higher number of epochs.

Original: This leads to the superior compute beneﬁts of the proposed scheme on these benchmarks.
Revised: This leads to the superior compute beneﬁts of the proposed scheme on these benchmarks.

Original: [Table]
Revised: [Table]

Original: 3. E XECUTION T IME B ENEFITS FOR M ULTI -GPU T RAINING
Revised: 3. E XECUTION T IME B ENEFITS FOR M ULTI -GPU T RAINING

Original: [Figure]
Revised: [Figure]

Original: We analyze the memory footprint of the ResNet50 network when trained with LoCal+SGD on the ImageNet dataset.
Revised: We analyze the memory footprint of the ResNet50 network when trained with LoCal+SGD on the ImageNet dataset.

Original: Training ﬁrst commences with all layers updated with SGD, resulting in a high memory footprint.
Revised: Training ﬁrst commences with all layers updated with SGD, resulting in a high memory footprint.

Original: Due to the 10 GB capacity of the chosen GPU, the mini-batch size is set to 64 per GPU.
Revised: Due to the 10 GB capacity of the chosen GPU, the mini-batch size is set to 64 per GPU.

Original: As the Localized → SGD transition layer progresses across the network, the memory footprint required also gradually reduces across epochs.
Revised: As the Localized → SGD transition layer progresses across the network, the memory footprint required also gradually reduces across epochs.

Original: We take advantage of this reduction in memory footprint in the context of distributed training using 4 GPUs with data parallelism.
Revised: We take advantage of this reduction in memory footprint in the context of distributed training using 4 GPUs with data parallelism.

Original: Speciﬁcally, we extract additional runtime beneﬁts by increasing the batch size on each GPU, which reduces the frequency of gradient aggregation between devices and alleviates the communication overhead.
Revised: Speciﬁcally, we extract additional runtime beneﬁts by increasing the batch size on each GPU, which reduces the frequency of gradient aggregation between devices and alleviates the communication overhead.

Original: At epoch 33, the memory footprint per GPU reduces to less than 5 GB, allowing training with an increased mini-batch size of 128 per GPU from epoch 33 onwards.
Revised: At epoch 33, the memory footprint per GPU reduces to less than 5 GB, allowing training with an increased mini-batch size of 128 per GPU from epoch 33 onwards.

Original: The doubling of the batch-size provides an additional 6% runtime improvement, when measured across the entire training period.
Revised: The doubling of the batch-size provides an additional 6% runtime improvement, when measured across the entire training period.

Original: We note that other training techniques such as training with stochastic depth cannot exploit this feature, due to minimal reduction in memory footprint.
Revised: We note that other training techniques such as training with stochastic depth cannot exploit this feature, due to minimal reduction in memory footprint.

Original: 3. D ESIGN S PACE E XPLORATION
Revised: 3. A BLATION A NALYSIS

Original: As mentioned in Section 2, the hyper-parameter α limits the rate at which the boundaryprogresses across the network.
Revised: As mentioned in Section 2, the hyper-parameters α , t shift and L max control the progression of the boundary across the network.

Original: Different values of α can result in different learning mode conﬁgurations
Revised: Different values of either parameter can result in dif-

Original: [Table]
Revised: [Table]

Original: Figure during training, resulting in different points in the computational efﬁciency vs. accuracy trade-off space.
Revised: ferent learning mode conﬁgurations during training, resulting in different points in the computational efﬁciency vs. accuracy trade-off space.

Revised: To understand the trade-off space between accuracy and runtime beneﬁts, we now individually study the impact of each parameter.

Original: Figure 5 depicts the best compute beneﬁts achieved for different α , for accuracy losses ranging from 0.1%-1.5% for the ResNet50 and MobileNetV benchmarks on ImageNet.
Revised: [Table] Impact of α : Figuredepicts the best compute beneﬁts achieved for different α , for accuracy losses ranging from 0.1%-1.5% for the ResNet50 and MobileNetV2 benchmarks on ImageNet.

Original: On the
Revised: On the

Original: ResNet50 benchmark, even while limiting the loss in accuracy 0.1%, LoCal+SGD achieves 1.1 × speedup over traditional SGD.
Revised: ResNet benchmark, even while limiting the loss in accuracy 0.1%, LoCal+SGD achieves 1.1 × speedup over traditional SGD.

Original: The speedups increase to 1.38 × -1.47 × when around 1.5% loss in accuracy is tolerable. 
Revised: The speedups increase to 1.38 × -1.47 × when around 1.5% loss in accuracy is tolerable. [Figure] [Table] [Figure]

Revised: Here, we denote t shift as a percentage of the total network depth.

Original: R ELATED W ORK
Revised: For each value of t shift , we identify α and L max that result in the best accuracy while providing considerable runtime savings (at least greater than 10%). The graph indicates that accuracy is severely impacted for extremely small values of t shift that are less than 3%. The accuracy is largely stable in the regime of t shift between 5-12%, and begins to experience small degradations again when t shift exceeds 12%.

Revised: These trends can be explained by analyzing the rate at which the transition layer progresses, and the number of layers transitioning to localized updates in an epoch for different t shift values.

Revised: Smaller values of t shift (<3%) give rise to low values of k ( ∼ 1-2 epochs), the minimum number of epochs that must elapse before the transition layer can shift again.

Revised: This results in fast progression of the transition layer across the network, leading to rapid changes in the learning mode at the boundary, thereby negatively impacting accuracy.

Revised: In contrast, while larger t shift values (>12%) encourage slow progression of the boundary, a larger number of layers transition from SGD to localized updates in a single epoch, thereby impacting performance.

Revised: We note here that in both cases, while α and L max can be tuned to control the progression and mitigate the loss in accuracy, the runtime savings is vastly reduced (<10%).

Revised: Furthermore, for ﬁxed values of L max and α , t shift is largely insensitive to runtime beneﬁts, as the average number of layers updated with localized updates remains similar.

Original: This section discusses related research efforts that achieve compute efﬁcient DNN training.
Revised: Hence, for best accuracy and runtime beneﬁts we set t shift in the range of 5-10% for all networks. Impact of L max : Figure 6(b) depicts the impact of L max on accuracy for the ResNet50 network. For each L max , we identify the α and t shift that provide the best compute beneﬁts for less than 0.5% loss in accuracy. As with t shift , we denote L max as a percentage of the total network depth.

Original: All the techniques listed below utilize gradient-descent algorithms to train the DNN model.
Revised: As seen in the ﬁgure, the degradation in accuracy increases slowly for L max in the initial layers - it is merely 0.1% at around L max = 30%, and increases to 0.4-0.5% for L max = 60-70%. However, the accuracy degradation sharply increases beyond 2% once L max exceeds 90% of the network depth. Further, runtime beneﬁts increase linearly with higher values of L max , for ﬁxed t shift and α . Hence, for achieving a good accuracy versus runtime trade-off, we usually set L max to 75% for all networks.

Original: These techniques are largely complementary to LoCal+SGD , as they canpotentially be applied to the parts of the
Revised: This section discusses related research efforts to the proposed LoCal + SGD training technique. These efforts can be broadly categorized into two classes.

Original:  DNN model updated with SGD.
Revised: The ﬁrst class of efforts focus on compute efﬁcient DNN training. All efforts belonging to this class utilize gradient-descent algorithms to train the DNN model. These techniques are largely complementary to LoCal+SGD , as they can potentially be applied to the parts of the DNN model updated with SGD.

Original: In Section 3, we demonstrated how LoCal+SGD achieves superior accuracy versus computational efﬁciency trade-off than some of these efforts.
Revised: In Section 3, we demonstrated how LoCal+SGD achieves superior accuracy versus computational efﬁciency trade-off than some of these efforts.

Revised: Further, the second class of efforts involve neuro-scientiﬁc faithful learning rules, such as feedback alignment based efforts etc (Nøkland, 2016).

Revised: Our work is orthogonal to such efforts, as we selectively combine localized learning rules with SGD for better computational efﬁciency.

Revised: We elucidate upon the different research efforts in both directions as follows.

Original: Hyper-parameter tuning :
Revised: Hyper-parameter tuning :

Original: Many notable algorithmic efforts are directed towards achieving training efﬁciency by controlling the hyper-parameters involved in gradient-descent, notably the learning rate.
Revised: Many notable algorithmic efforts are directed towards achieving training efﬁciency by controlling the hyper-parameters involved in gradient-descent, notably the learning rate.

Original: (You et al., 2017a; Akiba et al., 2017; Goyal et al., 2017; You et al., 2017b) propose learning rate tuning algorithms that achieve training in less than an hour with no loss in accuracy, when distributed to over hundreds of CPU/GPU cores.
Revised: (You et al., 2017a; Akiba et al., 2017; Goyal et al., 2017; You et al., 2017b) propose learning rate tuning algorithms that achieve training in less than an hour with no loss in accuracy, when distributed to over hundreds of CPU/GPU cores.

Original: Model size reduction during training : Model size reduction via pruning and quantization is a popular technique to reduce compute costs during inference.
Revised: Model size reduction during training : Model size reduction via pruning and quantization is a popular technique to reduce compute costs during inference.

Original: In many of these efforts, a dense or full precision model is re-trained or ﬁne-tuned to obtain a pruned or quantized model.
Revised: In many of these efforts, a dense or full precision model is re-trained or ﬁne-tuned to obtain a pruned or quantized model.

Original: Recently, several efforts have also investigated dynamically pruning (Lym et al., 2019) or quantizing (Sun et al., 2019) a model during training itself.
Revised: Recently, several efforts have also investigated dynamically pruning (Lym et al., 2019) or quantizing (Sun et al., 2019) a model during training itself.

Original: The reduction in model size results in training speed-ups.
Revised: The reduction in model size results in training speed-ups.

Original: Taking a slightly different approach (Huang et al., 2016) proposes stochastically dropping residual blocks on extremely deep networks such as ResNet-1202, not only for training runtime beneﬁts but also better accuracies due to improved gradient strength.
Revised: Taking a slightly different approach (Huang et al., 2016) proposes stochastically dropping residual blocks on extremely deep networks such as ResNet-1202, not only for training runtime beneﬁts but also better accuracies due to improved gradient strength.

Original: Instance importance based training :
Revised: Instance importance based training :

Original: Recent research efforts have discovered that not all training samples are required for improving loss minimization during SGD training (Jiang et al., 2019; Zhang et al., 2019).
Revised: Recent research efforts have discovered that not all training samples are required for improving loss minimization during SGD training (Jiang et al., 2019; Zhang et al., 2019).

Original: That is, a sizable fraction of the samples can be skipped during several epochs, depending on their impact on the classiﬁcation loss evaluated during FP .
Revised: That is, a sizable fraction of the samples can be skipped during several epochs, depending on their impact on the classiﬁcation loss evaluated during FP .

Original: This translates to a reduction in mini-batches, providing considerable runtime beneﬁts.
Revised: This translates to a reduction in mini-batches, providing considerable runtime beneﬁts.

Revised: Neuro-scientiﬁc learning rules : Back-propagation algorithms utilized in DNN training are not biologically plausible, and do not explain how learning actually happens in the brain.

Revised: To this end, there have been several efforts that develop biological faithful learning algorithms, and demonstrate considerable success on complex benchmarks including Cifar10 and ImageNet.

Revised: For example, unlike conventional DNN training, feedback alignmnent algorithms (Nøkland, 2016) tackle the weight transport problem (Liao et al., 2015) by allowing for asymmetry in the weight values during forward and back propagation.

Revised: Likewise, Target-Propagation (Lee et al., 2015) encourages neural activity to reach desired target activations evaluated during forward propagation itself, instead of utilizing loss gradients.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduce a new approach to improve the training efﬁciency of state-of-the-art
Revised: In this paper, we introduce a new approach to improve the training efﬁciency of state-of-the-art

Original: DNNs.
Revised: DNNs.

Original: Speciﬁcally, we take advantage of the computationally efﬁcient nature of localized learning rules and selectively update some layers with these rules instead of SGD.
Revised: Speciﬁcally, we take advantage of the computationally efﬁcient nature of localized learning rules and selectively update some layers with these rules instead of SGD.

Original: We design an intelligent learning mode selection algorithm that determines the update method for the convolutional layers of the network in every epoch while maintaining the accuracy level and extracting maximum beneﬁts.
Revised: We design an intelligent learning mode selection algorithm that determines the update method for the convolutional layers of the network in every epoch while maintaining the accuracy level and extracting maximum beneﬁts.

Original: Further, we also implement a low-cost weak supervision scheme that brings the accuracy of the proposed scheme closer to traditional SGD training.
Revised: Further, we also implement a low-cost weak supervision scheme that brings the accuracy of the proposed scheme closer to traditional SGD training.

Original: Across a benchmark suite of 8 DNNs, we achieve upto 1.5 × reduction in training times, as measured on a modern GPU platform.
Revised: Across a benchmark suite of 8 DNNs, we achieve upto 1.5 × reduction in training times, as measured on a modern GPU platform.

Original:  A PPENDIX 6. E XPERIMENTAL S ETUP
Revised:  A PPENDIX 6. E XPERIMENTAL S ETUP

Original: This subsection describes the experimental setup used for realizing the baseline and proposed
Revised: This subsection describes the experimental setup used for realizing the baseline and proposed

Original: LoCal+SGD training schemes, on the benchmarks speciﬁed in Section 3 of the main paper.
Revised: LoCal+SGD training schemes, on the benchmarks speciﬁed in Section 3 of the main paper.

Original: We conduct our experiments on the complete training and test datasets of each benchmark, using the
Revised: We conduct our experiments on the complete training and test datasets of each benchmark, using the

Original: PyTorch (Paszke et al., 2019) framework.
Revised: PyTorch (Paszke et al., 2019) framework.

Original: Baseline : We consider end-to-end SGD training as the baseline in our experiments.
Revised: Baseline : We consider end-to-end SGD training as the baseline in our experiments.

Original: The hyperparameters used in SGD training of each of the benchmarks are described below.
Revised: The hyperparameters used in SGD training of each of the benchmarks are described below.

Original: ImageNet: For experiments in Section 3.1 we utilize a batch-size of 64 per GPU, for all benchmarks.
Revised: ImageNet: For experiments in Section 3.1 we utilize a batch-size of 64 per GPU, for all benchmarks.

Original: For the ResNet50 and ResNet34 benchmarks the initial learning rate set to 0.025.
Revised: For the ResNet50 and ResNet34 benchmarks the initial learning rate set to 0.025.

Original: The learning rate is decreased by 0.1 every 30 epochs, for a total training duration of 90 epochs, and the weight decay is 4 e − 5 .
Revised: The learning rate is decreased by 0.1 every 30 epochs, for a total training duration of 90 epochs, and the weight decay is 4 e − 5 .

Original: The MobileNetV2 benchmark utilizes an initial learning rate of 0.0125.
Revised: The MobileNetV2 benchmark utilizes an initial learning rate of 0.0125.

Original: We use a cosine learning rate decay schedule, as in (Li et al., 2019) for 150 epochs.
Revised: We use a cosine learning rate decay schedule, as in (Li et al., 2019) for 150 epochs.

Original: The weight decay is set to 4 e − 5 .
Revised: The weight decay is set to 4 e − 5 .

Original: Both benchmarks use an input size of 224*224*3.
Revised: Both benchmarks use an input size of 224*224*3.

Original: For the experiments in Section 3.2, the total batch-size at epoch 1 is 256 (64*4), with the initial learning rate set to 0.1 for the ResNet benchmarks and 0.05 for the MobileNetV2 benchmark.
Revised: For the experiments in Section 3.2, the total batch-size at epoch 1 is 256 (64*4), with the initial learning rate set to 0.1 for the ResNet benchmarks and 0.05 for the MobileNetV2 benchmark.

Original: All other parameters remain the same.
Revised: All other parameters remain the same.

Original: Cifar10 and Cifar100 : All Cifar10 and Cifar100 experiments utilize a batch-size of 64.
Revised: Cifar10 and Cifar100: All Cifar10 and Cifar100 experiments utilize a batch-size of 64.

Original: The Cifar benchmarks are trained with an initial learning rate of 0 .
Revised: The Cifar benchmarks are trained with an initial learning rate of 0 .

Original: that is decayed by 0.1 every 10 epochs, across 90 epochs.
Revised: that is decayed by 0.1 every 10 epochs, across 90 epochs.

Original: The initial learning rate of the Cifar100 benchmarks is 0 .
Revised: The initial learning rate of the Cifar100 benchmarks is 0 .

Original: 025 and decayed by 0.
Revised: 025 and decayed by 0.

Original: every 20 epochs, for 150 epochs in total.
Revised: every 20 epochs, for 150 epochs in total.

Original: The weight decay is set to 5 e − 4 .
Revised: The weight decay is set to 5 e − 4 .

Original: Both benchmarks utilize an input size of 32*32*3.
Revised: Both benchmarks utilize an input size of 32*32*3.

Original: LoCal+SGD :
Revised: LoCal+SGD :

Original: In the proposed LoCal+SGD training scheme, the layers updated with SGD are trained with the same hyper-parameters used in the baseline implementation. 
Revised: In the proposed LoCal+SGD training scheme, the layers updated with SGD are trained with the same hyper-parameters used in the baseline implementation. Further, LoCal+SGD training is conducted using the same number of epochs as baseline SGD training.

Original: When a layer is updated locally, the initial learning rate is 0 .
Revised: When a layer is updated locally, the initial learning rate is 0 .

Original: and is decayed by a factor of 2 and 10 every 50 epochs, for the Cifar and the ImageNet benchmarks respectively.
Revised: and is decayed by a factor of 2 and 10 every 30 epochs, for the Cifar and the ImageNet benchmarks respectively.

Original: In all experiments, the α parameter is set to 0.95.
Revised: In all experiments, the α parameter is set to 0.95.

Original: We measure the accuracy and runtime of the proposed scheme for the same number of training epochs as the baseline implementations. 
Revised: We measure the accuracy and runtime of the proposed scheme for the same number of training epochs as the baseline implementations. 6. H YPER -P ARAMETER T UNING

Revised: To realize LoCal+SGD , we introduce three hyper-parameters: , t shift and L max .

Revised: t shift controls the number of layers that switch to SGD-based updates every epoch, L max is the maximum number of layers that can be updated with localized learning rules, and α determines the position of the transition layer every epoch by analyzing the gradient information at the boundary between the localized and

Revised: SGD updates.

Revised: To obtain optimized values for these hyper-parameters, we ﬁrst perform simple grid search using a single network for a particular dataset (for example, we choose the ResNet50 network for ImageNet).

Revised: We transfer the same hyper-parameter values to other networks for the same dataset.

Revised: We justify our use of common hyper-parameter values by the following experiment.

Revised: In Table 4 below, we depict the results on other ImageNet benchmarks (ResNet34 and MobileNetV2) when hyper-parameter tuning is performed for each benchmark individually.

Revised: As can be seen, the accuracy and runtime beneﬁts are only marginally better than those obtained using a common set of hyper-parameters obtained by tuning on the ResNet50 benchmark.

Revised: We thus utilize common values for a dataset, effectively rendering them constants.

Revised: The time taken to obtain these constants is thus a one-time cost

Revised: [Table]

Revised: and does not impact the speedups obtained by LoCal+SGD .

Revised: [Table]

Revised: [Table]

Revised: I MPACT OF W EAK S UPERVISION

Revised: In Table

Revised: [Table]

Revised: we highlight the impact of the weak supervision technique on ﬁnal classiﬁcation accuracy.

Revised: As can be seen, across all our benchmarks, the weak supervision technique clearly improves accuracy by nearly 0.06%-0.17%, bringing the ﬁnal accuracy of LoCal + SGD closer to baseline SGD.

Revised: [Table]

Revised: 6. A DDITIONAL C OMPARATIVE A NALYSIS

Revised: In addition to the experiments performed in Section 3 to compare the performance of LoCal+SGD against existing techniques such as pruning during training (Lym et al., 2019) and training with stochastic depth (Huang et al., 2016), we conduct additional experiments to further solidify the superiority of our approach.

Revised: We elucidate upon these comparisons as follows.

Revised: 6.4. C OMPARING LoCal + SGD AGAINST SGD AT ISO - ACCURACY

Revised: We compare the proposed LoCal+SGD training strategy against a SGD baseline that is trained with fewer epochs, i.e., the number of epochs required to reach the highest accuracy obtained by

Revised: LoCal + SGD in a 90-epoch training period.

Revised: For the ImageNet benchmarks, the runtime improvements are listed in Table 6 below.

Revised: Clearly, LoCal+SGD continues to achieve signiﬁcant speed-ups (around 1.25 × ) compared to the SGD baseline, even for complex benchmarks such as ResNet50 and

Revised: MobileNetV2.

Revised: 6.4. C OMPARING LoCal+SGD AGAINST FREEZING LAYERS DURING TRAINING

Revised: In Section 3, we compare LoCal+SGD against a technique, freezing layers during training, wherein instead of updating the layers using localized learning, the weights are held ﬁxed.

Revised: In this section,

Revised: [Table]

Revised: we perform a more thorough comparison of LoCal+SGD against freezing layers during training.

Revised: Speciﬁcally

Revised: [Table]

Revised: we perform this comparison at iso-runtime

Revised: [Table]

Revised: and analyze the resulting accuracy of either approach.

Revised: To elaborate, we ﬁrst identify the best LoCal+SGD conﬁguration that can reach within 0.05%, 0.1%, 0.25%, 0.5% and 1% of the baseline SGD accuracy.

Revised: Then, for the same runtimes taken by each LoCal+SGD conﬁguration

Revised: [Table]

Revised: we identify the conﬁguration that provides the best accuracy for the freezing layers approach.

Revised: Our results for the Cifar10 ResNet18 benchmark can be found in Table 7.

Revised: LoCal+SGD performs superior to freezing layers during training on 3 out of theconﬁgurations studied,

Revised: i

Revised: [Table]

Revised: e., is a superior technique when the loss compared to SGD is allowed to exceed 0.1%

Revised: [Table]

Revised: [Table]

Revised: A NALYSIS OF STATIC SCHEDULES FOR LEARNING MODE SELECTION

Revised: The current LoCal+SGD framework is realized with the help of an automatic learning mode selection algorithm, which determines the position of the transition layer every epoch.

Revised: Instead of a dynamic data-dependent algorithm, we investigate the beneﬁts of using a static schedule - that is, the position of the transition layer is determined using some pre-deﬁned scheduling function.

Revised: To this end, we have implemented a simple static schedule that favors aggressive application of the localized learning rule in initial layers, and gradually decreases the number of epochs localized learning is applied in the deeper layers.

Revised: As shown in Equation 3, we opt for a quadratic scheduling function, as we empirically observe they perform better compared to the linear functions studied.

Revised: Here N determines the position of the transition layer every epoch, E max is the maximum number of training epochs, and c 1 and care constants obtained using grid search.

Revised: [Equation]

Revised: We report the results using this static schedule in Table 8 for the ImageNet-ResNet50 and MobileNetV2 benchmarks.

Revised: Compared to the results reported in Table 1, we ﬁnd that the static schedule achieves slightly higher runtime beneﬁts, for marginally lower accuracies.

Revised: However, static schedules suffer from some drawbacks – several static scheduling functions are feasible, e.g. exponential, quadratic, etc., and identifying the best scheduling function for each network requires extensive empirical analysis.

Revised: The learning mode selection algorithm utilized in the paper helps alleviate this by automatically identifying the position of the transition layer every epoch, leveraging the gradient information at the boundary between localized updates and SGD.

Revised: [Table]