Original: P IPELINED T RAINING WITH S TALE W EIGHTS OF D EEP C ONVOLUTIONAL N EURAL N ETWORKS
Revised: P IPELINED T RAINING WITH S TALE W EIGHTS OF D EEP C ONVOLUTIONAL N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The growth in the complexity of Convolutional Neural Networks (CNNs) is increasing interest in partitioning a network across multiple accelerators during training and pipelining the backpropagation computations over the accelerators.
Revised: The growth in the complexity of Convolutional Neural Networks (CNNs) is increasing interest in partitioning a network across multiple accelerators during training and pipelining the backpropagation computations over the accelerators.

Original: Existing approaches avoid or limit the use of stale weights through techniques such as micro-batching or weight stashing.
Revised: Existing approaches avoid or limit the use of stale weights through techniques such as micro-batching or weight stashing.

Original: These techniques either underutilize of accelerators or increase memory footprint.
Revised: These techniques either underutilize of accelerators or increase memory footprint.

Original: We explore the impact of stale weights on the statistical efﬁciency and performance in a pipelined backpropagation scheme that maximizes accelerator utilization and keeps memory overhead modest.
Revised: We explore the impact of stale weights on the statistical efﬁciency and performance in a pipelined backpropagation scheme that maximizes accelerator utilization and keeps memory overhead modest.

Original: We use 4 CNNs (LeNet-5, AlexNet, VGG and ResNet) and show that when pipelining is limited to early layers in a network, training with stale weights converges and results in models with comparable inference accuracies to those resulting from non-pipelined training on MNIST and CIFAR-10 datasets; a drop in accuracy of 0.4%, 4%, 0.83% and 1.45% for the 4 networks, respectively.
Revised: We use 4 CNNs (LeNet-5, AlexNet, VGG and ResNet) and show that when pipelining is limited to early layers in a network, training with stale weights converges and results in models with comparable inference accuracies to those resulting from non-pipelined training on MNIST and CIFAR-10 datasets; a drop in accuracy of 0.4%, 4%, 0.83% and 1.45% for the 4 networks, respectively.

Original: However, when pipelining is deeper in the network, inference accuracies drop significantly.
Revised: However, when pipelining is deeper in the network, inference accuracies drop significantly.

Original: We propose combining pipelined and non-pipelined training in a hybrid scheme to address this drop.
Revised: We propose combining pipelined and non-pipelined training in a hybrid scheme to address this drop.

Original: We demonstrate the implementation and performance of our pipelined backpropagation in PyTorch on 2 GPUs using ResNet, achieving speedups of up to 1.8X over a 1-GPU baseline, with a small drop in inference accuracy.
Revised: We demonstrate the implementation and performance of our pipelined backpropagation in PyTorch on 2 GPUs using ResNet, achieving speedups of up to 1.8X over a 1-GPU baseline, with a small drop in inference accuracy.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Modern Convolutional Neural Networks (CNNs) have grown in size and complexity to demand considerable memory and computational resources, particularly for training.
Revised: Modern Convolutional Neural Networks (CNNs) have grown in size and complexity to demand considerable memory and computational resources, particularly for training.

Original: This growth makes it sometimes difﬁcult to train an entire network with a single accelerator (Huang et al., 2018;
Revised: This growth makes it sometimes difﬁcult to train an entire network with a single accelerator (Huang et al., 2018;

Original: Harlap et al., 2018; Chen et al., 2012).
Revised: Harlap et al., 2018; Chen et al., 2012).

Original: Instead, the network is partitioned among multiple accelerators, typically by partitioning its layers among the available accelerators, as shown in Figure 1 for an example 8-layer network.
Revised: Instead, the network is partitioned among multiple accelerators, typically by partitioning its layers among the available accelerators, as shown in Figure 1 for an example 8-layer network.

Original: The 8 layers are divided into 4 computationally-balanced partitions, P 0 ...
Revised: The 8 layers are divided into 4 computationally-balanced partitions, P 0 ...

Original: Pand each partition is mapped to one of the 4 accelerators, A 0 ...
Revised: Pand each partition is mapped to one of the 4 accelerators, A 0 ...

Original: A 3 .
Revised: A 3 .

Original: Each accelerator is responsible for the computations associated with the layers mapped to it.
Revised: Each accelerator is responsible for the computations associated with the layers mapped to it.

Original: However, the nature of the backpropagation algorithm used to train CNNs (Rumelhart et al., 1986) is that the computations of a layer are performed only after the computations of the preceding layer in the forward pass of the algorithm and only after the computations of the succeeding layer in the backward pass.
Revised: However, the nature of the backpropagation algorithm used to train CNNs (Rumelhart et al., 1986) is that the computations of a layer are performed only after the computations of the preceding layer in the forward pass of the algorithm and only after the computations of the succeeding layer in the backward pass.

Original: Further, the computations for one batch of input data are only performed after the computations of the preceding batch have updated the parameters (i.e., weights) of the network.
Revised: Further, the computations for one batch of input data are only performed after the computations of the preceding batch have updated the parameters (i.e., weights) of the network.

Original: These dependences underutilize the accelerators, as shown by the space-time diagram in Figure 2; only one accelerator can be active at any given point in time.
Revised: These dependences underutilize the accelerators, as shown by the space-time diagram in Figure 2; only one accelerator can be active at any given point in time.

Original: The underutilization of accelerators can be alleviated by pipelining the computations of the backpropagation algorithm over the accelerators (Huang et al., 2018;
Revised: The underutilization of accelerators can be alleviated by pipelining the computations of the backpropagation algorithm over the accelerators (Huang et al., 2018;

Original: Harlap et al., 2018; Chen et al., 2012).
Revised: Harlap et al., 2018; Chen et al., 2012).

Original: That is, by overlapping the computations of different input data batches using the multiple accelerators.
Revised: That is, by overlapping the computations of different input data batches using the multiple accelerators.

Original: However, pipelining causes an accelerator to potentially use weights that are yet to be updated by an accelerator further down in the pipeline.
Revised: However, pipelining causes an accelerator to potentially use weights that are yet to be updated by an accelerator further down in the pipeline.

Original: The use of such stale weights can negatively affect the statistical efﬁciency of the network, preventing the convergence of training or producing a model with lower inference accuracy.
Revised: The use of such stale weights can negatively affect the statistical efﬁciency of the network, preventing the convergence of training or producing a model with lower inference accuracy.

Original: L
Revised: L

Original: L L
Revised: L L

Original: L L
Revised: L L

Original: L L
Revised: L L

Original: A A
Revised: A A

Original: A A
Revised: A A

Original: P P
Revised: P P

Original: P P
Revised: P P

Original: A
Revised: A

Original: A 1 A
Revised: A 1 A

Original: A
Revised: A

Original: P 0 P
Revised: P 0 P

Original: P 2 P
Revised: P 2 P

Original: Idle Time
Revised: Idle Time

Original: P 0 P
Revised: P 0 P

Original: P 2 P
Revised: P 2 P

Original: Forward
Revised: Forward

Original: Backward
Revised: Backward

Original: Common wisdom is that the use of stale weights must either be avoided, e.g., with the use of microbatches (Huang et al., 2018), be constrained to ensure the consistency of the weights within an accelerator using stashing (Harlap et al., 2018), or by limiting the use of pipelining to very small networks (Mostafa et al., 2017).
Revised: Common wisdom is that the use of stale weights must either be avoided, e.g., with the use of microbatches (Huang et al., 2018), be constrained to ensure the consistency of the weights within an accelerator using stashing (Harlap et al., 2018), or by limiting the use of pipelining to very small networks (Mostafa et al., 2017).

Original: However, these approaches either underutilize accelerators (Huang et al., 2018) or inﬂate memory usage to stash multiple copies of weights (Harlap et al., 2018).
Revised: However, these approaches either underutilize accelerators (Huang et al., 2018) or inﬂate memory usage to stash multiple copies of weights (Harlap et al., 2018).

Original: In this paper we question this common wisdom and explore pipelining that allows for the full utilization of accelerators while using stale weights.
Revised: In this paper we question this common wisdom and explore pipelining that allows for the full utilization of accelerators while using stale weights.

Original: This results in a pipelining scheme that, compared to existing schemes, is simpler to implement, fully utilizes the accelerators and has lower memory overhead.
Revised: This results in a pipelining scheme that, compared to existing schemes, is simpler to implement, fully utilizes the accelerators and has lower memory overhead.

Original: We evaluate this pipelining scheme using 4 CNNs:
Revised: We evaluate this pipelining scheme using 4 CNNs:

Original: LeNet-5 (trained on MNIST),
Revised: LeNet-5 (trained on MNIST),

Original: AlexNet, VGG and ResNet (all trained on CIFAR-10).
Revised: AlexNet, VGG and ResNet (all trained on CIFAR-10).

Original: We analyze the impact of weight staleness and show that if pipelining is limited to early layers in the network, training does converge and the quality of the resulting models is comparable to that of models obtained with non-pipelined training.
Revised: We analyze the impact of weight staleness and show that if pipelining is limited to early layers in the network, training does converge and the quality of the resulting models is comparable to that of models obtained with non-pipelined training.

Original: For the 4 networks, the drop in accuracy is 0.4%, 4%, 0.83% and 1.45%, respectively.
Revised: For the 4 networks, the drop in accuracy is 0.4%, 4%, 0.83% and 1.45%, respectively.

Original: However, inference accuracies drop signiﬁcantly when the pipelining is deeper in the network.
Revised: However, inference accuracies drop signiﬁcantly when the pipelining is deeper in the network.

Original: While this is not a limitation since the bulk of computations that can beneﬁt from pipelining are in the early convolutional layers, we address this through a hybrid scheme that combines pipelined and non-pipelined training to maintain inference accuracy while still delivering performance improvement.
Revised: While this is not a limitation since the bulk of computations that can beneﬁt from pipelining are in the early convolutional layers, we address this through a hybrid scheme that combines pipelined and non-pipelined training to maintain inference accuracy while still delivering performance improvement.

Original: Evaluation shows that our pipelined training delivers a speedup of up to 1.8X on a 2-GPU system.
Revised: Evaluation shows that our pipelined training delivers a speedup of up to 1.8X on a 2-GPU system.

Original: The remainder of this paper is organized as follows.
Revised: The remainder of this paper is organized as follows.

Original: Section 2 brieﬂy describes the backpropagation for training of CNNs.
Revised: Section 2 brieﬂy describes the backpropagation for training of CNNs.

Original: Section 3 details our pipelining scheme.
Revised: Section 3 details our pipelining scheme.

Original: Section 4 describes how non-pipelined and pipelined backpropagation are combined.
Revised: Section 4 describes how non-pipelined and pipelined backpropagation are combined.

Original: Section 5 highlights some of the implementation details.
Revised: Section 5 highlights some of the implementation details.

Original: Experimental evaluation is presented in Section 6.
Revised: Experimental evaluation is presented in Section 6.

Original: Related work is reviewed in Section 7.
Revised: Related work is reviewed in Section 7.

Original: Finally, Section 8 gives concluding remarks and directions for future work.
Revised: Finally, Section 8 gives concluding remarks and directions for future work.

Original:  T HE B ACKPROPAGATION A LGORITHM
Revised:  T HE B ACKPROPAGATION A LGORITHM

Original: The backpropagation algorithm (Rumelhart et al., 1986) consists of two passes: a forward pass that calculates the output error and a backward pass that calculates the error gradients and updates the weights of the network.
Revised: The backpropagation algorithm (Rumelhart et al., 1986) consists of two passes: a forward pass that calculates the output error and a backward pass that calculates the error gradients and updates the weights of the network.

Original: The two passes are performed for input data one mini-batch at a time.
Revised: The two passes are performed for input data one mini-batch at a time.

Original: In the forward pass, a mini-batch is fed into the network, propagating from the ﬁrst to the last layer.
Revised: In the forward pass, a mini-batch is fed into the network, propagating from the ﬁrst to the last layer.

Original: At each layer l , the activations of the layer, denoted by x ( l ) , are computed using the weights of the layer, denoted by W ( l ) .
Revised: At each layer l , the activations of the layer, denoted by x ( l ) , are computed using the weights of the layer, denoted by W ( l ) .

Original: When the output of the network (layer L ) x ( L ) is produced, it is used with the true data label to obtain a training error e for the mini-batch.
Revised: When the output of the network (layer L ) x ( L ) is produced, it is used with the true data label to obtain a training error e for the mini-batch.

Original: In the backward pass, the error e is propagated from the last to the ﬁrst layer.
Revised: In the backward pass, the error e is propagated from the last to the ﬁrst layer.

Original: The error gradients with respect to pre-activations of layer l , denoted by δ ( l ) , are calculated.
Revised: The error gradients with respect to pre-activations of layer l , denoted by δ ( l ) , are calculated.

Original: Further, the error gradients with respect to weights of layer l , ∂e ∂ W ( l ) , are computed using the activations from layer l − 1 (i.e., x ( l − 1) ) and δ ( l ) .
Revised: Further, the error gradients with respect to weights of layer l , ∂e ∂ W ( l ) , are computed using the activations from layer l − 1 (i.e., x ( l − 1) ) and δ ( l ) .

Original: Subsequently, δ ( l ) is used to calculate the δ ( l − 1) .
Revised: Subsequently, δ ( l ) is used to calculate the δ ( l − 1) .

Original: When ∂e ∂ W ( l ) is computed for every layer, the weights are updated using the error gradients.
Revised: When ∂e ∂ W ( l ) is computed for every layer, the weights are updated using the error gradients.

Original: In the forward pass, the activations of the layer l , x ( l ) , cannot be computed until the activations of the previous layers, i.e., x ( l − 1) , are computed.
Revised: In the forward pass, the activations of the layer l , x ( l ) , cannot be computed until the activations of the previous layers, i.e., x ( l − 1) , are computed.

Original: In backward pass, ∂e ∂ W ( l ) can only be computed once x ( l − 1) and δ ( l ) have been computed.
Revised: In backward pass, ∂e ∂ W ( l ) can only be computed once x ( l − 1) and δ ( l ) have been computed.

Original: Moreover, δ ( l ) depends on δ ( l +1) .
Revised: Moreover, δ ( l ) depends on δ ( l +1) .

Original: Finally, for a given mini-batch the backward pass cannot be started until the forward pass is completed and the error e has been determined.
Revised: Finally, for a given mini-batch the backward pass cannot be started until the forward pass is completed and the error e has been determined.

Original: The above dependences ensure that the weights of the layers are updated using the activations and error gradients calculated from the same batch of training data in one iteration of the backpropagation algorithm.
Revised: The above dependences ensure that the weights of the layers are updated using the activations and error gradients calculated from the same batch of training data in one iteration of the backpropagation algorithm.

Original: Only when the weights are updated is the next batch of training data fed into the
Revised: Only when the weights are updated is the next batch of training data fed into the

Original: l l +
Revised: l l +

Original: L Layer:
Revised: L Layer:

Original: W ( 1 )
Revised: W ( 1 )

Original: W ( l )
Revised: W ( l )

Original: W ( l +)
Revised: W ( l +)

Original: W ( L ) x (1) … … … …
Revised: W ( L ) x (1) … … … …

Original: x (l) 𝞭 (l+1)
Revised: x (l) 𝞭 (l+1)

Original: R e g .
Revised: R e g .

Original: output error
Revised: output error

Original: Forward Stage Forward Stage
Revised: Forward Stage Forward Stage

Original: Backward Stage Backward Stage
Revised: Backward Stage Backward Stage

Original: Forward
Revised: Forward

Original: Backward x (l) 𝞭 (l+1) x (L) (L)
Revised: Backward x (l) 𝞭 (l+1) x (L) (L)

Original: 𝞭
Revised: 𝞭

Original: R e g .
Revised: R e g .

Original: network.
Revised: network.

Original: These dependences limit parallelism when a network is partitioned across multiple accelerators and allow only one accelerator to be active at any point.
Revised: These dependences limit parallelism when a network is partitioned across multiple accelerators and allow only one accelerator to be active at any point.

Original: This results in under-utilization of the accelerators.
Revised: This results in under-utilization of the accelerators.

Original: It is this limitation that pipelining addresses.
Revised: It is this limitation that pipelining addresses.

Original:  P IPELINED B ACKPROPAGATION
Revised:  P IPELINED B ACKPROPAGATION

Original: We illustrate our pipelined backpropagation implementation with the L layer network shown in
Revised: We illustrate our pipelined backpropagation implementation with the L layer network shown in

Original: Figure 3, using conceptual pipeline registers.
Revised: Figure 3, using conceptual pipeline registers.

Original: Two registers are inserted between layers l and l + 1 ; one register for the forward pass and a second for the backward pass.
Revised: Two registers are inserted between layers l and l + 1 ; one register for the forward pass and a second for the backward pass.

Original: The forward register stores the activations of layer l ( x ( l ) ).
Revised: The forward register stores the activations of layer l ( x ( l ) ).

Original: The backward register stores the gradients δ ( l +1) of layer l +1 .
Revised: The backward register stores the gradients δ ( l +1) of layer l +1 .

Original: This deﬁnes a 4-stage pipelined backpropagation.
Revised: This deﬁnes a 4-stage pipelined backpropagation.

Original: The forward pass for layers 1 to l forms forward stage
Revised: The forward pass for layers 1 to l forms forward stage

Original: FS 1 .
Revised: FS 1 .

Original: The forward pass for layers l + 1 to L form forward stage FS 2 .
Revised: The forward pass for layers l + 1 to L form forward stage FS 2 .

Original: Similarly, the backwards pass for layers l + 1 to L and 1 to l form backward stages BKS 1 and BKS 2 respectively.
Revised: Similarly, the backwards pass for layers l + 1 to L and 1 to l form backward stages BKS 1 and BKS 2 respectively.

Original: The forward and backward stages are executed in a pipelined fashion on 3 accelerators: one for
Revised: The forward and backward stages are executed in a pipelined fashion on 3 accelerators: one for

Original: FS 1 , one for both FS 2 and BKS 1 , and one for BKS 2 1 .
Revised: FS 1 , one for both FS 2 and BKS 1 , and one for BKS 2 1 .

Original: In cycle 0, mini-batch 0 is fed to FS 1 .
Revised: In cycle 0, mini-batch 0 is fed to FS 1 .

Original: The computations of the forward pass are done as in the traditional non-pipelined implementation.
Revised: The computations of the forward pass are done as in the traditional non-pipelined implementation.

Original: In cycle 1, layer l activations x ( l ) are fed to FS 2 and mini-batch 1 is fed to FS 1 .
Revised: In cycle 1, layer l activations x ( l ) are fed to FS 2 and mini-batch 1 is fed to FS 1 .

Original: In cycle 2, the error for mini-batch 0 computed in FS 2 is directly fed to BKS 1 , the activations of layer l x ( l ) are forwarded to FS 2 and mini-batch 2 is fed to FS 1 .
Revised: In cycle 2, the error for mini-batch 0 computed in FS 2 is directly fed to BKS 1 , the activations of layer l x ( l ) are forwarded to FS 2 and mini-batch 2 is fed to FS 1 .

Original: This pipelined execution is illustrated by the space-time diagram in Figure 4 for 5 mini-batches.
Revised: This pipelined execution is illustrated by the space-time diagram in Figure 4 for 5 mini-batches.

Original: The ﬁgure depicts the mini-batch processed by each accelerator cycles 0 to 6.
Revised: The ﬁgure depicts the mini-batch processed by each accelerator cycles 0 to 6.

Original: At steady state, all the accelerators are active in each cycle of execution.
Revised: At steady state, all the accelerators are active in each cycle of execution.

Original: The above pipelining scheme utilizes weights in FS 1 that are yet to be updated by the errors calculated by FS 2 and BKS 1 .
Revised: The above pipelining scheme utilizes weights in FS 1 that are yet to be updated by the errors calculated by FS 2 and BKS 1 .

Original: At steady state, the activations of a mini-batch in FS 1 are calculated using weights that are 2 execution cycles old, or 2 cycles stale .
Revised: At steady state, the activations of a mini-batch in FS 1 are calculated using weights that are 2 execution cycles old, or 2 cycles stale .

Original: This is reﬂected in Figure 4 by indicating the weights used by each forward stage and the weights updated by each backward stage.
Revised: This is reﬂected in Figure 4 by indicating the weights used by each forward stage and the weights updated by each backward stage.

Original: The weights of a forward stage are subscripted by how stale they are (-ve subscripts).
Revised: The weights of a forward stage are subscripted by how stale they are (-ve subscripts).

Original: Similarly, the weights updated by a backward stage are subscripted by how delayed they are (+ve subscripts).
Revised: Similarly, the weights updated by a backward stage are subscripted by how delayed they are (+ve subscripts).

Original: Further, since the updates of the weights by BKS 2 requires activations calculated for the same mini-batch in FS 1 for all layers in the stage, it is necessary to save these activations until the error gradients with respect to the weights are calculated by BKS 2 .
Revised: Further, since the updates of the weights by BKS 2 requires activations calculated for the same mini-batch in FS 1 for all layers in the stage, it is necessary to save these activations until the error gradients with respect to the weights are calculated by BKS 2 .

Original: Only when the weights are updated using the gradients can these activations be discarded.
Revised: Only when the weights are updated using the gradients can these activations be discarded.

Original: In the general case, we use K pairs of pipeline registers (each pair consisting of a forward register and a backward register) inserted between the layers of the network.
Revised: In the general case, we use K pairs of pipeline registers (each pair consisting of a forward register and a backward register) inserted between the layers of the network.

Original: We describe the placement of the register pairs by the Pipeline Placement Vector , PPV =
Revised: We describe the placement of the register pairs by the Pipeline Placement Vector , PPV =

Original: ( p 1 , p 2 , ..., p K ) , where p i represents the layer number after which a pipeline register pair is inserted.
Revised: ( p 1 , p 2 , ..., p K ) , where p i represents the layer number after which a pipeline register pair is inserted.

Original: Such a placement creates ( K + 1) forward stages, labeled FS i , i = 1 , 2 , ..., K + 

Revised: Such a placement creates ( K + 1) forward stages, labeled FS i , i = 1 , 2 , ..., K +

Original: and ( K + 1) backward stages, labeled BKS i , i = 1 , 2 , ..., K + 1 .
Revised: and ( K + 1) backward stages, labeled BKS i , i = 1 , 2 , ..., K + 1 .

Original: Forward stage FS i and backward stage BKS K − i +2 correspond to the same set of layers.
Revised: Forward stage FS i and backward stage BKS K − i +2 correspond to the same set of layers.

Original: Speciﬁcally, stage FS i contains layers p i
Revised: Speciﬁcally, stage FS i contains layers p i

Original: + 1 to p
Revised: + 1 to p

Original: i +1 , inclusive.
Revised: i +1 , inclusive.

Original: We assign each forward stage and each backward stage to an accelerator, with the exception of the FS K +1 and backward stage BKS 1 , which are assigned to the same accelerator to reduce weight staleness by an execution cycle.
Revised: We assign each forward stage and each backward stage to an accelerator, with the exception of the FS K +1 and backward stage BKS 1 , which are assigned to the same accelerator to reduce weight staleness by an execution cycle.

Original: In total 2 K + 1 accelerators are used.
Revised: In total 2 K + 1 accelerators are used.

Original: We quantify weight staleness as follows.
Revised: We quantify weight staleness as follows.

Original: A forward stage FS i and backward stage BKS K − i +2 use the same weights that are 2( K − i +1) cycles old.
Revised: A forward stage FS i and backward stage BKS K − i +2 use the same weights that are 2( K − i +1) cycles old.

Original: A forward stage FS i must store the activations of
Revised: A forward stage FS i must store the activations of

Original: Steady State
Revised: Steady State

Original: Forward Stage
Revised: Forward Stage

Original: Backward Stage
Revised: Backward Stage

Original: Forward Stage
Revised: Forward Stage

Original: Backward Stage
Revised: Backward Stage

Original: Pipeline Cycles idle mini-batchmini-batchmini-batchmini-batchmini-batch
Revised: Pipeline Cycles idle mini-batchmini-batchmini-batchmini-batchmini-batch

Original: W (1-l)
Revised: W (1-l)

Original: W (1-l) - W (1-l) -
Revised: W (1-l) - W (1-l) -

Original: W (1-l) - W (1-l) -
Revised: W (1-l) - W (1-l) -

Original: W (l+1-L) 
Revised: W (l+1-L) W (l+1-L) W (l+1-L) W (l+1-L)

Original: W (l+1-L) W (l+1-L) 
Revised: W (l+1-L) W (l+1-L) W (l+1-L)

Original: W (l+1-L) W (l+1-L)W (l+1-L)
Revised: W (l+1-L) W (l+1-L)

Original: W (l+1-L)W (l+1-L) W (l+1-L) W (l+1-L)
Revised: W (l+1-L)

Original: W (l+1-L) W (l+1-L)
Revised: W (l+1-L) W (l+1-L)

Original: W (l+1-L) W (l+1-L)
Revised: W (l+1-L) W (l+1-L)

Original: W (l+1-L)
Revised: W (l+1-L)

Original: On the one hand the above pipelined execution allows a potential speedup of 2 K + 1 over the nonpipelined implementation, keeping all the accelerators active at steady state.
Revised: On the one hand the above pipelined execution allows a potential speedup of 2 K + 1 over the nonpipelined implementation, keeping all the accelerators active at steady state.

Original: On the other hand, the use of stale weights may prevent training convergence or may result in a model that has an inferior inference accuracy.
Revised: On the other hand, the use of stale weights may prevent training convergence or may result in a model that has an inferior inference accuracy.

Original: Further, it requires an increase in storage for activations.
Revised: Further, it requires an increase in storage for activations.

Original: Our goal is to assess the beneﬁt of this pipelined execution and the impact of its down sides.
Revised: Our goal is to assess the beneﬁt of this pipelined execution and the impact of its down sides.

Original:  H YBRID P IPELINED /N ON -P IPELINED B ACKPROPAGATION
Revised:  H YBRID P IPELINED /N ON -P IPELINED B ACKPROPAGATION

Original: Hybrid training combines pipelined training with non-pipelined training.
Revised: Hybrid training combines pipelined training with non-pipelined training.

Original: We start with pipelined training and after a number of iterations, we switch to non-pipelined training.
Revised: We start with pipelined training and after a number of iterations, we switch to non-pipelined training.

Original: This can address drops in inference accuracy of resulting models because of weight staleness, but it reduces the performance beneﬁt since during non-pipelined training, the accelerators are under-utilized.
Revised: This can address drops in inference accuracy of resulting models because of weight staleness, but it reduces the performance beneﬁt since during non-pipelined training, the accelerators are under-utilized.

Original: The extent of the speedup obtained by hybrid training with a given number of accelerators is determined by the number of iterations used for pipelined and non-pipelined training.
Revised: The extent of the speedup obtained by hybrid training with a given number of accelerators is determined by the number of iterations used for pipelined and non-pipelined training.

Original: Assume that n np iterations are used to reach the best inference accuracy for non-pipelined training, and that in hybrid training, n p iterations ( n p ≤ n np ) are pipelined followed by n np − n p iterations of non-pipelined training to reach the same inference accuracy as non-pipelined training.
Revised: Assume that n np iterations are used to reach the best inference accuracy for non-pipelined training, and that in hybrid training, n p iterations ( n p ≤ n np ) are pipelined followed by n np − n p iterations of non-pipelined training to reach the same inference accuracy as non-pipelined training.

Original: The speedup of hybrid training with respect to the non-pipelined training with 2 K + 1 accelerators is n np / ( n p / (2 K + 1) + ( n np − n p )) .
Revised: The speedup of hybrid training with respect to the non-pipelined training with 2 K + 1 accelerators is n np / ( n p / (2 K + 1) + ( n np − n p )) .

Original: For large K , then using Amdahl’s law, the speedup approaches an upper bound of n np / ( n np − n p ) .
Revised: For large K , then using Amdahl’s law, the speedup approaches an upper bound of n np / ( n np − n p ) .

Original:  I MPLEMENTATION
Revised:  I MPLEMENTATION

Original: We implement pipelined training in two ways: simulated in Caffe (Jia et al., 2014), where the whole training process is performed on one process with no parallelism, and actual with parallelism across accelerators in PyTorch (Paszke et al., 2017).
Revised: We implement pipelined training in two ways: simulated in Caffe (Jia et al., 2014), where the whole training process is performed on one process with no parallelism, and actual with parallelism across accelerators in PyTorch (Paszke et al., 2017).

Original: The simulated execution is used to analyze statistical convergence, inference accuracy and impact of weight staleness unconstrained by parallelism and communication overhead.
Revised: The simulated execution is used to analyze statistical convergence, inference accuracy and impact of weight staleness unconstrained by parallelism and communication overhead.

Original: The actual execution is used to report performance and PyTorch is used instead of Caffe to leverage its support for collective communication protocols and its ﬂexibility in partitioning a network across multiple accelerators.
Revised: The actual execution is used to report performance and PyTorch is used instead of Caffe to leverage its support for collective communication protocols and its ﬂexibility in partitioning a network across multiple accelerators.

Original: Both Caffe and PyTorch have no support for pipelined training.
Revised: Both Caffe and PyTorch have no support for pipelined training.

Original: Thus both were extended to provide such support.
Revised: Thus both were extended to provide such support.

Original: We develop a custom Caffe layer in Python, which we call a Pipeline Manager Layer (PML), to facilitate the simulated pipelining.
Revised: We develop a custom Caffe layer in Python, which we call a Pipeline Manager Layer (PML), to facilitate the simulated pipelining.

Original: During the forward pass, a PML registers the input from a previous layer and passes the activation to the next layer.
Revised: During the forward pass, a PML registers the input from a previous layer and passes the activation to the next layer.

Original: It also saves the activations for the layers connected to it to be used in the backward pass.
Revised: It also saves the activations for the layers connected to it to be used in the backward pass.

Original: During the backward pass, a PML passes the appropriate error gradients.
Revised: During the backward pass, a PML passes the appropriate error gradients.

Original: It uses the corresponding activations saved during the forward pass to update weights and generate error gradients for the previous stage, using existing weight update mechanisms in Caffe .
Revised: It uses the corresponding activations saved during the forward pass to update weights and generate error gradients for the previous stage, using existing weight update mechanisms in Caffe .

Original: [Table]
Revised: [Table]

Original: To implement actual hardware-accelerated pipelined training, we partition the network onto different accelerators (GPUs), each running its own process.
Revised: To implement actual hardware-accelerated pipelined training, we partition the network onto different accelerators (GPUs), each running its own process.

Original: Asynchronous sends and receives are used for data transfers, but all communication must go through the host CPU, since point-to-point communication between accelerators is not supported in PyTorch .
Revised: Asynchronous sends and receives are used for data transfers, but all communication must go through the host CPU, since point-to-point communication between accelerators is not supported in PyTorch .

Original: This increases communication overhead.
Revised: This increases communication overhead.

Original: Similar to the PMLs in Caffe , the activations computed on one GPU are copied to the next GPU (via the CPU) in the forward pass and the error gradients are sent (again via the CPU) to the preceding
Revised: Similar to the PMLs in Caffe , the activations computed on one GPU are copied to the next GPU (via the CPU) in the forward pass and the error gradients are sent (again via the CPU) to the preceding

Original: GPU during the backward pass.
Revised: GPU during the backward pass.

Original: The GPUs are running concurrently, achieving pipeline parallelism.
Revised: The GPUs are running concurrently, achieving pipeline parallelism.

Original:  E VALUATION 6. S ETUP , M ETHODOLOGY AND M ETRICS
Revised:  E VALUATION 6. S ETUP , M ETHODOLOGY AND M ETRICS

Original: Simulated pipelining is evaluated on a machine with one Nvidia GTX

Revised: Simulated pipelining is evaluated on a machine with one Nvidia GTX

Original: GPU with 6 GB of memory and an Intel i9-7940X CPU with 64 GB of RAM.
Revised: GPU with 6 GB of memory and an Intel i9-7940X CPU with 64 GB of RAM.

Original: The performance of actual pipelining is evaluated using two Nvidia GTX1060 GPUs, each with 6 GB of memory, hosted in an Intel i79700K machine with 32 GB of RAM.
Revised: The performance of actual pipelining is evaluated using two Nvidia GTX1060 GPUs, each with 6 GB of memory, hosted in an Intel i79700K machine with 32 GB of RAM.

Original: We use four CNNs in our evaluation:
Revised: We use four CNNs in our evaluation:

Original: LeNet-5 (LeCun et al., 1998) trained on MNIST (LeCun & Cortes), AlexNet (Krizhevsky et al., 2012), VGG-16 (Simonyan & Zisserman, 2014) and
Revised: LeNet-5 (LeCun et al., 1998) trained on MNIST (LeCun & Cortes), AlexNet (Krizhevsky et al., 2012), VGG-16 (Simonyan & Zisserman, 2014) and

Original: ResNet (He et al., 2016), all trained on CIFAR-10 (Krizhevsky et al.).
Revised: ResNet (He et al., 2016), all trained on CIFAR-10 (Krizhevsky et al.).

Original: For ResNet, we experiment with different depths: 20, 56, 110, 224 and 362.
Revised: For ResNet, we experiment with different depths: 20, 56, 110, 224 and 362.

Original: We train these CNNs mostly following their original setting (LeCun et al., 1998)
Revised: We train these CNNs mostly following their original setting (LeCun et al., 1998)

Original: (Krizhevsky et al., 2012) (Simonyan & Zisserman, 2014)
Revised: (Krizhevsky et al., 2012) (Simonyan & Zisserman, 2014)

Original: (He et al., 2016) with minor variations to the hyperparameters, as described in Appendix A.
Revised: (He et al., 2016) with minor variations to the hyperparameters, as described in Appendix 8.

Original: We evaluate the effectiveness of pipelined training in terms of its training convergence and its Topinference accuracy, compared to those of the non-pipelined training.
Revised: We evaluate the effectiveness of pipelined training in terms of its training convergence and its Topinference accuracy, compared to those of the non-pipelined training.

Original: We use the speedup to evaluate performance improvements.
Revised: We use the speedup to evaluate performance improvements.

Original: It is deﬁned as the ratio of the training time of the non-pipelined implementation on single communication-free GPU to the training time of the pipelined training.
Revised: It is deﬁned as the ratio of the training time of the non-pipelined implementation on single communication-free GPU to the training time of the pipelined training.

Original: 6. T RAINING C ONVERGENCE AND I NFERENCE A CCURACY
Revised: 6. T RAINING C ONVERGENCE AND I NFERENCE A CCURACY

Original: Figure 5 shows the improvements in the inference accuracies for both pipelined and non-pipelined training as a function of the number of training iterations (each iteration corresponds to a minibatch).
Revised: Figure 5 shows the improvements in the inference accuracies for both pipelined and non-pipelined training as a function of the number of training iterations (each iteration corresponds to a minibatch).

Original: The pipelined training is done using 4, 6, 8 and 10 stages.
Revised: The pipelined training is done using 4, 6, 8 and 10 stages.

Original: Table 1 shows where the registers are inserted in the networks using their PPV deﬁned in Section 3.
Revised: Table 1 shows where the registers are inserted in the networks using their PPV deﬁned in Section 3.

Original: Figure 5 shows that for all the networks, both pipelined and non-pipelined training have similar convergence patterns.
Revised: Figure 5 shows that for all the networks, both pipelined and non-pipelined training have similar convergence patterns.

Original: They converge in more or less the same number of iterations for a given number of pipeline stages, albeit to different inference accuracies.
Revised: They converge in more or less the same number of iterations for a given number of pipeline stages, albeit to different inference accuracies.

Original: This indicates that our approach to pipelined training with stale weights does converge, similar to non-pipelined training.
Revised: This indicates that our approach to pipelined training with stale weights does converge, similar to non-pipelined training.

Original: Table 2 shows the inference accuracy obtained after up to 30,000 iterations of training.
Revised: Table 2 shows the inference accuracy obtained after up to 30,000 iterations of training.

Original: For LeNet-5, the inference accuracy drop is within 0.5%.
Revised: For LeNet-5, the inference accuracy drop is within 0.5%.

Original: However, for the other networks, there is a small drop in inference accuracy with 4 and 6 stages.
Revised: However, for the other networks, there is a small drop in inference accuracy with 4 and 6 stages.

Original: AlexNet has about 4% drop in inference accuracy, but for
Revised: AlexNet has about 4% drop in inference accuracy, but for

Original: VGG-16 the inference accuracy drop is within 2.4%, and for ResNet-20 the accuracy drop is within 3.5%.
Revised: VGG-16 the inference accuracy drop is within 2.4%, and for ResNet-20 the accuracy drop is within 3.5%.

Original: Thus, the resulting model quality is comparable to that of a non-pipelining-trained model.
Revised: Thus, the resulting model quality is comparable to that of a non-pipelining-trained model.

Original: However, with deeper pipelining (i.e., 8 and 10 stages), inference accuracy signiﬁcantly drops.
Revised: However, with deeper pipelining (i.e., 8 and 10 stages), inference accuracy signiﬁcantly drops.

Original: There is a 12% and a 8.5% inference accuracy drop for VGG-16 and ResNet-20 respectively.
Revised: There is a 12% and a 8.5% inference accuracy drop for VGG-16 and ResNet-20 respectively.

Original: In this case, the model quality is not comparable to that of the non-pipelined training.
Revised: In this case, the model quality is not comparable to that of the non-pipelined training.

Original: This results conﬁrm what is reported in the literature (Harlap et al., 2018) and can be attributed to the use of stale weights.
Revised: This results conﬁrm what is reported in the literature (Harlap et al., 2018) and can be attributed to the use of stale weights.

Original: Below we further explore the impact of stale weights on inference accuracy.
Revised: Below we further explore the impact of stale weights on inference accuracy.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: 6. I MPACT OF W EIGHT S TALENESS
Revised: 6. I MPACT OF W EIGHT S TALENESS

Original: We wish to better understand the impact of the number of pipeline stages and the location of these stages in the network on inference accuracy.
Revised: We wish to better understand the impact of the number of pipeline stages and the location of these stages in the network on inference accuracy.

Original: We focus on ResNet-20 because of its relatively small size and regular structure.
Revised: We focus on ResNet-20 because of its relatively small size and regular structure.

Original: It consists of 3 residual function groups with 3 residual function blocks within each group.
Revised: It consists of 3 residual function groups with 3 residual function blocks within each group.

Original: In spite of this relatively small size and regular structure, it enables us to create pipelines with up to 20 stages by inserting pipeline register pairs within residual function blocks.
Revised: In spite of this relatively small size and regular structure, it enables us to create pipelines with up to 20 stages by inserting pipeline register pairs within residual function blocks.

Original: We conduct two experiments.
Revised: We conduct two experiments.

Original: In the ﬁrst, we increase the number of pipeline stages (from earlier layers to latter layers) and measure the inference accuracy of the resulting model.
Revised: In the ﬁrst, we increase the number of pipeline stages (from earlier layers to latter layers) and measure the inference accuracy of the resulting model.

Original: The results are shown in Table 3, which gives the inference accuracy of pipelined training after 100,000 iterations, as the number of pipeline stages increases.
Revised: The results are shown in Table 3, which gives the inference accuracy of pipelined training after 100,000 iterations, as the number of pipeline stages increases.

Original: The 8-stage pipelined training is created by a PPV of (3,5,7), and the subsequent pipeline schemes are created by adding pipeline registers after everylayers after layer 7.
Revised: The 8-stage pipelined training is created by a PPV of (3,5,7), and the subsequent pipeline schemes are created by adding pipeline registers after everylayers after layer 7.

Original: Clearly, the greater the number stages, the worse is the resulting model quality.
Revised: Clearly, the greater the number stages, the worse is the resulting model quality.

Original: The number of stale weights used in the pipelined training increases as the number of pipeline stages increases.
Revised: The number of stale weights used in the pipelined training increases as the number of pipeline stages increases.

Original: Thus, Figure 6 depicts the inference accuracy as a function of the percentage of weights that are stale.
Revised: Thus, Figure 6 depicts the inference accuracy as a function of the percentage of weights that are stale.

Original: The curve labeled “Increasing Stages” shows that the drop in inference accuracy increases as the percentage of stale weights increases.
Revised: The curve labeled “Increasing Stages” shows that the drop in inference accuracy increases as the percentage of stale weights increases.

Original: In the second experiment, we investigate the impact of the degree of staleness (Section 3).
Revised: In the second experiment, we investigate the impact of the degree of staleness (Section 3).

Original: Only one pair of pipeline registers is inserted.
Revised: Only one pair of pipeline registers is inserted.

Original: The position of this register slides from the beginning of the network to its end.
Revised: The position of this register slides from the beginning of the network to its end.

Original: At every position, the percentage of stale weights remains the same as in the ﬁrst experiment, but all stale weights have the same degree of staleness.
Revised: At every position, the percentage of stale weights remains the same as in the ﬁrst experiment, but all stale weights have the same degree of staleness.

Original: The result of this experiment is shown by the curve labeled “Sliding Stage” in Figure 6.
Revised: The result of this experiment is shown by the curve labeled “Sliding Stage” in Figure 6.

Original: The curve shows the inference accuracy also drops as the percentage of stale weights increases.
Revised: The curve shows the inference accuracy also drops as the percentage of stale weights increases.

Original: However, it also indicates that the drop of inference accuracy remains more or less the same as in the ﬁrst experiment in which the degree of staleness is higher.
Revised: However, it also indicates that the drop of inference accuracy remains more or less the same as in the ﬁrst experiment in which the degree of staleness is higher.

Original: Thus, the percentage of stale weight appears to be what determines the drop in inference accuracy and not the degree of staleness of the weights.
Revised: Thus, the percentage of stale weight appears to be what determines the drop in inference accuracy and not the degree of staleness of the weights.

Original: The percentage of stale weight is determined by where the last pair of pipeline registers are placed in the network.
Revised: The percentage of stale weight is determined by where the last pair of pipeline registers are placed in the network.

Original: It is the position of this pair that determines the loss in inference accuracy.
Revised: It is the position of this pair that determines the loss in inference accuracy.

Original: Therefore, it is desirable to place this last pair of registers as early as possible in the network so as to minimize the drop in inference accuracy.
Revised: Therefore, it is desirable to place this last pair of registers as early as possible in the network so as to minimize the drop in inference accuracy.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: While at ﬁrst glance this may seem to limit pipelining, it is important to note that the bulk of computations in a CNN is in the ﬁrst few convolutional layers in the network.
Revised: While at ﬁrst glance this may seem to limit pipelining, it is important to note that the bulk of computations in a CNN is in the ﬁrst few convolutional layers in the network.

Original: Inserting pipeline registers for these early layers can result in both a large number of stages that are computationally balanced.
Revised: Inserting pipeline registers for these early layers can result in both a large number of stages that are computationally balanced.

Original: For example, our proﬁling of the runtime of ResNet-20 shows that the ﬁrst three residual functions take more than 50% of the training runtime.
Revised: For example, our proﬁling of the runtime of ResNet-20 shows that the ﬁrst three residual functions take more than 50% of the training runtime.

Original: This favors more pipeline stages at the beginning of the network.
Revised: This favors more pipeline stages at the beginning of the network.

Original: Such placement has the desirable effect of reducing the drop in inference accuracy while obtaining relatively computationally balanced pipeline stages.
Revised: Such placement has the desirable effect of reducing the drop in inference accuracy while obtaining relatively computationally balanced pipeline stages.

Original: 6. E FFECTIVENESS OF H YBRID T RAINING
Revised: 6. E FFECTIVENESS OF H YBRID T RAINING

Original: We demonstrate the effectiveness of hybrid training using only ResNet-20 for brevity.
Revised: We demonstrate the effectiveness of hybrid training using only ResNet-20 for brevity.

Original: Figureshows the inference accuracy for 20K iterations of pipelined training followed by either 10K or 20K iterations of non-pipelined training.
Revised: Figureshows the inference accuracy for 20K iterations of pipelined training followed by either 10K or 20K iterations of non-pipelined training.

Original: This inference accuracy is compared to 30K iterations of either non-pipelined or pipelined training with PPV (5,12,17).
Revised: This inference accuracy is compared to 30K iterations of either non-pipelined or pipelined training with PPV (5,12,17).

Original: The ﬁgure demonstrates that hybrid training converges in a similar manner to both pipelined and non-pipelined training.
Revised: The ﬁgure demonstrates that hybrid training converges in a similar manner to both pipelined and non-pipelined training.

Original: Table 4 shows the resulting inference accuracies.
Revised: Table 4 shows the resulting inference accuracies.

Original: The table shows that the 20K+10K hybrid training produces a model with accuracy that is comparable to that of the non-pipelined model.
Revised: The table shows that the 20K+10K hybrid training produces a model with accuracy that is comparable to that of the non-pipelined model.

Original: Further, with an additional 10K iterations of non-pipelined training, the model quality is slightly better than that of the non-pipelined model.
Revised: Further, with an additional 10K iterations of non-pipelined training, the model quality is slightly better than that of the non-pipelined model.

Original: This demonstrates the effectiveness of hybrid training.
Revised: This demonstrates the effectiveness of hybrid training.

Original: 6. P IPELINED AND H YBRID T RAINING P ERFORMANCE
Revised: 6. P IPELINED AND H YBRID T RAINING P ERFORMANCE

Original: We implement 4-stage pipelined training ResNet-20/56/110/224/362 on a 2-GPU system.
Revised: We implement 4-stage pipelined training ResNet-20/56/110/224/362 on a 2-GPU system.

Original: Each
Revised: Each

Original: GPU is responsible for one forward stage and one backward stage.
Revised: GPU is responsible for one forward stage and one backward stage.

Original: Thus, the maximum speedup that can be obtained is 2.
Revised: Thus, the maximum speedup that can be obtained is 2.

Original: We train every ResNet for 164 epochs.
Revised: We train every ResNet for 200 epochs.

Original: Table 5 shows the inference accuracies with and without pipelining as model as the measured speedups of pipelined training over the non-pipelined one.
Revised: Table 5 shows the inference accuracies with and without pipelining as model as the measured speedups of pipelined training over the non-pipelined one.

Original: The table indicates that the quality of the models produced by pipelined training is comparable to those achieved by the simulated pipelining on Caffe .
Revised: The table indicates that the quality of the models produced by pipelined training is comparable to those achieved by the simulated pipelining on Caffe .

Original: The table further shows that speedup exists for all networks.
Revised: The table further shows that speedup exists for all networks.

Original: Indeed, for ResNet-362, the speedup is 1.81X.
Revised: Indeed, for ResNet-362, the speedup is 1.82X.

Original: This is equivalent to about 90% utilization for each GPU.
Revised: This is equivalent to about 90% utilization for each GPU.

Original: The table also reﬂects that as the networks get larger, the speedup improves.
Revised: The table also reﬂects that as the networks get larger, the speedup improves.

Original: This reﬂects that with larger networks, the ratio of computation to communication overhead is higher, leading to better speedups.
Revised: This reﬂects that with larger networks, the ratio of computation to communication overhead is higher, leading to better speedups.

Original: Moreover, we combine the 4-stage pipelined training described above with non-pipelined training to demonstrate the performance of hybrid training.
Revised: Moreover, we combine the 4-stage pipelined training described above with non-pipelined training to demonstrate the performance of hybrid training.

Original: We train every ResNet using pipelined training for 82 epochs and follow it up by 82 epochs of non-pipelined training.
Revised: We train every ResNet using pipelined training for 100 epochs and follow it up by 100 epochs of non-pipelined training.

Original: Because the maximum speedup for the pipelined training is 2 and only half the training epochs is accelerated, the maximum speedup for this hybrid training is s =
Revised: Because the maximum speedup for the pipelined training is 2 and only half the training epochs is accelerated, the maximum speedup for this hybrid training is s =

Original: t/ ( t/ 2 + t/ 4) = 1 .
Revised: t/ ( t/ 2 + t/ 4) = 1 .

Original: 33 , where t is the training time of non-
Revised: 33 , where t is the training time of non-

Original: pipelined

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  training.
Revised: pipelined training.

Original: Table 5 shows the inference accuracies and speedup of the hybrid training for each ResNet and validates that hybrid training can produce a model quality that is comparable to the baseline non-pipelined training while speeding up the training process.
Revised: Table 5 shows the inference accuracies and speedup of the hybrid training for each ResNet and validates that hybrid training can produce a model quality that is comparable to the baseline non-pipelined training while speeding up the training process.

Original: As network size grows, the speedup reaches 1.29X, approaching the theoretical limit 1.33X.
Revised: As network size grows, the speedup reaches 1.29X, approaching the theoretical limit 1.33X.

Original: 6. M EMORY U SAGE
Revised: 6. M EMORY U SAGE

Original: Pipelined training requires the saving of intermediate activations, as described earlier in Section 3, leading to an increase in memory footprint.
Revised: Pipelined training requires the saving of intermediate activations, as described earlier in Section 3, leading to an increase in memory footprint.

Original: This increase in memory is a function of not only the placement of the pipeline registers, but also of the network architecture and the number of inputs in a mini-batch (batch size).
Revised: This increase in memory is a function of not only the placement of the pipeline registers, but also of the network architecture and the number of inputs in a mini-batch (batch size).

Original: We calculate the memory usage of the 4-stage pipelined ResNet training above to show that this increase is modest for our pipelining scheme.
Revised: We calculate the memory usage of the 4-stage pipelined ResNet training above to show that this increase is modest for our pipelining scheme.

Original: Speciﬁcally, we use torchsummary in PyTorch to report memory usage for weights and activations for a network and calculate the additional memory required by the additional copies of activations.
Revised: Speciﬁcally, we use torchsummary in PyTorch to report memory usage for weights and activations for a network and calculate the additional memory required by the additional copies of activations.

Original: The results are shown in Table 6.
Revised: The results are shown in Table 6.

Original: Assuming a batch size of 128, the percentage increase in size is close to 60% except for ResNet-20.
Revised: Assuming a batch size of 128, the percentage increase in size is close to 60% except for ResNet-20.

Original: 6. C OMPARISON TO E XISTING W ORK
Revised: 6. C OMPARISON TO E XISTING W ORK

Original: We compare our pipelined training scheme with two key existing systems: PipeDream (Harlap et al., 2018) and GPipe (Huang et al., 2018).
Revised: We compare our pipelined training scheme with two key existing systems: PipeDream (Harlap et al., 2018) and GPipe (Huang et al., 2018).

Original: We do so on three aspects: the pipelining scheme, performance and memory usage.
Revised: We do so on three aspects: the pipelining scheme, performance and memory usage.

Revised: We believe that PipeDream and GPipe are representative of existing key approaches that implement pipelined training, including Decoupled Backpropagation (DDG) (Huo et al., 2018b) and Feature Replay (FR) (Huo et al., 2018a)

Original:  Our pipelining scheme is simpler than that of PipeDream and GPipe in that we do not require weight stashing nor do we divide mini-batches into micro-batches.
Revised: (discussed in Section 7). Our pipelining scheme is simpler than that of PipeDream and GPipe in that we do not require weight stashing nor do we divide mini-batches into micro-batches.

Original: This leads to less communication overhead, and is amicable to rapid realization in machine learning framework such as PyTorch or in actual hardware such as Xilinx’s
Revised: This leads to less communication overhead, and is amicable to rapid realization in machine learning framework such as PyTorch or in actual hardware such as Xilinx’s

Original: xDNN
Revised: xDNN

Original: FPGA accelerators (Xilinx, 2019).
Revised: FPGA accelerators (Xilinx, 2019).

Original: Our pipelining scheme, as PipeDream, eliminates bubbles that exist in the pipeline leading to better performance.
Revised: Our pipelining scheme, as PipeDream, eliminates bubbles that exist in the pipeline leading to better performance.

Original: For example, we obtain a speedup of 1.7X for ResNet-110 using 2 GPUs in contrast to GPipe that obtains a speedup of roughly 1.3X for ResNet-101 using 2 TPUs.
Revised: For example, we obtain a speedup of 1.7X for ResNet-110 using 2 GPUs in contrast to GPipe that obtains a speedup of roughly 1.3X for ResNet-101 using 2 TPUs.

Original: We also obtain similar performance compared to PipeDream for similar networks.
Revised: We also obtain similar performance compared to PipeDream for similar networks.

Original: When the number of pipeline stages grows, pipeline bubbles exhibits more negative effect on performance shown in GPipe on a 4-partition pipelined ResNet-101 using 4 TPUs as its bubble overhead doubled compared to that of the 2-partition pipelined ResNet-101.
Revised: When the number of pipeline stages grows, pipeline bubbles exhibits more negative effect on performance shown in GPipe on a 4-partition pipelined ResNet-101 using 4 TPUs as its bubble overhead doubled compared to that of the 2-partition pipelined ResNet-101.

Original: Our scheme uses less memory compared to PipeDream, although it introduces more memory overhead compared to GPipe.
Revised: Our scheme uses less memory compared to PipeDream, although it introduces more memory overhead compared to GPipe.

Original: PipeDream saves intermediate activations during training, as we do.
Revised: PipeDream saves intermediate activations during training, as we do.

Original: However, it also saves multiple copies of a network’s weights for weight stashing.
Revised: However, it also saves multiple copies of a network’s weights for weight stashing.

Original: The memory footprint increase due to this weight stashing depends on the network architecture, including the number of weights and activations, as well as on the size of the mini-batch.
Revised: The memory footprint increase due to this weight stashing depends on the network architecture, including the number of weights and activations, as well as on the size of the mini-batch.

Original: For example, for VGG-16 trained on
Revised: For example, for VGG-16 trained on

Original: CIFAR-10 with a mini-batch size of 128 using a 4-stage, pipelined training, we estimate our pipelin- ing methodology to use 49% less memory compared PipeDream.
Revised: CIFAR-10 with a mini-batch size of 128 using a 4-stage, pipelined training, we estimate our pipelining methodology to use 49% less memory compared PipeDream.

Original: Similarly for VGG-16 trained on
Revised: Similarly for VGG-16 trained on

Original: ImageNet (Deng et al., 2009) and a mini-batch size of 32, our scheme uses 29% less memory.
Revised: ImageNet (Deng et al., 2009) and a mini-batch size of 32, our scheme uses 29% less memory.

Original: We estimate the memory increase due to weight stashing also using tourchsummary .
Revised: We estimate the memory increase due to weight stashing also using tourchsummary .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: There has been considerable work that explores parallelism in the training of deep neural networks, both data parallelism (Chen et al., 2016;
Revised: There has been considerable work that explores parallelism in the training of deep neural networks. There are several approaches to exploiting parallelism. One approach is to exploit data parallelism (Chen et al., 2016;

Original: Cui et al., 2016; Goyal et al., 2017; Zhang et al., 2017) and model parallelism (Kim et al., 2016; Lee et al., 2014; Chilimbi et al., 2014; Dean et al., 2012;
Revised: Cui et al., 2016; Goyal et al., 2017; Zhang et al., 2017; Dean et al., 2012; Wang et al., 2019), in which each accelerator obtains a full copy of the model and processes different mini-batches of training data simultaneously. At the end of each training iteration, the gradients produced by all accelerators are aggregated and used to update weights for all copies of the model, synchronously (Chen et al., 2016; Goyal et al., 2017) or asynchronously (Dean et al., 2012).

Revised: A centralized parameter server is usually used to facilitate data communication (Cui et al., 2016; Dean et al., 2012).

Revised: Although the training is performed in parallel, the communication overhead can be signiﬁcant (Wang et al., 2019).

Revised: A second approach is to exploit model parallelism (Kim et al., 2016; Lee et al., 2014; Chilimbi et al., 2014; Dean et al., 2012;

Original: Low et al., 2012).
Revised: Low et al., 2012).

Revised: In this approach, a model is partitioned onto different accelerators (Kim et al., 2016; Lee et al., 2014;

Revised: Low et al., 2012; Chilimbi et al., 2014; Dean et al., 2012).

Original: In contrast, there are been only  a few studies that explore pipelined parallelism (Petrowski et al., 1993;
Revised: Each accelerator is only responsible for updating the weights for the portion of the model assigned to it. This approach is often used when a model is large and cannot ﬁt into the memory of a single accelerator. However, because of the data dependences described in Section 2, only one accelerator is active during the training process, resulting in under-utilization of accelerators resources. Moreover, inter-layer activations and gradients across two consecutive stages needs to be communicated during training, adding more overhead to the entire process.

Revised: Pipelined parallelism addresses the under-utilization of accelerators resources for the training of large models.

Original:  Chen et al., 2012; Mostafa et al., 2017; Harlap et al., 2018; Huang et al., 2018).
Revised: There have been a few studies that explore pipelined parallelism (Petrowski et al., 1993; Chen et al., 2012; Mostafa et al., 2017; Harlap et al., 2018; Huang et al., 2018; Huo et al., 2018b;a), which we review in this section. PipeDream (Harlap et al., 2018) implements pipelined training for large neural networks such as VGG-16, Inception-v3 and S2VT across multiple GPUs.

Original: This work either limited the use of pipelining to small networks (Petrowski et al., 1993; Chen et al., 2012; Mostafa et al., 2017) or explored pipelining for large networks but either avoided the use of stale weights (Huang et al., 2018), reducing pipeline efﬁciency, or constrained the staleness through the use of  multiple versions of weights (Harlap et al., 2018), leading to increased memory footprint.
Revised: However, in their implementation, they limited the usage of stale weights by weight stashing , i.e., keeping multiple versions of network parameters (weights) during training. This increases the memory footprint of training. In contrast, we do not maintain multiple copies of weights during training, therefore reducing the memory footprint of pipelined training. GPipe (Huang et al., 2018) implements a library in Tensorﬂow to enable pipelined parallelism for the training of large neural networks.

Revised: GPipe pipelines micro-batches within each mini-batch to keep the gradients consistently accumulated.

Revised: This eliminates the use of stale weight during training, but it does so at the expense of “pipeline bubbles” at steady state.

Revised: GPipe utilizes these bubbles to reduce the memory footprint by re-computing forward activations instead of storing them.

Revised: In contrast, our work has no pipeline bubble and thus dedicates computing resources to compute forward pass and backward pass only once during each training iteration.

Revised: Huo et al.

Revised: (Huo et al., 2018b) implement decoupled backpropagation (DDG) using delayed gradient updates.

Revised: They show that DDG guarantees convergence through a rigorous convergence analysis.

Original: In contrast to  the above work on pipelined training, we use pipelining with unconstrained stale weights, resulting in full pipeline utilization with no increase in memory usage.
Revised: Similar to PipeDream, DDG uses multiple copies of the weights and thus increases memory footprint. Further, DDG pipelines only the backward pass of training, leaving forward pass un-pipelined. Huo et al.

Revised: (Huo et al., 2018a) follow up by proposing feature replay (FR) that re-computes activations during backward pass, similar to GPipe, resulting less memory footprint and improved inference accuracy than DDG.

Revised: In contrast, we pipeline both forward and backward pass without maintaining multiple copies of weights or re-computing forward activations during backward pass.

Revised: Thus, in summary, our work contrasts to the above work on pipelined training, in that we use pipelining with unconstrained stale weights, resulting in full pipeline utilization with a modest increase in memory usage.

Original: We extend earlier work by studying the impact of weights staleness on the quality of the model.
Revised: We extend earlier work by studying the impact of weights staleness on the quality of the model.

Original: We show that it is effective to use stale weights if the pipelining is in early layers, which is where the bulk of computations exist.
Revised: We show that it is effective to use stale weights if the pipelining is in early layers, which is where the bulk of computations exist.

Original: Further we also extend earlier work through hybrid training, which combines both pipelined and non-pipelined training.
Revised: Further we also extend earlier work through hybrid training, which combines both pipelined and non-pipelined training.

Original: We compare the performance and memory footprint increase of our scheme to existing work in Section 6.7.
Revised: We compare the performance and memory footprint increase of our scheme to existing work in Section 6.7.

Original:  C ONCLUDING R EMARKS
Revised:  C ONCLUDING R EMARKS

Original: We evaluate pipelined execution of backpropagation for the training of CNNs in a way that fully utilizes accelerators, achieving a speedup of 1.81X on the 2-GPU system, and does not signiﬁcantly increase memory usage, unlike previous work.
Revised: We evaluate pipelined execution of backpropagation for the training of CNNs in a way that fully utilizes accelerators, achieving a speedup of 1.82X on the 2-GPU system, and does not signiﬁcantly increase memory usage, unlike previous work.

Original: We show that pipelining training with stale weights does converge.
Revised: We show that pipelining training with stale weights does converge.

Original: Further, we show that the inference accuracies of the resulting models are comparable to those of models obtained with traditional backpropagation, but only when pipelining is implemented in the early layers of the network, with inference accuracy drop within 1.45% on 4stage pipelined training except for AlexNet.
Revised: Further, we show that the inference accuracies of the resulting models are comparable to those of models obtained with traditional backpropagation, but only when pipelining is implemented in the early layers of the network, with inference accuracy drop within 1.45% on 4stage pipelined training except for AlexNet.

Original: This does not limit the beneﬁt of pipelining since the bulk of computations is in the early convolutional layers.
Revised: This does not limit the beneﬁt of pipelining since the bulk of computations is in the early convolutional layers.

Original: When pipelining is implemented deeper in the network, the inference accuracies do drop signiﬁcantly, but we can compensate for this drop by combining pipelined with non-pipelined training, albeit with lower performance gains, obtaining model quality with an average of 0.19% better than the baseline in inference accuracies for ResNets.
Revised: When pipelining is implemented deeper in the network, the inference accuracies do drop signiﬁcantly, but we can compensate for this drop by combining pipelined with non-pipelined training, albeit with lower performance gains, obtaining model quality with an average of 0.19% better than the baseline in inference accuracies for ResNets.

Original: This work can be extended in a number of directions.
Revised: This work can be extended in a number of directions.

Original: One direction is to evaluate the approach with a larger number of accelerators since pipelined parallelism is known to scale naturally with the number of accelerators.
Revised: One direction is to evaluate the approach with a larger number of accelerators since pipelined parallelism is known to scale naturally with the number of accelerators.

Original: Another is to evaluate the approach on larger datasets, such as ImageNet.
Revised: Another is to evaluate the approach on larger datasets, such as ImageNet.

Original: Finally, pipelined parallelism lends itself to hardware implementation.
Revised: Finally, pipelined parallelism lends itself to hardware implementation.

Original: Thus, another direction for future work is to evaluate pipelined parallelism using Field Programmable Gate Array (FPGA) or
Revised: Thus, another direction for future work is to evaluate pipelined parallelism using Field Programmable Gate Array (FPGA) or

Original: ASIC accelerators.
Revised: ASIC accelerators.

Original: A T RAINING H YPERPARAMETERS 
Revised: A T RAINING H YPERPARAMETERS FOR S IMULATED P IPELINED T RAINING

Original: LeNet-5 is trained on the MNIST dataset with Stochastic Gradient Descent (SGD) using a learning rate of 0.01 with inverse learning policy, a momentum of 0.9, a weight decay of 0.0005 and a minibatch size of 100 and for 30,000 iterations.
Revised: LeNet-5 is trained on the MNIST dataset with Stochastic Gradient Descent (SGD) using a learning rate of 0.01 with inverse learning policy, a momentum of 0.9, a weight decay of 0.0005 and a minibatch size of 100 and for 30,000 iterations.

Original: The progression of inference accuracy during training is recorded with 300 tests
Revised: The progression of inference accuracy during training is recorded with 300 tests

Original: AlexNet is trained on the CIFAR-10 dataset with SGD with Nesterov momentum using a learning rate of 0.001 that is decreased by 10x twice during training, a momentum of 0.9, a weight decay of 0.004 and a mini-batch size of 100 for 250,000 iterations.
Revised: AlexNet is trained on the CIFAR-10 dataset with SGD with Nesterov momentum using a learning rate of 0.001 that is decreased by 10x twice during training, a momentum of 0.9, a weight decay of 0.004 and a mini-batch size of 100 for 250,000 iterations.

Original: One test is performed every epoch to record the progression of inference accuracy.
Revised: One test is performed every epoch to record the progression of inference accuracy.

Original: VGG-16 is trained on CIFAR-10 dataset with SGD with Nesterov momentum using a learning rate starting at 0.1 that is decreased by half every 50 epochs during training, a momentum of 0.9, a weight decay of 0.0005 and a mini-batch size of 100 for 250,000.
Revised: VGG-16 is trained on CIFAR-10 dataset with SGD with Nesterov momentum using a learning rate starting at 0.1 that is decreased by half every 50 epochs during training, a momentum of 0.9, a weight decay of 0.0005 and a mini-batch size of 100 for 250,000.

Original: Since it is relatively more difﬁcult to train VGG-16 compared to other models, batch normalization and dropout are used during training throughout the network.
Revised: Since it is relatively more difﬁcult to train VGG-16 compared to other models, batch normalization and dropout are used during training throughout the network.

Original: One test is performed every epoch to record the progression of inference accuracy.
Revised: One test is performed every epoch to record the progression of inference accuracy.

Original: ResNet is trained on CIFAR-10 dataset with SGD using a learning rate starting at 0.1 and 0.01 for non-pipelined and pipelined training respectively, that is decreased by 10x twice during training, a momentum of 0.9, a weight decay of 0.0001 and a mini-batch size of 128 for 100,000 iterations.
Revised: ResNet is trained on CIFAR-10 dataset with SGD using a learning rate starting at 0.1 and 0.01 for non-pipelined and pipelined training respectively, that is decreased by 10x twice during training, a momentum of 0.9, a weight decay of 0.0001 and a mini-batch size of 128 for 100,000 iterations.

Original: Batch normalization is used during training throughout the network.
Revised: Batch normalization is used during training throughout the network.

Original: One test is performed every 100 iterations to record the progression of inference accuracy.
Revised: One test is performed every 100 iterations to record the progression of inference accuracy.

Revised: B T RAINING H YPERPARAMETERS FOR A CTUAL P IPELINED T RAINING

Revised: For the baseline non-piplined training, ResNet-20/56/110/224/362 is trained on CIFAR-10 dataset for 200 epochs with SGD using a learning rate of 0.1 that is decreased by a factor of 10 twice (at epoch 100 and 150), a momentum of 0.9, a weight decay of 0.0001 and a mini-batch size of 128.

Revised: Batch normalization is used during training throughout the network.

Revised: This set of hyperparameters can be found at https://github.com/akamaster/pytorch resnet cifar10.

Revised: For the 4-stage pipelined training, the hyperparameters are the same as the non-pipelined baseline, except for the BKS

Revised: [Table]

Revised: rate.

Revised: Table 7 shows that learning rate for all ResNet experimented.

Revised: [Table]