Original: S

Original: PARSE P ERSISTENT RNN S :

Original: S

Original: QUEEZING

Original: L

Original:  ARGE R ECURRENT N ETWORKS O NC HIPAnonymous authors Paper under double-blind review
Revised: S PARSE P ERSISTENT RNN S : S QUEEZING L ARGE R ECURRENT N ETWORKS O NC HIP

Original: A BSTRACT
Revised: A BSTRACT

Original: Recurrent Neural Networks (RNNs) are powerful tools for solving sequence-based problems, but their efﬁcacy and execution time are dependent on the size of the network.
Revised: Recurrent Neural Networks (RNNs) are powerful tools for solving sequence-based problems, but their efﬁcacy and execution time are dependent on the size of the network.

Original: Following recent work in simplifying these networks with model pruning and a novel mapping of work onto GPUs, we design an efﬁcient implementation for sparse RNNs.
Revised: Following recent work in simplifying these networks with model pruning and a novel mapping of work onto GPUs, we design an efﬁcient implementation for sparse RNNs.

Original: We investigate several optimizations and tradeoffs: Lamport timestamps, wide memory loads and a bank-aware weight layout.
Revised: We investigate several optimizations and tradeoffs: Lamport timestamps, wide memory loads, and a bank-aware weight layout.

Original: With these optimizations, we achieve speedups of over 6 × over the next best algorithm for a hidden layer of size 2304, batch size of 4, and a density of 30%.
Revised: With these optimizations, we achieve speedups of over 6 × over the next best algorithm for a hidden layer of size 2304, batch size of 4, and a density of 30%.

Original: Further, our technique allows for models of over 5 × the size to ﬁt on a GPU for a speedup of 2 × , enabling larger networks to help advance the state-of-the-art.
Revised: Further, our technique allows for models of over 5 × the size to ﬁt on a GPU for a speedup of 2 × , enabling larger networks to help advance the state-of-the-art.

Original: We perform case studies on NMT and speech recognition tasks in the appendix, accelerating their recurrent layers by up to 3 × .
Revised: We perform case studies on NMT and speech recognition tasks in the appendix, accelerating their recurrent layers by up to 3 × .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many sequence-based problems, including Speech Recognition (Amodei et al., 2015) and Neural
Revised: Many sequence-based problems, including Speech Recognition (Amodei et al., 2015) and Neural

Original: Machine Translation (NMT) (Bahdanau et al., 2014), can be solved effectively with Recurrent
Revised: Machine Translation (NMT) (Bahdanau et al., 2014) can be solved effectively with Recurrent

Original: Neural Networks (RNNs).
Revised: Neural Networks (RNNs).

Original: Appleyard et al.
Revised: Appleyard et al.

Original: (2016) showed that these networks can run efﬁciently on massively parallel processors such as GPUs, and Diamos et al.
Revised: (2016) showed that these networks can run efﬁciently on massively parallel processors such as GPUs and Diamos et al.

Original: (2016) found that if the network is small enough to ﬁt in the register ﬁle of a GPU, a persistent approach can be used to increase performance.
Revised: (2016) found that if the network is small enough to ﬁt in the register ﬁle of a GPU, a persistent approach can be used to increase performance.

Original: In parallel, many network compression methods (Han et al., 2016c; 2015; Guo et al., 2016) have been shown to reduce the model size of both Convolutional Neural Networks (CNNs) and RNNs.
Revised: In parallel many network compression methods (Han et al., 2016c; 2015; Guo et al., 2016) have been shown to reduce the model size of both Convolutional Neural Networks (CNNs) and RNNs.

Original: Recent work in this area has found that model pruning, in particular, can lead to signiﬁcant reductions in the number of important network parameters for RNNs (Narang et al., 2017; See et al., 2016)
Revised: Recent work in this area has found that model pruning, in particular, can lead to signiﬁcant reductions in the number of important network parameters for RNNs (Narang et al., 2017; See et al., 2016)

Original: We present an approach that combines both of these techniques into an efﬁcient and expandable approach.
Revised: We present an approach that combines both of these techniques into an efﬁcient and expandable approach.

Original: In particular, our work makes the following contributions:
Revised: In particular, our work makes the following contributions:

Original: • Larger sparse RNNs can be run more efﬁciently on GPUs.
Revised: • Larger sparse RNNs can be run more efﬁciently on GPUs.

Original: • Sparse RNNs can be run with smaller batch sizes more efﬁciently on GPUs.
Revised: • Sparse RNNs can be run with smaller batch sizes more efﬁciently on GPUs.

Original: • Various optimizations on top of the naïve implementation, necessary to achieve high performance.
Revised: • Various optimizations on top of the naïve implementation, necessary to achieve high performance.

Original: • Case studies using our technique showing 1) generalization to LSTMs, 2) practical network design considerations, and 3) that our technique is necessary to exploit sparsity in a meaningful way for some recurrent networks.
Revised: • Case studies using our technique showing 1) generalization to LSTMs, 2) practical network design considerations, and 3) speedups of up to 3 × on two non-synthetic workloads.

Original: A naïve implementation of the idea, presented in Section 3, leads to limited beneﬁt; we present a series of optimizations in Section 4 that help to achieve a high level of performance.
Revised: A naïve implementation of the idea, presented in Section 3, leads to limited beneﬁt; we present a series of optimizations in Section 4 that help to achieve a high level of performance.

Original: Sectiondescribes the experimental setup and results, and we discuss future work and our conclusions in
Revised: Sectiondescribes the experimental setup and results, and we discuss future work and our conclusions in

Original: Sections 6 and 7.
Revised: Sections 6 and 7.

Original: The appendix presents a case study on a machine translation task.
Revised: The appendix presents a case study on a machine translation task.

Original:  R ELATED W ORK 2. RNN S
Revised:  R ELATED W ORK 2. RNN S

Original: Recurrent Neural Networks (RNNs) are powerful tools for solving series-based problems, such as
Revised: Recurrent Neural Networks (RNNs) are powerful tools for solving series-based problems, such as

Original: NMT (Bahdanau et al., 2014; Hannun et al., 2014; Amodei et al., 2015; Luong et al., 2015), language modeling (Mikolov et al., 2010), and various NLP tasks (Collobert et al., 2011).
Revised: NMT (Bahdanau et al., 2014; Hannun et al., 2014; Amodei et al., 2015; Luong et al., 2015), language modeling (Mikolov et al., 2010), and various NLP tasks (Collobert et al., 2011).

Original: More complex recurrent networks have been devised to build on the basic RNN structure, such as Long/Short Term
Revised: More complex recurrent networks have been devised to build on the basic RNN structure, such as Long/Short Term

Original: Memory networks (LSTMs) (Hochreiter & Schmidhuber, 1997) and Gated Recurrent Units (GRUs) (Chung et al., 2014).
Revised: Memory networks (LSTMs) (Hochreiter & Schmidhuber, 1997) and Gated Recurrent Units (GRUs) (Chung et al., 2014).

Original: Though we focus on RNNs in this work for simplicity, our approach can extend to other recurrent network types, such as the LSTMs used in our case study.
Revised: Though we focus on RNNs in this work for simplicity, our approach can extend to other recurrent network types, such as the LSTMs used in our case study.

Original: In particular, we build on a recent GPU-based method storing recurrent weights on-chip (Diamos et al., 2016).
Revised: In particular, we build on a recent GPU-based method storing recurrent weights on-chip (Diamos et al., 2016).

Original: 2. M ODEL S IMPLIFICATION
Revised: 2. M ODEL S IMPLIFICATION

Original: Since neural networks are tasked with solving incredibly complex problems, they can become incredibly complex models.
Revised: Since neural networks are tasked with solving incredibly complex problems, they can become incredibly complex models.

Original: So, considerable effort is spent simplifying network models so they run more efﬁciently and can be deployed on smaller hardware.
Revised: So, considerable effort is spent simplifying network models so they run more efﬁciently and can be deployed on smaller hardware.

Original: In this work, we focus on pruning, though other orthogonal simpliﬁcation techniques (such as quantization (Gong et al., 2014), weight sharing (Chen et al., 2015), and tensor approximations (Cai et al., 2014)) are also possible.
Revised: In this work, we focus on pruning, though other orthogonal simpliﬁcation techniques (such as quantization (Gong et al., 2014), weight sharing (Chen et al., 2015), and tensor approximations (Cai et al., 2014)) are also possible.

Original: Network pruning is the process of inducing sparsity in the weights of the network model (LeCun et al., 1990; Hassibi et al., 1993) and can be used as a regularizer (Thodberg, 1991; Giles & Omlin, 1994; Han et al., 2017), a method of compression (Han et al., 2015; 2016c), or a way to reduce the computational workload (Han et al., 2016b).
Revised: Network pruning is the process of inducing sparsity in the weights of the network model (LeCun et al., 1990; Hassibi et al., 1993) and can be used as a regularizer (Thodberg, 1991; Giles & Omlin, 1994; Han et al., 2017), a method of compression (Han et al., 2015; 2016c), or a way to reduce the computational workload (Han et al., 2016b).

Original: Recent results show that this technique is applicable to recurrent networks used in a variety of tasks, from speech recognition (Han et al., 2016a;
Revised: Recent results show that this technique is applicable to recurrent networks used in a variety of tasks, from speech recognition (Han et al., 2016a;

Original: Narang et al., 2017) to machine translation (See et al., 2016) and image captioning (Han et al., 2016b).
Revised: Narang et al., 2017) to machine translation (See et al., 2016) and image captioning (Han et al., 2016b).

Original: Different pruning techniques include ﬁne-grained, unstructured pruning (Han et al., 2015); regular, structured pruning at a small scale (Anwar et al., 

Revised: Different pruning techniques include ﬁne-grained, unstructured pruning (Han et al., 2015); regular, structured pruning at a small scale (Anwar et al.,

Original: ; Li et al., 2016; Wen et al., 2016); and pruning entire ﬁlters that results in a dense workload with smaller dimensions (Molchanov et al., 2017).
Revised: ; Li et al., 2016; Wen et al., 2016); and pruning entire ﬁlters that results in a dense workload with smaller dimensions (Molchanov et al., 2017).

Original: In this work, we assume the sparsity is unstructured so as to remain useful in the general case; there are some simpliﬁcations that could be made if structure is guaranteed.
Revised: In this work, we assume the sparsity is unstructured so as to remain useful in the general case; there are some simpliﬁcations that could be made if structure is guaranteed.

Original:  I MPLEMENTATION D ETAILS
Revised:  I MPLEMENTATION D ETAILS

Original: We draw heavily from Diamos et al.
Revised: We draw heavily from Diamos et al.

Original: (2016), in which the authors use the large on-chip storage of
Revised: (2016), in which the authors use the large on-chip storage of

Original: GPUs to hold the recurrent weights.
Revised: GPUs to hold the recurrent weights.

Original: This approach is brieﬂy discussed here, followed by details of how our sparse technique differs.
Revised: This approach is brieﬂy discussed here, followed by details of how our sparse technique differs.

Original: 3. RNN O PERATION
Revised: 3. RNN O PERATION

Original: A recurrent network’s operation is conceptually simple, and each time step can be expressed by
Revised: A recurrent network’s operation is conceptually simple, and each time step can be expressed by

Original: Equation 1:
Revised: Equation 1:

Original: [Equation]
Revised: [Equation]

Original: where U r is the recurrent weight matrix, W is the input-to-hidden weight matrix, b is a bias term, and g is an elementwise activation function.
Revised: where U r is the recurrent weight matrix, W is the input-to-hidden weight matrix, b is a bias term, and g is an elementwise activation function.

Original: The input-to-hidden weight matrix ( Wx t ) calculation has no dependency, so it can be processed in parallel and added to b , becoming b (cid:48) .
Revised: The input-to-hidden weight matrix ( Wx t ) calculation has no dependency, so it can be processed in parallel and added to b , becoming b (cid:48) .

Original: The formula simpliﬁes to
Revised: The formula simpliﬁes to

Original: Equation 2:
Revised: Equation 2:

Original: [Equation]
Revised: [Equation]

Original: 3. P ERSISTENT RNN S
Revised: 3. P ERSISTENT RNN S

Original: In a persistent RNN implementation, weights U r are stored in on-chip register ﬁles so each thread keeps x × y ( rows × columns ) weights, while the activations ( h t − 1 ) are stored in shared memory.
Revised: In a persistent RNN implementation, weights U r are stored in on-chip register ﬁles so each thread keeps x × y ( rows × columns ) weights, while the activations ( h t − 1 ) are stored in shared memory.

Original: Each row is processed by one warp.
Revised: Each row is processed by one warp.

Original: The number of thread blocks is set to the number of Streaming
Revised: The number of thread blocks is set to the number of Streaming

Original: Multiprocessors (SMs) in the system, and blocks work together processing all rows to perform each matrix multiplication from Equation 2, shown in Figure 1.
Revised: Multiprocessors (SMs) in the system, and blocks work together processing all rows to perform each matrix multiplication from Equation 2, shown in Figure 1.

Original: There are 4 stages in a persistent RNN’s software pipeline: load, operate, reduce, and synchronize.
Revised: There are 4 stages in a persistent RNN’s software pipeline: load, operate, reduce, and synchronize.

Original: Load: the input activations in the current timestep are the output from previous timestep.
Revised: Load: the input activations in the current timestep are the output from previous timestep.

Original: All threads within a block cooperate to load these activations from global memory to shared memory for thread-level data re-use.
Revised: All threads within a block cooperate to load these activations from global memory to shared memory for thread-level data re-use.

Original: Operate: each thread holds x × y weights and executes acc
Revised: Operate: each thread holds x × y weights and executes acc

Original: [ i ]+ = weight
Revised: [ i ]+ = weight

Original: [ i ][ j ] ∗ activation _ shm
Revised: [ i ][ j ] ∗ activation _ shm

Original: [ j ] computations x × y times to compute x accumulations.
Revised: [ j ] computations x × y times to compute x accumulations.

Original: In particular, acc [ x ] and weight [ x ][ y ] are stored in registers, and activation _ shm
Revised: In particular, acc [ x ] and weight [ x ][ y ] are stored in registers, and activation _ shm

Original: [ hiddensize ] are stored in shared memory.
Revised: [ hiddensize ] are stored in shared memory.

Original: Reduce: after each thread in a warp ﬁnishes accumulating, threads belonging to the same row perform a ﬁnal accumulation through shared memory and output the ﬁnal result to global memory.
Revised: Reduce: after each thread in a warp ﬁnishes accumulating, threads belonging to the same row perform a ﬁnal accumulation through shared memory and output the ﬁnal result to global memory.

Original: Synchronize: all threads in different blocks are synchronized by a global barrier.
Revised: Synchronize: all threads in different blocks are synchronized by a global barrier.

Original: Overall performance is largely dominated by math throughput in the operate stage.
Revised: Overall performance is largely dominated by math throughput in the operate stage.

Original: Since the addresses of 32 activations loaded by a warp are contiguous, there is no divergence in the shared memory load.
Revised: Since the addresses of 32 activations loaded by a warp are contiguous, there is no divergence in the shared memory load.

Original: Similarly, in one thread, weights in different rows of the same column are used by a single activation, so one thread can re-use this activation for many rows to amortize the shared memory load cost.
Revised: Similarly, in one thread, weights in different rows of the same column are used by a single activation, so one thread can re-use this activation for many rows to amortize the shared memory load cost.

Original: To meet FMA and shared memory throughput, one element loaded from shared memory should do at least 4 FMA operations.
Revised: To meet FMA and shared memory throughput, one element loaded from shared memory should do at least 4 FMA operations.

Original: So the best practice is that each thread holds more than 4 rows’ weights for one column, which means one activation is re-used more than 4 times by current thread.
Revised: So the best practice is that each thread holds more than 4 rows’ weights for one column, which means one activation is re-used more than 4 times by current thread.

Original: In this way, the math units are fully utilized and not starved by memory.
Revised: In this way, the math units are fully utilized and not starved by memory.

Original: 3. S PARSE P ERSISTENT RNN S
Revised: 3. S PARSE P ERSISTENT RNN S

Original: We now describe our novel approach to support sparse, pruned RNNs with persistent weights.
Revised: We now describe our novel approach to support sparse, pruned RNNs with persistent weights.

Original: In contrast to dense persistent RNNs, the data format of sparse persistent RNNs are <index, value> pairs.
Revised: In contrast to dense persistent RNNs, the data format of sparse persistent RNNs are <index, value> pairs.

Original: One <index, value> pair represents the location and value of one nonzero weight, and each thread keeps a fraction of the total nonzero weights.
Revised: One <index, value> pair represents the location and value of one nonzero weight, and each thread keeps a fraction of the total nonzero weights.

Original: Additionally, in sparse persistent RNNs, all of the nonzero elements in one thread must belong to the same row, and the number of thread blocks depends on the hidden layer size and degree of sparsity, rather than being ﬁxed to the number of
Revised: Additionally, in sparse persistent RNNs, all of the nonzero elements in one thread must belong to the same row, and the number of thread blocks depends on the hidden layer size and degree of sparsity, rather than being ﬁxed to the number of

Original: SMs available on the chip, as in the dense case.
Revised: SMs available on the chip, as in the dense case.

Original: As before, the RNN’s weights are shared over all timesteps, so <index, value> pairs are stored in on-chip storage to avoid re-loading weights each timestep.
Revised: As before, the RNN’s weights are shared over all timesteps, so <index, value> pairs are stored in on-chip storage to avoid re-loading weights each timestep.

Original: Finally, there are again 4 stages in a sparse persistent RNN’s pipeline:
Revised: Finally, there are again 4 stages in a sparse persistent RNN’s pipeline:

Original: Load: same as the load stage of
Revised: Load: same as the load stage of

Original: dense
Revised: dense

Original: persistent RNNs
Revised: persistent RNNs.

Original: Operate: each thread holds n nonzero weights ( n <index, value> pairs) and executes acc + = value [ i ] ∗ activation _ in _ shm [ index [ i ]] computations n times in series.
Revised: Operate: each thread holds n nonzero weights ( n <index, value> pairs) and executes acc + = value [ i ] ∗ activation _ in _ shm [ index [ i ]] computations n times in series.

Original: Reduce: within each block, after each thread ﬁnishes operating and stores its accumulation into shared memory, several threads work together to generate one result for a row.
Revised: Reduce: within each block, after each thread ﬁnishes operating and stores its accumulation into shared memory, several threads work together to generate one result for a row.

Original: All the blocks work together to process all rows.
Revised: All the blocks work together to process all rows.

Original: Synchronize: same as the synchronize stage of dense persistent RNNs.
Revised: Synchronize: same as the synchronize stage of dense persistent RNNs.

Original: Overall performance is limited by shared memory throughput.
Revised: Overall performance is limited by shared memory throughput.

Original: Since the layout of the densly-packed sparse matrix doesn’t match that of the original dense matrix anymore, the nonzeros in different rows of the same storage column (as opposed to logical or layout column) require different activations.
Revised: Since the layout of the densly-packed sparse matrix doesn’t match that of the original dense matrix anymore, the nonzeros in different rows of the same storage column (as opposed to logical or layout column) require different activations.

Original: This is decided by each weight’s index, which means each thread cannot be guaranteed to re-use activations for nonzero weights, even though these nonzero weights may be located in the same column.
Revised: This is decided by each weight’s index, which means each thread cannot be guaranteed to re-use activations for nonzero weights, even though these nonzero weights may be located in the same column.

Original: Futher, the 32 unique indices in a warp may point to 32 different shared memory locations, which can lead to a maximum of 32 bank conﬂicts in shared memory.
Revised: Futher, the 32 unique indices in a warp may point to 32 different shared memory locations, which can lead to a maximum of 32 bank conﬂicts in shared memory.

Original: So, the performance of the operate stage is limited by shared memory rather than math unit, and shared memory bank conﬂicts become the main challenge for a sparse persistent RNN’s performance.
Revised: So, the performance of the operate stage is limited by shared memory rather than math unit, and shared memory bank conﬂicts become the main challenge for a sparse persistent RNN’s performance.

Original:  O PTIMIZATIONS
Revised:  O PTIMIZATIONS

Original: In this section, we identify and provide solutions for several shortcomings of the previously-described implementation.
Revised: In this section, we identify and provide solutions for several shortcomings of the previously-described implementation.

Original: The beneﬁts of these optimizations are shown in Table 1.
Revised: The beneﬁts of these optimizations are shown in Table 1.

Original: First, we note that straightforwardly pruning a network does not consider the distribution of the induced zeros.
Revised: First, we note that straightforwardly pruning a network does not consider the distribution of the induced zeros.

Original: The number of nonzero weights in different rows can vary, and these arbitrary distributions cause divergence and load imbalance.
Revised: The number of nonzero weights in different rows can vary, and these arbitrary distributions cause divergence and load imbalance.

Original: So, a baseline implementation must pad rows with fewer than the maximum number of nonzero weights with <index, 0> pairs, forcing all rows to have the same number of weights and, implicitly, the same workload.
Revised: So, a baseline implementation must pad rows with fewer than the maximum number of nonzero weights with <index, 0> pairs, forcing all rows to have the same number of weights and, implicitly, the same workload.

Original: This technique wastes roughly 20% of the useful registers on unneeded zeros, but it still gives a performance improvement.
Revised: This technique wastes roughly 20% of the useful registers on unneeded zeros, but it still gives a performance improvement.

Original: The behavior of the network does not change since this re-introduces pruned weights with a value of 0.
Revised: The behavior of the network does not change since this re-introduces pruned weights with a value of 0.

Original: 4. B ANK C ONFLICTS
Revised: 4. B ANK C ONFLICTS

Original: To achieve high bandwidth, shared memory is divided into 32 banks that can each service one request per cycle.
Revised: To achieve high bandwidth, shared memory is divided into 32 banks that can each service one request per cycle.

Original: With perfect request alignment, shared memory can achieve 32 requests per cycle, but multiple column indices landing in the same bank cause conﬂicts.
Revised: With perfect request alignment, shared memory can achieve 32 requests per cycle, but multiple column indices landing in the same bank cause conﬂicts.

Original: These bank conﬂicts are serialized and are the main bottleneck of our sparse persistent RNN implementation, since in the operate stage, the 32 indices of a warp may access 32 different shared memory locations, leading to up to 32 bank conﬂicts.
Revised: These bank conﬂicts are serialized and are the main bottleneck of our sparse persistent RNN implementation, since in the operate stage, the 32 indices of a warp may access 32 different shared memory locations, leading to up to 32 bank conﬂicts.

Original: In this section, we introduce two methods to mitigate this inefﬁciency: wide memory loads and bank-aware weight layout.
Revised: In this section, we introduce two methods to mitigate this inefﬁciency: wide memory loads and bank-aware weight layout.

Original: After applying these two optimizations, the total shared memory bank conﬂicts can be reduced by more than 80%.
Revised: After applying these two optimizations, the total shared memory bank conﬂicts can be reduced by more than 80%.

Original: Wide Memory Loads:
Revised: Wide Memory Loads:

Original: In a dense persistent RNN, one activation can be reused by many weights in the same column.
Revised: In a dense persistent RNN, one activation can be reused by many weights in the same column.

Original: Weight reuse across samples can occur if the input to the network is a minibatch of size larger than one.
Revised: Weight reuse across samples can occur if the input to the network is a minibatch of size larger than one.

Original: Similarly, in a sparse persistent RNN, we can batch 4 activations from different samples together and process them at a same time(minibatch size = 4).
Revised: Similarly, in a sparse persistent RNN, we can batch 4 activations from different samples together and process them at a same time(minibatch size = 4).

Original: These 4 activations can be loaded from shared memory at once by one ld.shared.v4 instruction (NVIDIA, 2017).
Revised: These 4 activations can be loaded from shared memory at once by one ld.shared.v4 instruction (NVIDIA, 2017).

Original: As the addresses of 4 activations are contiguous, they belong to 4 different shared memory banks.
Revised: As the addresses of 4 activations are contiguous, they belong to 4 different shared memory banks.

Original: So, the total bank conﬂicts can be reduced to at most 1 / 4 .
Revised: So, the total bank conﬂicts can be reduced to at most 1 / 4 .

Original: As activations are stored in shared memory, the side-effect of using this wide memory load is that it consumes 4 × the shared memory.
Revised: As activations are stored in shared memory, the side-effect of using this wide memory load is that it consumes 4 × the shared memory.

Original: Since max hidden layer size is limited by the shared memory size, 4 × shared memory usage means that the maximum hidden layer size is reduced to 1 / 4 .
Revised: Since max hidden layer size is limited by the shared memory size, 4 × shared memory usage means that the maximum hidden layer size is reduced to 1 / 4 .

Original: So for large hidden layers, we can use ld.shared.v instead.
Revised: So for large hidden layers, we can use ld.shared.v instead.

Original: This only guarantees a bank conﬂict reduction of 2 × , but it only incurs a 2 × storage overhead.
Revised: This only guarantees a bank conﬂict reduction of 2 × , but it only incurs a 2 × storage overhead.

Original: This technique introduces a trade-off between efﬁciency and maximum supported hidden layer size.
Revised: This technique introduces a trade-off between efﬁciency and maximum supported hidden layer size.

Original: Bank-Aware Weight Layout:
Revised: Bank-Aware Weight Layout:

Original: Each thread holds n <index, value> pairs.
Revised: Each thread holds n <index, value> pairs.

Original: In the operate stage, each thread executes acc + = value [ i ] ∗ activation _ in _ shm [ index [ i ]] operations one by one.
Revised: In the operate stage, each thread executes acc + = value [ i ] ∗ activation _ in _ shm [ index [ i ]] operations one by one.

Original: Since both the activations and weight value use the same index, i , reordering nonzeros’ locations for each row does not affect the ﬁnal result, but it can change the access sequence to shared memory.
Revised: Since both the activations and weight value use the same index, i , reordering nonzeros’ locations for each row does not affect the ﬁnal result, but it can change the access sequence to shared memory.

Original: We can construct a better shared memory access sequence to reduce bank conﬂicts.
Revised: We can construct a better shared memory access sequence to reduce bank conﬂicts.

Original: Performing this reordering only has to happen when the network’s sparsity pattern changes.
Revised: Performing this reordering only has to happen when the network’s sparsity pattern changes.

Original: For inference, this is exactly once, and its beneﬁt can be used for the whole lifetime of the network.
Revised: For inference, this is exactly once, and its beneﬁt can be used for the whole lifetime of the network.

Original: Or, if training a sparse network with modiﬁcations to the pruned weight distribution over time (Guo et al., 2016; Narang et al., 2017; Anonymous, 2018), this cost can be amortized over all timesteps in the network for however many training iterations the sparsity pattern is constant.
Revised: Or, if training a sparse network with modiﬁcations to the pruned weight distribution over time (Guo et al., 2016; Narang et al., 2017; Zhu & Gupta, 2017), this cost can be amortized over all timesteps in the network for however many training iterations the sparsity pattern is constant.

Original: Listing 1  shows a greedy algorithm to generate a weight layout that reduces the amount of shared memory bank conﬂicts.Invoked for independently do do
Revised: Listing 1 in Appendix A shows a greedy algorithm to generate a weight layout that reduces the amount of shared memory bank conﬂicts.

Original: Use a unfilled Color end location end

Original: 4. S YNCHRONIZATION
Revised: 4. S YNCHRONIZATION

Original: We considered two options for ensuring correct ordering between work from different thread blocks: global synchronization and Lamport Timestamps (Lamport, 1974).
Revised: We considered two options for ensuring correct ordering between work from different thread blocks: global synchronization and Lamport Timestamps (Lamport, 1974).

Original: Lamport timestamps add a ﬂag for each output value used to indicate whether or not the value has been computed.
Revised: Lamport timestamps add a ﬂag for each output value used to indicate whether or not the value has been computed.

Original: So, the load stage can make partial progress as values are marked complete, allowing the operate stage to proceed before all values are ﬁnished updating from the previous time step, as is required by global barriers.
Revised: So, the load stage can make partial progress as values are marked complete, allowing the operate stage to proceed before all values are ﬁnished updating from the previous time step, as is required by global barriers.

Original: A side effect of using Lamport timestamps is that it allows a software pipeline to be used over time steps: as we process iteration n , we can load the states for iteration n + 1 .
Revised: A side effect of using Lamport timestamps is that it allows a software pipeline to be used over time steps: as we process iteration n , we can load the states for iteration n + 1 .

Original: Later, in iteration n + 1 , we need to make sure the ﬂags loaded by this prefetch indicate valid data; this can sometimes require another load for some values.
Revised: Later, in iteration n + 1 , we need to make sure the ﬂags loaded by this prefetch indicate valid data; this can sometimes require another load for some values.

Original: A basic implementation of Lamport timestamps uses one extra ﬂag for every output value.
Revised: A basic implementation of Lamport timestamps uses one extra ﬂag for every output value.

Original: This, coupled with the software pipeline, quadruples the used buffer size: 2 × the data for the ﬂags, and another 2 × for the two concurrent pipeline stages.
Revised: This, coupled with the software pipeline, quadruples the used buffer size: 2 × the data for the ﬂags, and another 2 × for the two concurrent pipeline stages.

Original: Our optimized version removes the ﬂags by initializing the output buffers for each time step to -0.0f.
Revised: Our optimized version removes the ﬂags by initializing the output buffers for each time step to -0.0f.

Original: Once each element is updated, it is implicitly valid by virtue of having a non negative zero value.
Revised: Once each element is updated, it is implicitly valid by virtue of having a non negative zero value.

Original: (We convert all -0.0f result values to +0.0f to avoid aliasing.)
Revised: (We convert all -0.0f result values to +0.0f to avoid aliasing.)

Original: So, our ﬁnal implementation only doubles the memory requirements.
Revised: So, our ﬁnal implementation only doubles the memory requirements.

Original: Each method has its own advantages and disadvantages.
Revised: Each method has its own advantages and disadvantages.

Original: For example, a global synchronization requires several memory round trips to implement, but only needs to be called once per timestep.
Revised: For example, a global synchronization requires several memory round trips to implement, but only needs to be called once per timestep.

Original: Lamport timestamps ideally require no extra memory movement, but do require multiple checks to ensure the loaded values are current.
Revised: Lamport timestamps ideally require no extra memory movement, but do require multiple checks to ensure the loaded values are current.

Original: Also, to overlap the load and operate stages by preloading activations requires double-buffering, so the shared memory usage is doubled and maximum effective layer size is halved.
Revised: Also, to overlap the load and operate stages by preloading activations requires double-buffering, so the shared memory usage is doubled and maximum effective layer size is halved.

Original: Thus, we expose another tradeoff: larger layer sizes can be accommodated with global barriers due to the reduced shared memory usage.
Revised: Thus, we expose another tradeoff: larger layer sizes can be accommodated with global barriers due to the reduced shared memory usage.

Original: We found Lamport timestamps to be faster in our experiments, except for very large layer sizes ( > 5760).
Revised: We found Lamport timestamps to be faster in our experiments, except for very large layer sizes ( > 5760).

Revised: 1: A naïve implementation has limited performance; our optimizations are necessary to achieve good results.

Revised: (Layer size = 1152, batch size = 4, density = 10%, #timesteps = 256.)

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we describe the setup and experiments performed to show the beneﬁts of our sparse persistent RNN technique.
Revised: In this section, we describe the setup and experiments performed to show the beneﬁts of our sparse persistent RNN technique.

Original: 5. W ORKLOADS
Revised: 5. W ORKLOADS

Original: Sparsity: Recent efforts in pruning recurrent networks result in varying degrees of sparsity.
Revised: Sparsity: Recent efforts in pruning recurrent networks result in varying degrees of sparsity.

Original: Han et al.
Revised: Han et al.

Original: (2016a) were able to prune 90% of the weights in a speech recognition LSTM.
Revised: (2016a) were able to prune 90% of the weights in a speech recognition LSTM.

Original: Similarly, the recurrent layers of NeuralTalk (Karpathy & Li, 2014), a caption-generating network, have been pruned to around 10% density (Han et al., 2016b) without loss of accuracy.
Revised: Similarly, the recurrent layers of NeuralTalk (Karpathy & Li, 2014), a caption-generating network, have been pruned to around 10% density (Han et al., 2016b) without loss of accuracy.

Original: A different way to use pruning is to start with a model that is “too large" for the target hardware and prune it down to size.
Revised: A different way to use pruning is to start with a model that is “too large" for the target hardware and prune it down to size.

Original: In this way, it is possible to produce a network with the same effective number of parameters (nonzero coefﬁcients) that represents the sparse version of a much larger network.
Revised: In this way, it is possible to produce a network with the same effective number of parameters (nonzero coefﬁcients) that represents the sparse version of a much larger network.

Original: This has been shown to be very fruitful for RNNs (Narang et al., 2017; Anonymous, 2018) with sparsities of up to 95% for RNNs and GRUs.
Revised: This has been shown to be very fruitful for RNNs (Narang et al., 2017; Zhu & Gupta, 2017) with sparsities of up to 95% for RNNs and GRUs.

Original: So, we will focus on sparsities around this common 90% target, from 80-99% sparse.
Revised: So, we will focus on sparsities around this common 90% target, from 80-99% sparse.

Original: Network size: Common sizes of today’s RNN hidden layers are 1024 to 3072.
Revised: Network size: Common sizes of today’s RNN hidden layers are 1024 to 3072.

Original: We also include larger layer sizes since 1) larger sparse networks have been shown to outperform smaller dense networks with similar capacities (Narang et al., 2017; Anonymous, 2018), and 2) network sizes in practice have historically increased as improvements in processing hardware made them feasible (Gray et al., 2017).
Revised: We also include larger layer sizes since 1) larger sparse networks have been shown to outperform smaller dense networks with similar capacities (Narang et al., 2017; Zhu & Gupta, 2017), and 2) network sizes in practice have historically increased as improvements in processing hardware made them feasible (Gray et al., 2017).

Original: Batch size: For deployment on edge devices, smaller batch sizes of 1 to 4 are common.
Revised: Batch size: For deployment on edge devices, smaller batch sizes of 1 to 4 are common.

Original: In data centers, inference batch sizes may be larger, and batches during training can be several hundred inputs large.
Revised: In data centers, inference batch sizes may be larger, and batches during training can be several hundred inputs large.

Original: We focus on the lower end, as pruning is most commonly used for deployment.
Revised: We focus on the lower end, as pruning is most commonly used for deployment.

Original: Timesteps: The number of timesteps used by the network depends on the use case.
Revised: Timesteps: The number of timesteps used by the network depends on the use case.

Original: Translation tasks can use on the order of 10 time steps (context around the word being translated), but speech recognition may use hundreds of timesteps (audio samples).
Revised: Translation tasks can use on the order of 10 time steps (context around the word being translated), but speech recognition may use hundreds of timesteps (audio samples).

Original: So, we explore a wide range of timesteps.
Revised: So, we explore a wide range of timesteps.

Original: 5. A LGORITHMS
Revised: 5. A LGORITHMS

Original: To show the beneﬁts of our sparse persistent approach, we compare against three other algorithms that can target a pruned RNN: dense GEMM (cuDNN 7.0.3), sparse GEMM (spMM in cuSparse 9.0) , and a dense persistent approach (cuDNN 7.0.3).
Revised: To show the beneﬁts of our sparse persistent approach, we compare against three other algorithms that can target a pruned RNN: dense GEMM (cuDNN 7.0.3), sparse GEMM (spMM in cuSparse 9.0), and a dense persistent approach (cuDNN 7.0.3).

Original: Note that the persistent approaches will not be applicable to all layer sizes due to resource constraints, with the dense persistent kernels suffering more quickly than our sparse approach.
Revised: Note that the persistent approaches will not be applicable to all layer sizes due to resource constraints, with the dense persistent kernels suffering more quickly than our sparse approach.

Original: Our sparse persistent code is compiled in CUDA 9.0, and all tests are run on a NVIDIA Tesla V100.
Revised: Our sparse persistent code is compiled in CUDA 9.0, and all tests are run on a NVIDIA Tesla V100.

Original: 5. R ESULTS
Revised: 5. R ESULTS

Revised: [Figure]

Original: We use a layer size of 1796, density of 10%, batch size of 4, and 256 time steps as a baseline for
Revised: We use a layer size of 1796, density of 10%, batch size of 4, and 256 time steps as a baseline for

Original: Figure 2.
Revised: Figure 2.

Original: Figure 2 (a) shows the effect of varying the density of the network.
Revised: Figure 2 (a) shows the effect of varying the density of the network.

Original: For a density of 10%, a dense persistent RNN achieves a 4 . 1 × speedup over dense GEMM, while a sparse persistent RNN can achieve a 7 . 3 × speedup over the same dense GEMM.
Revised: For a density of 10%, a dense persistent RNN achieves a 4 . 1 × speedup over dense GEMM, while a sparse persistent RNN can achieve a 7 . 3 × speedup over the same dense GEMM.

Original: When the density decreases to 1%, our approach achieves a 15 . 

Revised: When the density decreases to 1%, our approach achieves a 15 .

Original: × speedup over dense GEMM.
Revised: × speedup over dense GEMM.

Original: Figure 2 (b) shows the effect of layer size on performance.
Revised: Figure 2 (b) shows the effect of layer size on performance.

Original: Dense persistent RNNs fail after a layer size of 1792 due to insufﬁcient registers, and the same is true for sparse persistent RNNs and a layer size of 5632 at 10% density.
Revised: Dense persistent RNNs fail after a layer size of 1792 due to insufﬁcient registers, and the same is true for sparse persistent RNNs and a layer size of 5632 at 10% density.

Original: As the layer size grows, increased pressure on shared memory and the necessary switch to using global barriers for synchronization limit performance.
Revised: As the layer size grows, increased pressure on shared memory and the necessary switch to using global barriers for synchronization limit performance.

Original: Figure 2 (c) shows how the batch size affects performance.
Revised: Figure 2 (c) shows how the batch size affects performance.

Original: Dense GEMM cannot fully utilize the
Revised: Dense GEMM cannot fully utilize the

Original: GPU with a small batch size, so the other algorithms all outperform this simple approach.
Revised: GPU with a small batch size, so the other algorithms all outperform this simple approach.

Original: Our technique scales well to larger batch sizes, as opposed to dense persistent kernels.
Revised: Our technique scales well to larger batch sizes, as opposed to dense persistent kernels.

Original: Figure 2 (d) shows that the number of timesteps does not signiﬁcantly affect relative performance.
Revised: Figure 2 (d) shows that the number of timesteps does not signiﬁcantly affect relative performance.

Original: 1% 5% 10% 20% 30% (a) Density

Original: T i m e ( m s ) 2.1x 1.9x 1.8x 1.6x 1.6x     5632 (b) Layer Size

Original: T i m e ( m s ) 1.7x 1.8x 4.3x 3.5x 1.9x 0.5x

Original: [Figure]

Original: T i m e ( m s ) 1.8x 1.1x 1.8x 3.6x 3.7x 2.7x 1.8x

Original: [Figure]

Original: [Figure]

Original: T i

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: 2.1x 1.8x 1.8x

Original: [Figure]

Original: Persistent

Original: Sparse

Original: [Figure] Sparse Persistent ( [Figure] A larger network with the same number of nonzero parameters is expected to outperform the smaller, dense network (Narang et al., 2017; Anonymous, 2018; Gray et al., 2017).
Revised: A larger network with the same number of nonzero parameters is expected to outperform the smaller, dense network (Narang et al., 2017; Zhu & Gupta, 2017; Gray et al., 2017).

Original: So, we perform two experiments to show how our sparse persistent RNN technique can take advantage of this observation:
Revised: So, we perform two experiments to show how our sparse persistent RNN technique can take advantage of this observation:

Original: First, we vary the density of two large layer sizes, showing very large models resident on the GPU.
Revised: First, we vary the density of two large layer sizes, showing very large models resident on the GPU.

Original: In Figure 3 (a), we show a layer size of 2304 and vary the density.
Revised: In Figure 3 (a), we show a layer size of 2304 and vary the density.

Original: For a density of 10%, our sparse persistent RNN achieves a 9 .
Revised: For a density of 10%, our sparse persistent RNN achieves a 9 .

Original: 9 × speedup over dense GEMM.
Revised: 9 × speedup over dense GEMM.

Original: Even at higher densities, we can improve performance by more than a factor of the sparsity by allowing a persistent approach to be used.
Revised: Even at higher densities, we can improve performance by more than a factor of the sparsity by allowing a persistent approach to be used.

Original: Pushing the model size further, Figure 3 (b) shows our sparse persistent RNN approach allowing for an extremely large network with a high sparsity.
Revised: Pushing the model size further, Figure 3 (b) shows our sparse persistent RNN approach allowing for an extremely large network with a high sparsity.

Original: For a density of 1% and a layer size of 5760, our sparse persistent RNN can achieve a 10 .
Revised: For a density of 1% and a layer size of 5760, our sparse persistent RNN can achieve a 10 .

Original: 6 × speedup over dense GEMM.
Revised: 6 × speedup over dense GEMM.

Original: At 5% density, though, the overhead of the loading phase outweighs beneﬁts of a persistent approach, and a sparse
Revised: At 5% density, though, the overhead of the loading phase outweighs beneﬁts of a persistent approach, and a sparse

Original: GEMM becomes faster.
Revised: GEMM becomes faster.

Original: Second, we ﬁx the number of nonzero parameters present in the layer (1.32 million) and increase the layer size from 2304 (25% dense) to 11520 (1% dense) in Figure 4.
Revised: Second, we ﬁx the number of nonzero parameters present in the layer (1.32 million) and increase the layer size from 2304 (25% dense) to 11520 (1% dense) in Figure 4.

Original: As in prior work that explored this change in workload (Narang et al., 2017; Gray et al., 2017), a sparse GEMM can outperform a dense GEMM.
Revised: As in prior work that explored this change in workload (Narang et al., 2017; Gray et al., 2017), a sparse GEMM can outperform a dense GEMM.

Original: Our sparse persistent approach can improve performance even further.
Revised: Our sparse persistent approach can improve performance even further.

Original: [Figure]
Revised: [Figure]

Original: Performance of Large Sparse RNNs

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original:  D ISCUSSION
Revised: 5. D ISCUSSION

Original: Sparse persistent RNNs increase the performance of pruned networks, but without a series of optimizations we described in Section 4, the improvement is much lower.
Revised: Sparse persistent RNNs increase the performance of pruned networks, but without a series of optimizations we described in Section 4, the improvement is much lower.

Original: An optimized implementation is critical to achieve peak performance.
Revised: An optimized implementation is critical to achieve peak performance.

Original: After these optimizations, we see that our approach works best for sparser workloads, and, for most layer sizes, 10% density is sufﬁcient to beat all other approaches, though for smaller layer sizes (up to 2304), our approach is the winner  up to a density of 20%.
Revised: After these optimizations, we see that our approach works best for sparser workloads [Figure] and, for most layer sizes, 10% density is sufﬁcient to beat all other approaches, though for smaller layer sizes (up to 2304), our approach is the winner even up to a density of 30%.

Original: Different batch sizes can affect the efﬁciency of all techniques; in general, dense GEMMs and our approach scale the best, which tempers the beneﬁt offered by other algorithms.
Revised: Different batch sizes can affect the efﬁciency of all techniques; in general, dense GEMMs and our approach scale the best, which tempers the beneﬁt offered by other algorithms.

Original: Finally, the number of timesteps is largely immaterial to the relative performance of the various techniques.
Revised: Finally, the number of timesteps is largely immaterial to the relative performance of the various techniques

Revised: [Figure]

Original: We have described an efﬁcient algorithm for recurrent layers.
Revised: We have described an efﬁcient algorithm for recurrent layers.

Revised: To exploit this algorithm, one must ﬁrst prune the recurrent layers of a target network.

Revised: We perform this pruning during the training process for a nureal machine translation network (Klein et al., 2017), and we look to past work (Narang et al., 2017) that has pruned a commercial speech recognition network, Deep Speech 2 (Amodei et al., 2015).

Original: Concretely: for a given accuracy, this leads to a speedup of between 2 . 

Revised: The results on these real workloads are promising: for a given accuracy, we show a speedup of between 2 .

Original: × to 3 .
Revised: × to 3 .

Original: 1 × in a machine translation network’s recurrent layers pruned with load-balancing in mind, and a speedup of 1 . 2 × to 2 .
Revised: 1 × in the machine translation network’s recurrent layers pruned with load-balancing in mind, and a speedup of 1 . 2 × to 2 .

Original: 9 on a speech recognition network’s recurrent layers pruned without load-balancing.
Revised: 9 on Deep Speech 2’s recurrent layers pruned without load-balancing.

Original: Please refer to our appendix for more details.
Revised: Please refer to Appendix B for more details about the training processes and a full discussion of the results.

Original:  F UTURE W ORK
Revised:  F UTURE W ORK

Original: Currently, we use two 32-bit registers to store a <column_index, value> pair.
Revised: Currently, we use two 32-bit registers to store a <column_index, value> pair.

Original: However, most layers are not large enough to require more than 16 bits of index data, so two column indices can be compressed into a 32-bit register, freeing up registers for more nonzeros.
Revised: However, most layers are not large enough to require more than 16 bits of index data, so two column indices can be compressed into a 32-bit register, freeing up registers for more nonzeros.

Original: Similarly, we could use a lower-precision data type such as fp16 for the weights themselves to achieve the same result.
Revised: Similarly, we could use a lower-precision data type such as fp16 for the weights themselves to achieve the same result.

Original: The current maximum layer size for our approach on a V100 GPU is 11520; this limit is imposed by the 96KB shared memory usage per block.
Revised: The current maximum layer size for our approach on a V100 GPU is 11520, shown in Figure 4; this limit is imposed by the 96KB shared memory usage per block.

Original: We can swap Lamport timestamps for a global barrier to halve shared memory use and double the maximum layer size, shown in Figure 4.
Revised: To support this layer size, we must swap Lamport timestamps for a global barrier to halve shared memory use and double the maximum layer size.

Original: The bottleneck for 1% density and a layer size of 11520 is loading activations into shared memory, rather than the operate stage.
Revised: The bottleneck for 1% density and a layer size of 11520 is loading activations into shared memory, rather than the operate stage.

Original: If we were to split one row among multiple blocks and each block only required a fraction of the output activations, the further reduced shared memory useage would allow for large layers to be more efﬁcient.
Revised: If we were to split one row among multiple blocks and each block only required a fraction of the output activations, the further reduced shared memory useage would allow for large layers to be more efﬁcient.

Revised: Using a lower-precision data type for the activations would remove the shared memory bandwidth and storage burden and achieve a similar result.

Original: Our work can be extended to multiple GPUs, allowing for larger layer sizes and more nonzero parameters.
Revised: Our work can be extended to multiple GPUs, allowing for larger layer sizes and more nonzero parameters.

Original: Load balancing (due to non-uniform sparsity) is handled today with zero-padding, though several other approaches exist.
Revised: Load balancing (due to non-uniform sparsity) is handled today with zero-padding, though several other approaches exist.

Original: Without altering the network, we can deﬁne a number of classes for different amounts of sparsity and handle each class separately, assigning each row to one class or another based on its sparsity.
Revised: Without altering the network, we can deﬁne a number of classes for different amounts of sparsity and handle each class separately, assigning each row to one class or another based on its sparsity.

Original: If we have some control over the network itself, Han et al.
Revised: If we have some control over the network itself, Han et al.

Original: (2016a) have shown that load-balance aware pruning is an option, or the rows with fewer than the maximum number of nonzeros could have nonzero values re-instated to improve network accuracy ( Han et al.
Revised: (2016a) have shown that load-balance aware pruning is an option, or the rows with fewer than the maximum number of nonzeros could have nonzero values re-instated to improve network accuracy ( Han et al.

Original: with no impact to performance.
Revised: with no impact to performance.

Original: Without considering accuracy, we found that for a 25% dense layer of size 2304, batch size of 4, and 256 timesteps with both straightforward (unbalanced) and load-balanced pruning, and no code modiﬁcations to take advantage of any regularity, sparse GEMM’s performance does not change with load-balancing, but our approach’s beneﬁt over a dense GEMM increases from 1 .
Revised: Without considering accuracy, we found that for a 25% dense layer of size 2304, batch size of 4, and 256 timesteps with both straightforward (unbalanced) and load-balanced pruning, and no code modiﬁcations to take advantage of any regularity, sparse GEMM’s performance does not change with load-balancing, but our approach’s beneﬁt over a dense GEMM increases from 1 .

Original: 14 × to 1 .
Revised: 14 × to 1 .

Original: 94 × .
Revised: 94 × .

Original: Accuracy and performance results with this technique  are in the appendix, but the other load-balancing approaches remain future work.
Revised: Accuracy and performance results with this technique applied to a machine translation network are in the appendix, but the other load-balancing approaches remain future work.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduced sparse persistent RNNs, an efﬁcient new algorithm for accelerating pruned recurrent networks.
Revised: We introduced sparse persistent RNNs, an efﬁcient new algorithm for accelerating pruned recurrent networks.

Original: Further, we explored several optimizations that were needed to achieve these results on
Revised: Further, we explored several optimizations that were needed to achieve these results on

Original: V100, a recent GPU.
Revised: V100, a recent GPU.

Original: Our optimized technique allows for 7 . 3 × , 3 . 4 × , and 1 . 8 × speedups against dense GEMM, sparse GEMM, and dense persistent implementations for a hidden layer of size 1792, batch size of 4, and a density of 10%.
Revised: Our optimized technique allows for 7 . 3 × , 3 . 4 × , and 1 . 8 × speedups against dense GEMM, sparse GEMM, and dense persistent implementations for a hidden layer of size 1792, batch size of 4, and a density of 10%.

Original: We show that much larger networks can be deployed onto a GPU of a ﬁxed size with performance increases of around 5 × over the next best solution for a density of 1-30% on a layer size of 2304; notably, this sparsity range includes denser workloads than typically lead to a throughput increase.
Revised: We show that much larger networks can be deployed onto a GPU of a ﬁxed size with performance increases of around 5 × over the next best solution for a density of 1-30% on a layer size of 2304; notably, this sparsity range includes denser workloads than typically perform worse with sparse optimizations.

Original: We also show promising results on much larger layers7168 and 11520 achieve speedups of 1 . 9 × for 2.6% and 1% densities, respectively.
Revised: We also show promising results on much larger layers - 7168 and 11520 achieve speedups of 1 . 9 × for 2.6% and 1% densities, respectively.

Original: Our approach improved the speed up pruned NMT and speech recognition networks’ recurrent layers by up to 3 × .
Revised: Our approach speeds up pruned NMT and speech recognition networks’ recurrent layers by up to 3 × .

Original: Finally, load-balanced pruning can signiﬁcantly improve a network’s throughput, and our technique is necessary to achieve both high performance and accuracy in some recurrent layers, as detailed in the case study in our appendix.
Revised: Finally, load-balanced pruning can signiﬁcantly improve a network’s throughput, and our technique is necessary to achieve both high performance and accuracy in some recurrent layers, as detailed in the case studies in our appendix.

Revised: A PPENDIX A: A LGORITHM FOR B ANK -A WARE W EIGHT L AYOUT

Revised: As discussed in Section 4, we decouple the physical weight layout from the logical view.

Revised: Here, we share the greedy algorithm we used, which is one possible approach.

Revised: Algorithm 1: Optimize a row of nonzero weights to minimize bank conﬂicts

Revised: Invoked for each row independently

Original: A PPENDIX : C ASE S TUDIES
Revised: Input : Row .N where Row i ) index i

Revised: >

Revised: Initialization: Color (0 .. 31) with

Revised: Output : Row awareX (0 ..N − 1) ———————————————————————————————————————— for <index,value> pair in Row X do index _ t = index ; bank = index _ t %32 ; (cid:46) find its bank while Color ( bank ) >

Revised: N do index _ t + + ; (cid:46) Color(bank) is occupied.

Revised: Use a unfilled Color bank = index _ t %32 ; end

Revised: Row awareX ( Color ( bank )) =

Revised: < index, value > ;

Revised: Color ( bank ) =

Revised: Color ( bank ) + 32 ; (cid:46) Update Color(bank)’s next location end

Revised: A PPENDIX B: C ASE S TUDIES

Original: Having described our efﬁcient algorithm, we now perform case studies showing its utility, as well as its generality.
Revised: Having described our efﬁcient algorithm for accelerating pruned recurrent layers, we now perform case studies showing its utility, as well as its generality.

Original: The results in the main body are for RNNs only, but in the ﬁrst section, we show performance results for LSTM (Hochreiter & Schmidhuber, 1997) layers.
Revised: The results in the main body are for RNNs only, but in this ﬁrst section, we show performance results for LSTM (Hochreiter & Schmidhuber, 1997) layers.

Original: Implementation differences are fairly straightforward.
Revised: Implementation differences are fairly straightforward.

Original: Instead of one gate in an RNN, each thread must be responsible for four gates in an LSTM.
Revised: Instead of one gate in an RNN, each thread must be responsible for four gates in an LSTM.

Original: As a result, each thread’s weights belong to one of four rows of the original weight matrix, and each row requires a different activation function.
Revised: As a result, each thread’s weights belong to one of four rows of the original weight matrix, and each row requires a different activation function.

Original: The increased gate count results in an increased number of weights for the same layer size.
Revised: The increased gate count results in an increased number of weights for the same layer size.

Original: Dense persistent approaches will not be available for an LSTM of layer size of 1024 as a result (on the V100 GPU used for the experiments in this section).
Revised: Dense persistent approaches will not be available for an LSTM of layer size of 1024 as a result (on the V GPU used for the experiments in this section).

Original: We ﬁrst explore pruning a neural machine translation network to see how accuracy changes with sparsity.
Revised: We ﬁrst explore pruning a neural machine translation network to see how accuracy changes with sparsity.

Original: More importantly, we care about execution speed and accuracy; sparsity is merely one way to trade off speed for accuracy.
Revised: More importantly, we care about execution speed and accuracy; sparsity is merely one way to trade off speed for accuracy.

Original: (Layer size is much more straightforward.)
Revised: (Layer size is much more straightforward.)

Original: We outline our simple pruning and training procedures here.
Revised: We outline our simple pruning and training procedures here.

Original: Next, we use results from prior work that pruned a productized speech recognition network.
Revised: Next, we use results from prior work that pruned a productized speech recognition network.

Original: We show that our technique allows for higher performance for a given accuracy; without our technique, sparsity may not lead to any gains whatsoever.
Revised: We show that our technique allows for higher performance for a given accuracy; without our technique, sparsity may not lead to any gains whatsoever.

Original: 1. N EURAL M ACHINE T RANSLATION A. S ETUP
Revised: 1. N EURAL M ACHINE T RANSLATION A. S ETUP

Original: We use OpenNMT (Klein et al.) to perform translation from English to German using the WMT data set as our training data and the newstest2013 data set for validation.
Revised: We use OpenNMT (Klein et al., 2017) to perform translation from English to German using the WMT15 data set as our training data and the newstest2013 data set for validation.

Original: The common network architecture is a 2-layer (for both encoder and decoders) LSTM; our only variation from network to network is the layer size.
Revised: The common network architecture is a 2-layer (for both encoder and decoders) LSTM; our only variation from network to network is the layer size.

Original: This can be reproduced by following the excellent tutorial.
Revised: This can be reproduced by following the excellent tutorial.

Original: We trained each network with only two GPUs.
Revised: We trained each network with only two GPUs.

Original: Performance resultsin this section were gathered on NVIDIA’s latest
Revised: Performance results were gathered on a single NVIDIA

Original: Tesla V100 accelerator.
Revised: Tesla V100 accelerator.

Original: B. P RUNING S TRATEGIES
Revised: B. P RUNING S TRATEGIES

Original: Each of our pruned networks use a magnitude-based pruning scheme, in which we consider only the magnitude of the weight when deciding which weights to prune.
Revised: Each of our pruned networks use a magnitude-based pruning scheme, in which we consider only the magnitude of the weight when deciding which weights to prune.

Original: See et al.
Revised: See et al.

Original: (2016) found that different pruning methods affect accuracy differently, and we look at two options: naïve and row-balanced.
Revised: (2016) found that different pruning methods affect accuracy differently, and we look at two options: naïve and row-balanced.

Original: In both cases, each layer of the network is pruned to the same target density.
Revised: In both cases, each layer of the network is pruned to the same target density.

Original: Naïve pruning does not consider where the weight is within a layer; that is, all gates and all rows are considered to be the same.
Revised: Naïve pruning does not consider where the weight is within a layer; that is, all gates and all rows are considered to be the same.

Original: The effect of this strategy, as noted in past work, is that the forget gate is much sparser than the other gates in an LSTM; intuitively, the relative lack of constraints on the sparsity should be less of a burden on the network’s convergence.
Revised: The effect of this strategy, as noted in past work, is that the forget gate is much sparser than the other gates in an LSTM; intuitively, the relative lack of constraints on the sparsity should be less of a burden on the network’s convergence.

Original: Row-balanced pruning has the constraint that each row in a layer must have the same number of nonzero values.
Revised: Row-balanced pruning has the constraint that each row in a layer must have the same number of nonzero values.

Original: The effect of this strategy is that the weights are naturally more load-balanced for use with our approach, which should allow for higher performance (as evidenced in Section 6).
Revised: The effect of this strategy is that the weights are naturally more load-balanced for use with our approach, which should allow for higher performance (as evidenced in Section 6).

Original: When examining the effective density of the naïve-pruned layers after padding with zeros for loadbalancing (see Section 4), we found that the amount of remaining sparsity was not sufﬁcient to exploit with a sparse method 1 .
Revised: When examining the effective density of the naïve-pruned layers after padding with zeros for loadbalancing (see Section 4), we found that the amount of remaining sparsity was not sufﬁcient to exploit with a sparse method 1 .

Original: Further, the difference between the network accuracy of the two pruning options was negligible: less than 0.1 BLUE points in each case.
Revised: Further, the difference between the network accuracy of the two pruning options was negligible: less than 0.1 BLUE points in each case.

Original: So, we focus only on row-balanced pruning.
Revised: So, we focus only on row-balanced pruning.

Original: C. T RAINING P ROCESS
Revised: C. T RAINING P ROCESS

Original: For all networks, we train for the default 13 epochs, with all other hyperparameters unchanged from the default.
Revised: For all networks, we train for the default 13 epochs, with all other hyperparameters unchanged from the default.

Original: Sparse techniques are not limited to inference; recent work has shown that training with pruned weights is a viable option (Guo et al., 2016; Narang et al., 2017; Anonymous, 2018).
Revised: Sparse techniques are not limited to inference; recent work has shown that training with pruned weights is a viable option (Guo et al., 2016; Narang et al., 2017; Zhu & Gupta, 2017).

Original: For our experiments, we adopt a simple methodology: after one-half epoch of training, we prune immediately to the target density.
Revised: For our experiments, we adopt a simple methodology: after one-half epoch of training, we prune immediately to the target density.

Original: Like Guo et al.
Revised: Like Guo et al.

Original: (2016), we continue to update a "master copy" of un-pruned parameters during the backwards pass, but the pruned weights are used for computation during the forwards and backwards passes.
Revised: (2016), we continue to update a "master copy" of un-pruned parameters during the backwards pass, but the pruned weights are used for computation during the forwards and backwards passes.

Original: As all the weights are updated, the weights pruned during one pruning step may be re-introduced in the following pruning step, changing the sparsity pattern between pruning phases.
Revised: As all the weights are updated, the weights pruned during one pruning step may be re-introduced in the following pruning step, changing the sparsity pattern between pruning phases.

Original: This pruning step could impose a large overhead, so we only prune every half-epoch for a total ofpruning steps.
Revised: This pruning step could impose a large overhead, so we only prune every half-epoch for a total ofpruning steps.

Original: Thus, the ﬁrst half epoch is fully dense, the sparsity pattern changes every half-epoch through the end of epoch 11, and epochs 12 and 13 ﬁne-tune the ﬁnal sparsity pattern.
Revised: Thus, the ﬁrst half epoch is fully dense, the sparsity pattern changes every half-epoch through the end of epoch 11, and epochs 12 and 13 ﬁne-tune the ﬁnal sparsity pattern.

Original: D. A CCURACY R ESULTS
Revised: D. A CCURACY R ESULTS

Original: [Figure]
Revised: [Figure]

Original: Figure 5 shows the accuracy of various networks trained with and without sparsity, and the ﬁrst four columns of Table 2 show the detailed results.
Revised: Figure 5 shows the accuracy of various networks trained with and without sparsity, and the ﬁrst four columns of Table 2 show the detailed results.

Original: As expected, network accuracy increases with layer size, and, as demonstrated in other work, a large and sparse network can outperform a smaller dense network with the same number of nonzero parameters (“Effective Layer Size").
Revised: As expected, network accuracy increases with layer size, and, as demonstrated in other work, a large and sparse network can outperform a smaller dense network with the same number of nonzero parameters (“Effective Layer Size").

Original: One interesting result suggests the scalability of these results: a larger, sparser network (layer size of 1448 at 4.7% density for an effective size of 322) out-performed a slightly-smaller, not-as-sparse network (layer size of 1024 at 12.5% density for an effective size of 362).
Revised: One interesting result suggests the scalability of these results: a larger, sparser network (layer size of 1448 at 4.7% density for an effective size of 322) out-performed a slightly-smaller, not-as-sparse network (layer size of 1024 at 12.5% density for an effective size of 362).

Original: This is why the upper tail of the “sparse" curve in
Revised: This is why the upper tail of the “sparse" curve in

Original: Figure 5 dips down; in truth, the method should continue to scale.
Revised: Figure 5 dips down; in truth, the method should continue to scale.

Original: Omitting the 1448 layer size result would show monotonically increasing accuracy, as would ordering the accuracies by underlying (unpruned) layer sizes for these densities, as shown in Table 2.
Revised: Omitting the 1448 layer size result would show monotonically increasing accuracy, as would ordering the accuracies by underlying (unpruned) layer sizes for these densities, as shown in Table 2.

Revised: Even with our relatively simple training procedure

Original: Even with our relatively simple training procedure, network accuracy was within a reasonable distance of the dense baseline, only 0.4 BLEU worse at the largest (and most accurate) conﬁguration.
Revised: [Table] network accuracy was within a reasonable distance of the dense baseline [Table] only 0.4 BLEU worse at the largest (and most accurate) conﬁguration.

Original: We note that these accuracy results were achieved with a very simple method and can likely be improved by:
Revised: We note that these accuracy results were achieved with a very simple method and can likely be improved by:

Original: • More elaborate pruning schedules during training • Performing a sensitivity analysis and pruning layers to different densities • Pruning gates within a layer to different densities • Pruning only the recurrent weights, rather than the recurrent and feed-forward weights • Adjusting hyperparameters, such as dropout • Performing more ﬁne-tuning of the ﬁnal network (at the cost of extra training time) 
Revised: • More elaborate pruning schedules during training • Performing a sensitivity analysis and pruning layers to different densities • Pruning gates within a layer to different densities • Pruning only the recurrent weights, rather than the recurrent and feed-forward weights • Adjusting hyperparameters, such as dropout • Performing more ﬁne-tuning of the ﬁnal network (at the cost of extra training time) [Table]

Original: E. P ERFORMANCE R ESULTS
Revised: E. P ERFORMANCE R ESULTS

Original: While these accuracy results are not necessarily a surprise, a missing part of most treatments of this behavior is the throughput of the network on a given architecture.
Revised: While these accuracy results are not necessarily a surprise, a missing part of most treatments of this behavior is the throughput of the network on a given architecture.

Original: To show that a large, sparse network is not only a good tradeoff for accuracy but also for performance, we compare different layers’ performance with all state-of-the-art algorithms that support each particular layer size.In [Table] particular
Revised: To show that a large, sparse network is not only a good tradeoff for accuracy but also for performance, we compare different layers’ performance with all state-of-the-art algorithms that support each particular layer size.

Original: [Table] we use a dense GEMM (cuBLAS), dense persistent GEMM (cuDNN), sparse GEMM (cuSPARSE) [Table] and our sparse persistent GEMM.
Revised: In particular, we use a dense GEMM (cuBLAS), dense persistent GEMM (cuDNN), sparse GEMM (cuSPARSE), and our sparse persistent GEMM.

Original: We proﬁle these LSTM recurrent kernels on an
Revised: We proﬁle these LSTM recurrent kernels on an

Original: NVIDIA V100 GPU.
Revised: NVIDIA V100 GPU.

Original: An important note is that this gives us access to 96KB of shared memory per thread block, in turn allowing for larger layer sizes to be supported more efﬁciently.
Revised: An important note is that this gives us access to 96KB of shared memory per thread block, in turn allowing for larger layer sizes to be supported more efﬁciently.

Original: Tablehas the details for each layer size; our approach’s performance on the sparse network is given with emphasis .
Revised: Tablehas the details for each layer size; our approach’s performance on the sparse network is given with emphasis .

Original: Note that these absolute improvements are for a single layer; a production network would be composed of multiple layers.
Revised: Note that these absolute improvements are for a single layer; a production network would be composed of multiple layers.

Original: More importantly, the relative improvement can lead to a proportional speedup in training time, which can be days or weeks for large networks.
Revised: More importantly, the relative improvement can lead to a proportional speedup in training time, which can be days or weeks for large networks.

Original: Our ﬁndings in Figure 6 show that while a pruned version of a network can run signiﬁcantly faster than the dense version of that same network size (see our main text), the same accuracy could be obtained with a smaller, dense network.
Revised: Our ﬁndings in Figure 6 show that while a pruned version of a network can run signiﬁcantly faster than the dense version of that same network size (see our main text), the same accuracy could be obtained with a smaller, dense network.

Original: If this small, dense network is able to be implemented with a persistent kernel, then it can be more performant – without our technique.
Revised: If this small, dense network is able to be implemented with a persistent kernel, then it can be more performant – without our technique.

Original: cuSPARSE is very effective at outperforming dense GEMM at network sizes that a dense persistent approach cannot handle, but our technique pushes this sparse/dense crossover point much lower.
Revised: cuSPARSE is very effective at outperforming dense GEMM at network sizes that a dense persistent approach cannot handle, but our technique pushes this sparse/dense crossover point much lower.

Original: Now, sparse networks can outperform dense ones, even at smaller effective layer sizes.
Revised: Now, sparse networks can outperform dense ones, even at smaller effective layer sizes.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: is proportional

Original: 2. S PEECH R ECOGNITION
Revised: 2. S PEECH R ECOGNITION

Original: Deep Speech 2 (Amodei et al., 2015) is a state-of-the-art speech recognition network that has been pruned to expose an accuracy/sparsity tradeoff (Narang et al., 2017), so we refer to these resources for details regarding the network and associated training and pruning approaches.
Revised: Deep Speech 2 (Amodei et al., 2015) is a state-of-the-art speech recognition network that has been pruned to expose an accuracy/sparsity tradeoff (Narang et al., 2017), so we refer to these resources for details regarding the network and associated training and pruning approaches.

Original: Our purpose here is to see the performance of various algorithms on the different versions of the network that they trained.
Revised: Our purpose here is to see the performance of various algorithms on the different versions of the network that they trained.

Original: Since we did not train these networks ourselves, we generated a random, unstructured nonzero pattern for each workload; no load-balancing has been done prior to the optimization steps of our algorithm.
Revised: Since we did not train these networks ourselves, we generated a random, unstructured nonzero pattern for each workload; no load-balancing has been done prior to the optimization steps of our algorithm.

Original: We reproduce the authors’ information from Table 3 for layer size, sparsity, and accuracy (character error rate, CER) in our own Table 3, adding the performance of different algorithms on V100 for a batch size of 1 and 256 time steps.
Revised: We reproduce the authors’ information from Table 3 for layer size, sparsity, and accuracy (character error rate, CER) in our own Table 3, adding the performance of different algorithms on V100 for a batch size of 1 and 256 time steps.

Original: (As shown above, varying time steps does not signiﬁcantly change the relative performance.
Revised: (As shown above, varying time steps does not signiﬁcantly change the relative performance.

Original: We discuss the batch size further below
Revised: We discuss the batch size further below.)

Original: [Table]
Revised: [Table]

Original: [Table]

Original: We see the familiar pattern: for a given effective layer size, larger and sparse layers lead to more accurate models than small and dense layers.
Revised: We see the familiar pattern: for a given effective layer size, larger and sparse layers lead to more accurate models than small and dense layers.

Original: However, Figure 7 reveals that the dense persistent approach is along the pareto-optimal curve in the absence of our technique .
Revised: However, Figure 7 reveals that the dense persistent approach is along the pareto-optimal curve in the absence of our technique .

Original: In other words, sparse methods, though they may be faster than a dense GEMM (especially for a batch size of 1), can not out-perform a dense persistent approach.
Revised: In other words, sparse methods, though they may be faster than a dense GEMM (especially for a batch size of 1), can not out-perform a dense persistent approach.

Original: If speed is a metric of importance, then dense persistent kernels are the answer.
Revised: If speed is a metric of importance, then dense persistent kernels are the answer.

Original: If accuracy is the most important metric, then it has notbeen shown that a sparse network can outperform any dense network [Table] [Figure] [Table]
Revised: If accuracy is the most important metric, then past work has not shown that a sparse network can outperform any dense network.

Original: [Figure]

Original: [Table]

Original: [Table]

Original: [Figure]

Original: Likewise, for all but the most stringent error rates, a pruned network accelerated with our algorithm gives the highest throughput.

Original: The size of each marker is proportional to the size of the underlying (unpruned) hidden layer.

Original: Including our sparse persistent kernels, however, shows that they are better than dense persistent kernels for a given network; for a particular speed target, they can provide higher accuracy with a pruned network.
Revised: Including our sparse persistent kernels, however, shows that they are better than dense persistent kernels for a given network; for a particular speed target, they can provide higher accuracy with a pruned network.

Original: At very low batch sizes, a sparse GEMM can occupy a useful spot in the accuracy/speed tradeoff curve, but with only a batch size of 8, they fall to last place at all instances of the network.
Revised: At very low batch sizes, a sparse GEMM can occupy a useful spot in the accuracy/speed tradeoff curve, but with only a batch size of 8, they fall to last place at all instances of the network.

Original: Thus, in the absence of our technique, there would be no reason to prune a networkto perform batched inference.
Revised: Thus, in the absence of our technique, there would be no reason to prune a network batched inference performance. [Figure]

Revised: 2: Accuracy vs. Performance, BS

Revised: [Figure]

Original: 3. C ONCLUSIONS
Revised: 3. C ONCLUSIONS

Original: Without consideration of the accuracy of pruned networks and their execution speed (on state-of-theart algorithms) together, the conclusion that a large, sparse network is better than a small, dense one is not sufﬁciently proven.
Revised: Without consideration of the accuracy of pruned networks and their execution speed (on state-of-theart algorithms) together, the conclusion that a large, sparse network is better than a small, dense one is not sufﬁciently proven.

Original: Our technique and this case study show, among these insights, that this can be the case, however:
Revised: Our technique and these case studies show, among these insights, that this can be the case, however:

Original: • Our technique extends to LSTMs with little effort.
Revised: • Our technique extends to LSTMs with little effort.

Original: • Our technique allows for pruned recurrent layers to run more efﬁciently than with any other existing algorithm.
Revised: • Our technique allows for pruned recurrent layers to run more efﬁciently than with any other existing algorithm.

Original: • Naïve pruning is not necessarily better than pruning with load-balancing in mind, especially when considering achievable performance.
Revised: • Naïve pruning is not necessarily better than pruning with load-balancing in mind, especially when considering achievable performance.

Original: • Like past work, we see that a larger, sparse network can be more accurate than a smaller, dense one.
Revised: • Like past work, we see that a larger, sparse network can be more accurate than a smaller, dense one.

Original: • Comparisons against persistent kernels, which can beat non-resident sparse approaches, show that for a given accuracy or performance target, pruning a network and using our technique is the best choice.
Revised: • Comparisons against persistent kernels, which can beat non-resident sparse approaches, show that for a given accuracy or performance target, pruning a network and using our technique is the best choice.