Original: N EURAL N ETWORKS WITH B LOCK D IAGONAL I NNER P RODUCT L AYERS
Revised: N EURAL N ETWORKS WITH B LOCK D IAGONAL I NNER P RODUCT L AYERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Two issues continue to burden deep learning researchers and users: (1) neural networks are cumbersome tools that grow with the complexity of the learning problem, and (2) the activity of the fully connected, or inner product, layers remains mysterious.
Revised: Two difﬁculties continue to burden deep learning researchers and users: (1) neural networks are cumbersome tools that grow with the complexity of the learning problem, and (2) the activity of the fully connected, or inner product, layers remains mysterious.

Original: We make contributions to these two difﬁculties by considering a modiﬁed version of the fully connected layer we call a block diagonal inner product layer.
Revised: We make contributions to these two issues by considering a modiﬁed version of the fully connected layer we call a block diagonal inner product layer.

Original: These modiﬁed layers have weight matrices that are block diagonal, turning a single fully connected layer into a set of densely connected neuron groups.
Revised: These modiﬁed layers have weight matrices that are block diagonal, turning a single fully connected layer into a set of densely connected neuron groups.

Original: This method condenses network storage and speeds up the run time without signiﬁcant adverse effect on the testing accuracy, thus offering a new approach to solve the ﬁrst problem.
Revised: This method condenses network storage and speeds up the run time without signiﬁcant adverse effect on the testing accuracy, thus offering a new approach to solving the ﬁrst problem.

Original: Comparing the change in variance and singular values of the weights through training in a layer when varying the number of blocks gives insight into the second problem.
Revised: Comparing the change in variance and singular values of the weights through training in a layer when varying the number of blocks gives insight into the second problem.

Original: The ratio of the variance of the weights remains constant throughout training.
Revised: The ratio of the variance of the weights remains constant throughout training.

Original: That is, the relationship in structure is preserved in the ﬁnal parameter distribution.
Revised: That is, the relationship in structure is preserved in the ﬁnal parameter distribution.

Original: We observe that trained inner product layers have structure similar to that of truly random matrices with iid entries, and that each block in a block inner product layer behaves like a smaller copy, giving a better understanding of the nature of inner product layers.
Revised: We observe that trained inner product layers have structure similar to that of truly random matrices with iid entries, and that each block in a block inner product layer behaves like a smaller copy, giving a better understanding of the nature of inner product layers.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Today, it is well known that larger neural networks can better represent complex data and hence achieve higher accuracy than smaller networks (Hornik et al., 1989; Simonyan & Zisserman, 2014;
Revised: Today, it is well known that larger neural networks can better represent complex data and hence achieve higher accuracy than smaller networks (Hornik et al., 1989; Simonyan & Zisserman, 2014;

Original: Sermanet et al., 2014).
Revised: Sermanet et al., 2014).

Original: While larger networks are more capable than their smaller counterparts, their size consumes signiﬁcant storage and computational resources.
Revised: While larger networks are more capable than their smaller counterparts, their size consumes signiﬁcant storage and computational resources.

Original: Ideally, efforts to reduce memory requirements would also lessen computational demand, but often these competing interests force a trade-off.
Revised: Ideally, efforts to reduce memory requirements would also lessen computational demand, but often these competing interests force a trade-off.

Original: With the performance gain of large networks, we also renounce some understanding.
Revised: With the performance gain of large networks, we also renounce some understanding.

Original: The convolutional layers learn high-level features in the data; adding fully connected layers allows the network to learn non-linear combinations of these features and model more complex global patterns.
Revised: The convolutional layers learn high-level features in the data; adding fully connected layers allows the network to learn non-linear combinations of these features and model more complex global patterns.

Original: While the activation of convolutional layers exhibits the features learned, the knowledge of the fully connected layers remains obscured.
Revised: While the activation of convolutional layers exhibits the features learned, the knowledge of the fully connected layers remains obscured.

Original: The fully connected layers are unwieldy and perplexing, yet they continue to be present in the most successful networks (Krizhevsky et al., 2012; Zeiler & Fergus, 2013; Simonyan & Zisserman, 2014).
Revised: The fully connected layers are unwieldy and perplexing, yet they continue to be present in the most successful networks (Krizhevsky et al., 2012; Zeiler & Fergus, 2013; Simonyan & Zisserman, 2014).

Original: Our work addresses both memory and computational demand without compromise.
Revised: Our work addresses both memory and computational demand without compromise.

Original: Focusing our attention on the inner product layers, we decrease network complexity and begin to uncover the mechanism of inner product layers.
Revised: Focusing our attention on the inner product layers, we decrease network complexity and begin to uncover the mechanism of inner product layers.

Original: While larger network architectures achieve higher accuracy, there are a variety of methods to condense them without much harm to the network accuracy.
Revised: While larger network architectures achieve higher accuracy, there are a variety of methods to condense them without much harm to the network accuracy.

Original: One such technique that has gained popularity is pruning (Reed, 1993; Han et al., 2015a;b), but traditional pruning has disadvantages related to network runtime.
Revised: One such technique that has gained popularity is pruning (Reed, 1993; Han et al., 2015a;b), but traditional pruning has disadvantages related to network runtime.

Original: Most existing pruning processes signiﬁcantly slow down network training, and the ﬁnal trained network is usually slower to execute (Han et al., 2015a).
Revised: Most existing pruning processes signiﬁcantly slow down network training, and the ﬁnal trained network is usually slower to execute (Han et al., 2015a).

Original: Sparse format operations require additional overhead that can greatly slow down performance unless one prunes nearly all weight entries, which can damage network accuracy.
Revised: Sparse format operations require additional overhead that can greatly slow down performance unless one prunes nearly all weight entries, which can damage network accuracy.

Original: Localized memory access patterns can be computed faster than non-localized lookups.
Revised: Localized memory access patterns can be computed faster than non-localized lookups.

Original: By implementing block diagonal inner product layers in place of fully connected layers, we condense neural networks in a structured manner that speeds up the ﬁnal runtime and does little harm to the ﬁnal accuracy.
Revised: By implementing block diagonal inner product layers in place of fully connected layers, we condense neural networks in a structured manner that speeds up the ﬁnal runtime and does little harm to the ﬁnal accuracy.

Original: Block diagonal inner product layers can be implemented by either initializing a purely block diagonal weight matrix or by initializing a fully connected layer and focusing pruning efforts off the diagonal blocks to coax the dense weight matrix into structured sparsity.
Revised: Block diagonal inner product layers can be implemented by either initializing a purely block diagonal weight matrix or by initializing a fully connected layer and focusing pruning efforts off the diagonal blocks to coax the dense weight matrix into structured sparsity.

Original: The ﬁrst method also reduces the gradient computation time and hence the overall training time.
Revised: The ﬁrst method also reduces the gradient computation time and hence the overall training time.

Original: The latter method can improve accuracy and supports the robustness of networks to shaping .
Revised: The latter method can improve accuracy and supports the robustness of networks to shaping .

Original: That is, pruning can be used as a mapping between architectures–in particular, a mapping to more convenient architectures.
Revised: That is, pruning can be used as a mapping between architectures–in particular, a mapping to more convenient architectures.

Original: Depending on how many iterations the pruning process takes, this method may also speed up training.
Revised: Depending on how many iterations the pruning process takes, this method may also speed up training.

Original: We have converted a single fully connected layer into a group of smaller inner product learners whose combined efforts form a stronger learner, in essence boosting the layer.
Revised: We have converted a single fully connected layer into a group of smaller inner product learners whose combined efforts form a stronger learner, in essence boosting the layer.

Original: These methods also bring artiﬁcial neural networks closer to the architecture of biological mammalian brains, which have more local connectivity (Herculano-Houzel, 2012).
Revised: These methods also bring artiﬁcial neural networks closer to the architecture of biological mammalian brains, which have more local connectivity (Herculano-Houzel, 2012).

Original: Another link with the mammalian brain is the relationship to random matrix theory.
Revised: Another link with the mammalian brain is the relationship to random matrix theory.

Original: In neuroscience, synaptic connectivity is often represented by a matrix with entries drawn randomly from an appropriate distribution (Rajan & Abbott, 2006; Rajan, 2010).
Revised: In neuroscience, synaptic connectivity is often represented by a matrix with entries drawn randomly from an appropriate distribution (Rajan & Abbott, 2006; Rajan, 2010).

Original: The distribution of the singular values of a large, random matrix behaves predictably according to the Marchenko-Pastur Law (Marchenko &
Revised: The distribution of the singular values of a large, random matrix behaves predictably according to the Marchenko-Pastur Law (Marchenko &

Original: Pastur, 1967).
Revised: Pastur, 1967).

Original: We show that this distribution also represents artiﬁcial neural activity matrices well.
Revised: We show that this distribution also represents artiﬁcial neural activity matrices well.

Original: This relationship allows us to compare the behavior of inner product layers in networks that have related structure.
Revised: This relationship allows us to compare the behavior of inner product layers in networks that have related structure.

Original: Looking at the distribution of an inner product layer through training, we gain insight into neural networks as random dynamical systems, thereby uncovering a piece of the inner product layer “black box”.
Revised: Looking at the distribution of an inner product layer through training, we gain insight into neural networks as random dynamical systems, thereby uncovering a piece of the inner product layer “black box”.

Original: Speciﬁcally, we observe that when varying the number of blocks in a layer, the initial ratio of the variance of the weights is preserved to ﬁrst order throughout training.
Revised: Speciﬁcally, we observe that when varying the number of blocks in a layer, the initial ratio of the variance of the weights is preserved to ﬁrst order throughout training.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Weight pruning comes in many forms including penalty and second derivative methods, sensitivity analysis and cross validation; it can be done iteratively during training or to a trained network before reﬁning the surviving weights (Reed, 1993; Castellano et al., 1997; LeCun et al., 1990; Hassibi &
Revised: Weight pruning comes in many forms including penalty and second derivative methods, sensitivity analysis and cross validation; it can be done iteratively during training or to a trained network before reﬁning the surviving weights (Reed, 1993; Castellano et al., 1997; LeCun et al., 1990; Hassibi &

Original: Stork, 1992; Engelbrecht, 2001).
Revised: Stork, 1992; Engelbrecht, 2001).

Original: With any of these methods, the result is a sparse network that takes less storage space than its fully connected counterpart.
Revised: With any of these methods, the result is a sparse network that takes less storage space than its fully connected counterpart.

Original: Han et al.
Revised: Han et al.

Original: (2015b) iteratively prune a network using the penalty method by adding a mask that disregards pruned parameters for each weight tensor.
Revised: (2015b) iteratively prune a network using the penalty method by adding a mask that disregards pruned parameters for each weight tensor.

Original: This means that the number of required ﬂoating point operations decreases, but the number performed stays the same.
Revised: This means that the number of required ﬂoating point operations decreases, but the number performed stays the same.

Original: Furthermore, masking out updates takes additional time.
Revised: Furthermore, masking out updates takes additional time.

Original: Quantization and Huffman coding can be used to compress a trained network further (Xie & Jabri, 1992;
Revised: Quantization and Huffman coding can be used to compress a trained network further (Xie & Jabri, 1992;

Original: Han et al., 2015a).
Revised: Han et al., 2015a).

Original: Han et al.
Revised: Han et al.

Original: (2015a) report the average time spent on a forward propagation after pruning is complete and the resulting sparse layers have been converted to CSR format; for batch sizes larger than one, the sparse computations are signiﬁcantly slower than the dense calculations.
Revised: (2015a) report the average time spent on a forward propagation after pruning is complete and the resulting sparse layers have been converted to CSR format; for batch sizes larger than one, the sparse computations are signiﬁcantly slower than the dense calculations.

Original: Node pruning (He et al., 2014; Srinivas & Babu, 2015) or group lasso (Yuan & Lin, 2006; Lebedev & Lempitsky, 2016; Wen et al., 2016) could be used to speed up training and ﬁnal execution time more easily than weight pruning since node pruning preserves some structure, but drastic node pruning can harm the network accuracy.
Revised: Node pruning (He et al., 2014; Srinivas & Babu, 2015) or group lasso (Yuan & Lin, 2006; Lebedev & Lempitsky, 2016; Wen et al., 2016) could be used to speed up training and ﬁnal execution time more easily than weight pruning since node pruning preserves some structure, but drastic node pruning can harm the network accuracy.

Original: In practice, node pruning requires additional weight ﬁne-tuning to maintain accuracy.
Revised: In practice, node pruning requires additional weight ﬁne-tuning to maintain accuracy.

Original: Sindhwani et al.
Revised: Sindhwani et al.

Original: (2015) propose structured parameter matrices characterized by low displacement rank that yield high compression rate as well as fast forward and gradient evaluation.
Revised: (2015) propose structured parameter matrices characterized by low displacement rank that yield high compression rate as well as fast forward and gradient evaluation.

Original: However speedup is generally only seen for compression of large weight matrices, and their structured sparsity methods require parameter sharing which may be undesirable in some cases.
Revised: However speedup is generally only seen for compression of large weight matrices, and their structured sparsity methods require parameter sharing which may be undesirable in some cases.

Original: Block sparsity yields compression with parameter ﬂexibility.
Revised: Block sparsity yields compression with parameter ﬂexibility.

Original: Other approaches include storing a low rank approximation for a layer’s weight matrix (Sainath et al., 2013), using speciﬁc parameter sharing mechanisms (Chen et al., 2015), training smaller models on outputs of larger models (distillation) (Hinton et al., 2014), allowing neurons to read stale gradient updates (Ho et al., 2013) and using lower precision (Vanhoucke et al., 2011; M.Courbariaux et al., 2015; Gupta et al., 2015).
Revised: Other approaches include storing a low rank approximation for a layer’s weight matrix (Sainath et al., 2013), using speciﬁc parameter sharing mechanisms (Chen et al., 2015), training smaller models on outputs of larger models (distillation) (Hinton et al., 2014), allowing neurons to read stale gradient updates (Ho et al., 2013) and using lower precision (Vanhoucke et al., 2011; M.Courbariaux et al., 2015; Gupta et al., 2015).

Original: Note that using lower precision or stale gradients are techniques that can be applied to any of the methods considered, including ours.
Revised: Note that using lower precision or stale gradients are techniques that can be applied to any of the methods considered, including ours.

Original: Methods to condense and speedup convolutional layers, like predicting redundant parameters (Denil et al., 2013) and Group Lasso (Yuan & Lin, 2006; Lebedev & Lempitsky, 2016;
Revised: Methods to condense and speedup convolutional layers, like predicting redundant parameters (Denil et al., 2013) and Group Lasso (Yuan & Lin, 2006; Lebedev & Lempitsky, 2016;

Original: Wen et al., 2016), can also be used in conjunction with our methods.
Revised: Wen et al., 2016), can also be used in conjunction with our methods.

Original: There is less work considering the distribution of weights in artiﬁcial neural networks.
Revised: There is less work considering the distribution of weights in artiﬁcial neural networks.

Original: Initialization distributions to combat vanishing gradients are supported by theoretical variances for back propagation gradients under the assumption that the weights are independent, which is not valid beyond the ﬁrst iteration (Glorot & Bengio, 2010; Sun, 2015).
Revised: Initialization distributions to combat vanishing gradients are supported by theoretical variances for back propagation gradients under the assumption that the weights are independent, which is not valid beyond the ﬁrst iteration (Glorot & Bengio, 2010; Sun, 2015).

Original: Random weights have been looked at as good predictors of successful network architecture (Saxe et al., 2011).
Revised: Random weights have been looked at as good predictors of successful network architecture (Saxe et al., 2011).

Original: More recently, N.Tishby (2017) discussed the trend of the distribution of weight updates as they relate to the Mutual Information
Revised: More recently, N.Tishby (2017) discussed the trend of the distribution of weight updates as they relate to the Mutual Information

Original: Plane.
Revised: Plane.

Original: To the best of our knowledge, the effects of architecture on the change in distribution of the weights through training and the connection between trained inner product layer weights and random matrices have not been explored.
Revised: To the best of our knowledge, the effects of architecture on the change in distribution of the weights through training and the connection between trained inner product layer weights and random matrices have not been explored.

Original: In nuclear physics, the behavior of individual nuclei is the result of predictable forces, but the behavior of a collection of interacting particles becomes chaotic.
Revised: In nuclear physics, the behavior of individual nuclei is the result of predictable forces, but the behavior of a collection of interacting particles becomes chaotic.

Original: Wigner (1995) proposes a statistical description of such systems that is still widely used for studying physical systems with particle-particle interactions, like thermal conductivity.
Revised: Wigner (1995) proposes a statistical description of such systems that is still widely used for studying physical systems with particle-particle interactions, like thermal conductivity.

Original: Wigner claimed that variation in the positions of compound nuclei resonances is well represented in terms of the eigenvalues of a large symmetric matrix with iid real entries.
Revised: Wigner claimed that variation in the positions of compound nuclei resonances is well represented in terms of the eigenvalues of a large symmetric matrix with iid real entries.

Original: Brain activity behaves similarly at the macro level; in theoretical neuroscience, random matrices are used to model synaptic connections and to study brain plasticity (Sompolinsky et al., 1988; Rajan & Abbott, 2006; Rajan, 2010).
Revised: Brain activity behaves similarly at the macro level; in theoretical neuroscience, random matrices are used to model synaptic connections and to study brain plasticity (Sompolinsky et al., 1988; Rajan & Abbott, 2006; Rajan, 2010).

Original: Random matrices also appear in the study of large ecological systems (Allesina & Tang, 2015).
Revised: Random matrices also appear in the study of large ecological systems (Allesina & Tang, 2015).

Original: Knowing that inner product layers in artiﬁcial neural networks are well modeled by random matrices opens the ﬁeld to a new range of analytical tools that may support a speciﬁc network’s robustness or plasticity.
Revised: Knowing that inner product layers in artiﬁcial neural networks are well modeled by random matrices opens the ﬁeld to a new range of analytical tools that may support a speciﬁc network’s robustness or plasticity.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: We consider two methods for implementing block diagonal inner product layers:
Revised: We consider two methods for implementing block diagonal inner product layers:

Original: We initialize a layer with a purely block diagonal weight matrix and keep the number of connections constant throughout training.
Revised: We initialize a layer with a purely block diagonal weight matrix and keep the number of connections constant throughout training.

Original: We initialize a fully connected layer and iteratively prune entries off the diagonal blocks to achieve a block substructure.
Revised: We initialize a fully connected layer and iteratively prune entries off the diagonal blocks to achieve a block substructure.

Original: Within a layer, all blocks have the same size.
Revised: Within a layer, all blocks have the same size.

Original: Method 2 is accomplished in three phases: a dense phase, an iterative pruning phase and a block diagonal phase.
Revised: Method 2 is accomplished in three phases: a dense phase, an iterative pruning phase and a block diagonal phase.

Original: In the dense phase a fully connected layer is initialized and trained in the standard way.
Revised: In the dense phase a fully connected layer is initialized and trained in the standard way.

Original: During the iterative pruning phase, focused pruning is applied to entries off the diagonal blocks using the weight decay method with L 1 -norm.
Revised: During the iterative pruning phase, focused pruning is applied to entries off the diagonal blocks using the weight decay method with L 1 -norm.

Original: That is, if W is the weight matrix for a fully connected layer we wish to push toward block diagonal, we add α (cid:80) i,j | 1 i,j W i,j | to the loss function during the iterative pruning phase, where α is a tuning parameter and 1 i,j indicates whether W i,j is off the diagonal blocks in W .
Revised: That is, if W is the weight matrix for a fully connected layer we wish to push toward block diagonal, we add α (cid:80) i,j | 1 i,j W i,j | to the loss function during the iterative pruning phase, where α is a tuning parameter and 1 i,j indicates whether W i,j is off the diagonal blocks in W .

Original: During this phase, masking out updates for pruned entries is more efﬁcient than maintaining sparse format.
Revised: During this phase, masking out updates for pruned entries is more efﬁcient than maintaining sparse format.

Original: When pruning is complete, to maximize speedup it is best to reformat the weight matrix once such that the blocks are condensed and adjacent in memory.
Revised: When pruning is complete, to maximize speedup it is best to reformat the weight matrix once such that the blocks are condensed and adjacent in memory.

Original: 1 Batched smaller dense calculations for the blocks use cuBLAS strided batched multiplication (Nickolls et al., 2008).
Revised: 1 Batched smaller dense calculations for the blocks use cuBLAS strided batched multiplication (Nickolls et al., 2008).

Original: There is a lot of ﬂexibility in method 2 that can be tuned for speciﬁc user needs.
Revised: There is a lot of ﬂexibility in method 2 that can be tuned for speciﬁc user needs.

Original: More pruning iterations may increase the total training time but can yield higher accuracy and reduce overﬁtting.
Revised: More pruning iterations may increase the total training time but can yield higher accuracy and reduce overﬁtting.

Original: It should be mentioned that as the number of blocks increases, the overhead to perform cuBLAS strided batched multiplication can become noticeable; this library is not yet well optimized for performing many small matrix products (Masliah et al., 2016).
Revised: It should be mentioned that as the number of blocks increases, the overhead to perform cuBLAS strided batched multiplication can become noticeable; this library is not yet well optimized for performing many small matrix products (Masliah et al., 2016).

Original: However, with specialized batched multiplications for many small matrices, Jhurani & Mullowney (2015) attain up to 6 fold speedup.
Revised: However, with specialized batched multiplications for many small matrices, Jhurani & Mullowney (2015) attain up to 6 fold speedup.

Original:  E XPERIMENTS : SPEEDUP AND ACCURACY
Revised:  E XPERIMENTS : SPEEDUP AND ACCURACY

Original: Our goal is to reduce memory storage of the inner product layers while maintaining or reducing the ﬁnal execution time of the network with minimal loss in accuracy.
Revised: Our goal is to reduce memory storage of the inner product layers while maintaining or reducing the ﬁnal execution time of the network with minimal loss in accuracy.

Original: We will also see the reduction of total training time in some cases.
Revised: We will also see the reduction of total training time in some cases.

Original: To test this, we ran experiments on the MNIST (LeCun et al.)
Revised: To test this, we ran experiments on the MNIST (LeCun et al.)

Original: and CIFAR10 (Krizhevsky, 2009) datasets.
Revised: and CIFAR10 (Krizhevsky, 2009) datasets.

Original: All experiments are run on the Bridges’ NVIDIA P
Revised: All experiments are run on the Bridges’ NVIDIA P

Original: GPUs through the Pittsburgh Supercomputing Center.
Revised: GPUs through the Pittsburgh Supercomputing Center.

Original: We implement our work in Caffe (Jia et al., 2014) which provides suggested network architectures for these datasets.
Revised: We implement our work in Caffe (Jia et al., 2014) which provides suggested network architectures for these datasets.

Original: Training is done with batched gradient descent using the cross-entropy loss function on the softmax of the output layer.
Revised: Training is done with batched gradient descent using the cross-entropy loss function on the softmax of the output layer.

Original: We report the forward time per inner product layer when the layers are purely block diagonal; this time only measures the matrix multiplication in the forward step and does not include the time to layers, one should alter the output format of the previous layer and the expected input format of the following layer accordingly, in particular to row major ordering.
Revised: We report the forward time per inner product layer when the layers are purely block diagonal; this time only measures the matrix multiplication in the forward step and does not include the time to layers, one should alter the output format of the previous layer and the expected input format of the following layer accordingly, in particular to row major ordering.

Original: prune.
Revised: prune.

Original: We compare these times to the runtime of sparse matrix multiplication with random entries in CSR format using cuSPARSE (Nickolls et al., 2008).
Revised: We compare these times to the runtime of sparse matrix multiplication with random entries in CSR format using cuSPARSE (Nickolls et al., 2008).

Original: We also report the combined forward and backward time to do the three matrix products involved in gradient descent training when the layers are purely block diagonal.
Revised: We also report the combined forward and backward time to do the three matrix products involved in gradient descent training when the layers are purely block diagonal.

Original: For brevity we refer to the block diagonal method as ( b 1 , . . .
Revised: For brevity we refer to the block diagonal method as ( b 1 , . . .

Original: , b n ) -BD; b i = 1 indicates that layer i is fully connected.
Revised: , b n ) -BD; b i = 1 indicates that layer i is fully connected.

Original: FC is short for all inner product layers being fully connected.
Revised: FC is short for all inner product layers being fully connected.

Original: We refer to networks that differ only by the number of blocks in a single inner product layer as sister networks.
Revised: We refer to networks that differ only by the number of blocks in a single inner product layer as sister networks.

Original: 4. MNIST
Revised: 4. MNIST

Original: We experimented on the MNIST dataset with the LeNet-5 framework (LeCun et al., 1998).
Revised: We experimented on the MNIST dataset with the LeNet-5 framework (LeCun et al., 1998).

Original: LeNethas two convolutional layers with pooling followed by two inner product layers with ReLU activation.
Revised: LeNethas two convolutional layers with pooling followed by two inner product layers with ReLU activation.

Original: The ﬁrst inner product layer, ip1, has a 500 × 800 weight matrix, and the output inner product layer, ip2, has a 10 × 500 weight matrix.
Revised: The ﬁrst inner product layer, ip1, has a 500 × 800 weight matrix, and the output inner product layer, ip2, has a 10 × 500 weight matrix.

Original: We initialize the inner product layers using the Xavier weight ﬁller (Glorot & Bengio, 2010), which samples a uniform distribution with variance 1 /n in , where n in is the number of neurons feeding into a node.
Revised: We initialize the inner product layers using the Xavier weight ﬁller (Glorot & Bengio, 2010), which samples a uniform distribution with variance 1 /n in , where n in is the number of neurons feeding into a node.

Original: We used a training batch size of 64 and all other hyperparameters provided by Caffe are left unchanged.
Revised: We used a training batch size of 64 and all other hyperparameters provided by Caffe are left unchanged.

Original: Figure 1 shows time and accuracy results for block diagonal method 1 without pruning.
Revised: Figure 1 shows time and accuracy results for block diagonal method 1 without pruning.

Original: The ( b 1 , b 2 ) -
Revised: The ( b 1 , b 2 ) -

Original: BD architecture has (800 × 500) /b
Revised: BD architecture has (800 × 500) /b

Original: 1 + (500 × 10) /b 2 nonzero weights across both inner product layers.
Revised: 1 + (500 × 10) /b 2 nonzero weights across both inner product layers.

Original: The points at which the forward sparse and forward block curves meet in Figure 1 (left) indicate the fully connected dense forward runtimes for each layer.
Revised: The points at which the forward sparse and forward block curves meet in Figure 1 (left) indicate the fully connected dense forward runtimes for each layer.

Original: There is ≥ 1 . 4 × speedup for b 1 ≤ 50 , or 8000 nonzero entries, when timing both forward and backward matrix products, and 1 . 6 × speedup when b 1 = 100 , or 4000 nonzero entries, in the forward only case.
Revised: There is ≥ 1 . 4 × speedup for b 1 ≤ 50 , or 8000 nonzero entries, when timing both forward and backward matrix products, and 1 . 6 × speedup when b 1 = 100 , or 4000 nonzero entries, in the forward only case.

Original: Sparse format times are slow until there are less than 50 nonzero entries.
Revised: Sparse format times are slow until there are less than 50 nonzero entries.

Original: Figure 1 (right) shows a slight decline in accuracy as the number of nonzero entries decreases.
Revised: Figure 1 (right) shows a slight decline in accuracy as the number of nonzero entries decreases.

Original: FC achieves a ﬁnal accuracy of 99.11 % .
Revised: FC achieves a ﬁnal accuracy of 99.11 % .

Original: Without pruning, (100 , 10) -BD has a ﬁnal accuracy of 98.52 % .
Revised: Without pruning, (100 , 10) -BD has a ﬁnal accuracy of 98.52 % .

Original: In all cases testing accuracy remains within 1 % of FC accuracy.
Revised: In all cases testing accuracy remains within 1 % of FC accuracy.

Original: Using traditional iterative pruning with L 2 regularization, as suggested in (Han et al., 2015b), pruning every ﬁfth iteration until 4000 and 500 nonzero entries survived in ip1 and ip2 respectively gave an accuracy of 98.55 % , but the forward multiplication was more than 8 times slower than the dense fully connected case.
Revised: Using traditional iterative pruning with L 2 regularization, as suggested in (Han et al., 2015b), pruning every ﬁfth iteration until 4000 and 500 nonzero entries survived in ip1 and ip2 respectively gave an accuracy of 98.55 % , but the forward multiplication was more than 8 times slower than the dense fully connected case.

Original: Implementing (100 , 10) -BD method 2 with pruning using 15 dense iterations and 350 pruning iterations gave a ﬁnal accuracy of 98.65 % .
Revised: Implementing (100 , 10) -BD method 2 with pruning using 15 dense iterations and 350 pruning iterations gave a ﬁnal accuracy of 98.65 % .

Original: Thus we saw no great beneﬁt to using pruning over initializing pure block diagonal inner product layers for the MNIST dataset.
Revised: Thus we saw no great beneﬁt to using pruning over initializing pure block diagonal inner product layers for the MNIST dataset.

Original: In (Sindhwani et al., 2015), Toeplitz (3) has error rate 2.09 % using a single hidden layer net with 1000 hidden nodes on MNIST.
Revised: In (Sindhwani et al., 2015), Toeplitz (3) has error rate 2.09 % using a single hidden layer net with 1000 hidden nodes on MNIST.

Original: This method yields 63.32 fold compression over the fully connected setting.
Revised: This method yields 63.32 fold compression over the fully connected setting.

Original: However from their Figure 3, this slows down the forward pass by around 1 .
Revised: However from their Figure 3, this slows down the forward pass by around 1 .

Original: 5 × and the backward pass by around 5 .
Revised: 5 × and the backward pass by around 5 .

Original: 5 × .
Revised: 5 × .

Original: Our net with one hidden layer, 980 hidden nodes using (49 , 1) -BD on MNIST has 29.43 fold compression and error rate 4.37 % using method 2 with pruning.
Revised: Our net with one hidden layer, 980 hidden nodes using (49 , 1) -BD on MNIST has 29.43 fold compression and error rate 4.37 % using method 2 with pruning.

Original: Our speedup is 1.53 for forward only and 1.04 when combining the forward and backward runtime.
Revised: Our speedup is 1.53 for forward only and 1.04 when combining the forward and backward runtime.

Original: This is curious because the blocks of neurons in the hidden layer can only see a portion of the input images.
Revised: This is curious because the blocks of neurons in the hidden layer can only see a portion of the input images.

Original: 4. CIFAR 
Revised: 4. CIFAR

Original: We experimented on the CIFAR10 dataset with Krizhevsky’s cuda-convnet (2012).
Revised: We experimented on the CIFAR10 dataset with Krizhevsky’s cuda-convnet (2012).

Original: Cuda-convnet has three convolutional layers with ReLu activation and pooling, followed by two fully connected layers with no activation.
Revised: Cuda-convnet has three convolutional layers with ReLu activation and pooling, followed by two fully connected layers with no activation.

Original: The ﬁrst inner product layer, ip1, has a 64 × 1024 weight matrix, and the second has a 10 × 64 weight matrix.
Revised: The ﬁrst inner product layer, ip1, has a 64 × 1024 weight matrix, and the second has a 10 × 64 weight matrix.

Original: In all methods the inner product layer weights are initialized using a Gaussian ﬁller with standard deviation 0.1, as suggested by Caffe.
Revised: In all methods the inner product layer weights are initialized using a Gaussian ﬁller with standard deviation 0.1, as suggested by Caffe.

Original: We used a training batch size of 100, and all other hyperparameters provided by Caffe’s “quick” model are left unchanged.
Revised: We used a training batch size of 100, and all other hyperparameters provided by Caffe’s “quick” model are left unchanged.

Original: Caffe reports ≈ 75% accuracy with the FC architecture.
Revised: Caffe reports ≈ 75% accuracy with the FC architecture.

Original: Figure 2 shows time and accuracy results for the block diagonal method 1 without pruning.
Revised: Figure 2 shows time and accuracy results for the block diagonal method 1 without pruning.

Original: The ( b 1 , b 2 ) -BD architecture has (1024 × 64) /b 1 + (64 × 10) /b 2 nonzero entries across both inner product layers.
Revised: The ( b 1 , b 2 ) -BD architecture has (1024 × 64) /b 1 + (64 × 10) /b 2 nonzero entries across both inner product layers.

Original: In the ip1 layer, there is speedup for b 1 ≤ 32 , or 2048 nonzero entries, when timing both forward and backward matrix products, and b 1 ≤ 64 , or 1024 nonzero entries, in the forward only case.
Revised: In the ip1 layer, there is speedup for b 1 ≤ 32 , or 2048 nonzero entries, when timing both forward and backward matrix products, and b 1 ≤ 64 , or 1024 nonzero entries, in the forward only case.

Original: Again, sparse format performs poorly until there are less than 50 nonzero entries.
Revised: Again, sparse format performs poorly until there are less than 50 nonzero entries.

Original: Figure
Revised: Figure

Original: Using traditional iterative pruning with L 2 regularization pruning every ﬁfth iteration untiland 320 nonzero entries survived in the ﬁnal two inner product layers gave an accuracy of 75.18 % , but again the forward multiplication was more than 8 times slower than the dense fully connected computation.
Revised: Using traditional iterative pruning with L 2 regularization pruning every ﬁfth iteration untiland 320 nonzero entries survived in the ﬁnal two inner product layers gave an accuracy of 75.18 % , but again the forward multiplication was more than 8 times slower than the dense fully connected computation.

Original: On the other hand, implementing (64 , 2) -BD method 2 with pruning, which has corresponding numbers of nonzero entries, with 500 dense iterations and 1000 pruning iterations gave a ﬁnal accuracy of 74.81 % .
Revised: On the other hand, implementing (64 , 2) -BD method 2 with pruning, which has corresponding numbers of nonzero entries, with 500 dense iterations and 1000 pruning iterations gave a ﬁnal accuracy of 74.81 % .

Original: This is a 35.97 fold compression of the inner product layer parameters with only a 1.5 % drop in accuracy.
Revised: This is a 35.97 fold compression of the inner product layer parameters with only a 1.5 % drop in accuracy.

Original: The total forward runtime of (64 , 2) -BD is 1.6 times faster than
Revised: The total forward runtime of (64 , 2) -BD is 1.6 times faster than

Original: FC.
Revised: FC.

Original: To achieve comparable speed with sparse format we used traditional iterative pruning to leaveand 40 nonzero entries in the ﬁnal inner product layers giving an accuracy of 73.01 % .
Revised: To achieve comparable speed with sparse format we used traditional iterative pruning to leaveand 40 nonzero entries in the ﬁnal inner product layers giving an accuracy of 73.01 % .

Original: Thus implementing block diagonal layers with pruning yields comparable accuracy and memory condensation to traditional iterative pruning with faster ﬁnal execution time.
Revised: Thus implementing block diagonal layers with pruning yields comparable accuracy and memory condensation to traditional iterative pruning with faster ﬁnal execution time.

Original: Whole node pruning decreases the accuracy more than corresponding reductions in the block diagonal setting.
Revised: Whole node pruning decreases the accuracy more than corresponding reductions in the block diagonal setting.

Original: Node pruning until ip1 had only 2 outputs, i.e. a 1024 × 2 weight matrix, and ip2 had a 2 × 10 weight matrix for a total of 2068 weights between the two layers gave a ﬁnal accuracy 0f 59.67 % .
Revised: Node pruning until ip1 had only 2 outputs, i.e. a 1024 × 2 weight matrix, and ip2 had a 2 × 10 weight matrix for a total of 2068 weights between the two layers gave a ﬁnal accuracy 0f 59.67 % .

Original: (64,2)-BD has a total of 1344 weights between the two inner product layers and had a ﬁnal accuracy 15.14 % higher with pruning.
Revised: (64,2)-BD has a total of 1344 weights between the two inner product layers and had a ﬁnal accuracy 15.14 % higher with pruning.

Original: The ﬁnal accuracy on an independent test set was 76.29 % on CIFAR10 using the FC net while the ﬁnal accuracy on the training set itself was 83.32 % .
Revised: The ﬁnal accuracy on an independent test set was 76.29 % on CIFAR10 using the FC net while the ﬁnal accuracy on the training set itself was 83.32 % .

Original: Using the (64,2)-BD net without pruning, the accuracy on an independent test set was 72.49 % , but on the training set was 75.63 % .
Revised: Using the (64,2)-BD net without pruning, the accuracy on an independent test set was 72.49 % , but on the training set was 75.63 % .

Original: With pruning, the accuracy of (64,2)-BD on an independent test set was 74.81 % , but on the training set was 76.85 % .
Revised: With pruning, the accuracy of (64,2)-BD on an independent test set was 74.81 % , but on the training set was 76.85 % .

Original: Both block diagonal methods decrease overﬁtting; the block diagonal method with pruning decreases overﬁtting slightly more.
Revised: Both block diagonal methods decrease overﬁtting; the block diagonal method with pruning decreases overﬁtting slightly more.

Original:  R ANDOM MATRIX THEORY OBSERVATIONS
Revised:  R ANDOM MATRIX THEORY OBSERVATIONS

Original: The Marchenko-Pastur distribution describes the asymptotic behavior of the singular values of large random matrices with iid entries (1967).
Revised: The Marchenko-Pastur distribution describes the asymptotic behavior of the singular values of large random matrices with iid entries (1967).

Original: Let X be an m × n matrix with iid entries x ij such that
Revised: Let X be an m × n matrix with iid entries x ij such that

Original: E[ x ij ] = 0 and Var[ x ij ] = σ 2 .
Revised: E[ x ij ] = 0 and Var[ x ij ] = σ 2 .

Original: The Marchenko-Pastur theorem states that as n, m → ∞ such that m/n → y > 0 , with probability 1 the empirical spectral distribution of 1 n XX (cid:62) converges in distribution to the density
Revised: The Marchenko-Pastur theorem states that as n, m → ∞ such that m/n → y > 0 , with probability 1 the empirical spectral distribution of 1 n XX (cid:62) converges in distribution to the density

Original: [Equation]
Revised: [Equation]

Original: otherwise (1) with point mass 1 − 1 /y at the origin if y > 1 where a = σ 2 (1 − y 2 ) and b = σ 2 (1 + y 2 ) .
Revised: otherwise (1) with point mass 1 − 1 /y at the origin if y > 1 where a = σ 2 (1 − y 2 ) and b = σ 2 (1 + y 2 ) .

Original: Network weights not independent
Revised: Network weights not independent

Original: [Equation]
Revised: [Equation]

Original: in an iteration, where λ is the learning rate, b is the batch size, L is the loss function and x i is sampled from the data distribution.
Revised: in an iteration, where λ is the learning rate, b is the batch size, L is the loss function and x i is sampled from the data distribution.

Original: One can easily verify,
Revised: One can easily verify,

Original: [Equation]
Revised: [Equation]

Original: using estimators for the right side of the equation, where a simple Hadamard (Schur) product provides a good estimate for the second term.
Revised: using estimators for the right side of the equation, where a simple Hadamard (Schur) product provides a good estimate for the second term.

Original: In our experience, the covariance quickly became the dominating term.
Revised: In our experience, the covariance quickly became the dominating term.

Original: However, for a large enough weight matrix, the singular values of an inner product layer weight matrix behave according to the Marchenko-Pastur distribution even after thousands of training iterations.
Revised: However, for a large enough weight matrix, the singular values of an inner product layer weight matrix behave according to the Marchenko-Pastur distribution even after thousands of training iterations.

Original: That is, after thousands of correlated updates, the weight matrix behaves like a matrix with iid entries.
Revised: That is, after thousands of correlated updates, the weight matrix behaves like a matrix with iid entries.

Original: The assumption of independence fails at the micro level when calculating the change in variance of the weights, but is accurate at the macro level when considering the behavior of the weight matrix as an operator.
Revised: The assumption of independence fails at the micro level when calculating the change in variance of the weights, but is accurate at the macro level when considering the behavior of the weight matrix as an operator.

Original: In this section we discuss only the ﬁrst method for implementing block diagonal layers without pruning.
Revised: In this section we discuss only the ﬁrst method for implementing block diagonal layers without pruning.

Original: While a relationship between fully connected layer weights and the corresponding block diagonal layer weights in trained sister networks is not evident from equation (2), indeed a relationship can be seen in the singular values of the weight matrices, and, in particular, in the change in the variance of the weight matrix entries throughout training.
Revised: While a relationship between fully connected layer weights and the corresponding block diagonal layer weights in trained sister networks is not evident from equation (2), indeed a relationship can be seen in the singular values of the weight matrices, and, in particular, in the change in the variance of the weight matrix entries throughout training.

Original: The initialization ratio of variances in corresponding layer weight matrices of sister networks persists through thousands of training iterations.
Revised: The initialization ratio of variances in corresponding layer weight matrices of sister networks persists through thousands of training iterations.

Original: This ﬁnding may provide a good mechanism for examining related network architectures and may support claims about a network’s malleability or ﬁtness.
Revised: This ﬁnding may provide a good mechanism for examining related network architectures and may support claims about a network’s malleability or ﬁtness.

Original: 5. MNIST
Revised: 5. MNIST

Original: As mentioned, in our experiments on the MNIST dataset the inner product layer weights are initialized using the Xavier algorithm
Revised: As mentioned, in our experiments on the MNIST dataset the inner product layer weights are initialized using the Xavier algorithm

Original: (Glorot & Bengio, 2010).
Revised: (Glorot & Bengio, 2010).

Original: Thus the initialization variance of the weights in ip1 is b 1 / 800 in the block diagonal case, where b 1 is the number of blocks, and the ratio of this variance over that of the fully connected case at initialization is b 1 .
Revised: Thus the initialization variance of the weights in ip1 is b 1 / 800 in the block diagonal case, where b 1 is the number of blocks, and the ratio of this variance over that of the fully connected case at initialization is b 1 .

Original: Figure 3 (left) shows that this ratio is a good ﬁrst order estimate for the ﬁnal ratio after 10000 iterations.
Revised: Figure 3 (left) shows that this ratio is a good ﬁrst order estimate for the ﬁnal ratio after 10000 iterations.

Original: The ﬁnal ratios are 5 .
Revised: The ﬁnal ratios are 5 .

Original: 32 and 96 .
Revised: 32 and 96 .

Original: for b 1 = 5 , 10 , 20 , 50 , and 100 respectively.
Revised: for b 1 = 5 , 10 , 20 , 50 , and 100 respectively.

Original: We can see that the relationship deteriorates as the number of blocks increases.
Revised: We can see that the relationship deteriorates as the number of blocks increases.

Original: This holds for any random input order and when the sigmoid activation function is used for layer ip1 as long as the activation function is consistent across sister networks.
Revised: This holds for any random input order and when the sigmoid activation function is used for layer ip1 as long as the activation function is consistent across sister networks.

Original: When using the sigmoid activation function, the ratio seemed to deteriorate less quickly; e.g. the ﬁnal variance ratio was 101 . 

Revised: When using the sigmoid activation function, the ratio seemed to deteriorate less quickly; e.g. the ﬁnal variance ratio was 101 .

Original: for b 1 = 100 .
Revised: for b 1 = 100 .

Original: Figure 3 (right) compares the singular values of the ip1 layer weight matrix for varying numbers of blocks after 10000 training iterations to the singular values of a truly random 800 × 500 matrix initialized with variance 6 .
Revised: Figure 3 (right) compares the singular values of the ip1 layer weight matrix for varying numbers of blocks after 10000 training iterations to the singular values of a truly random 800 × 500 matrix initialized with variance 6 .

Original: 6 × 10 − 4 , the ﬁnal variance of the FC ip1 layer weights.
Revised: 6 × 10 − 4 , the ﬁnal variance of the FC ip1 layer weights.

Original: The singular values of the ip1 weight matrix follow the curve that the singular values of the truly random matrix create with some error in the largest and smallest singular values.
Revised: The singular values of the ip1 weight matrix follow the curve that the singular values of the truly random matrix create with some error in the largest and smallest singular values.

Original: We believe the disparity in the extreme singular values is the result of over training (See Appendix B.1); using FC, the accuracy reaches 98.28 % by iteration 1000.
Revised: We believe the disparity in the extreme singular values is the result of over training (See Appendix B.1); using FC, the accuracy reaches 98.28 % by iteration 1000.

Original: In the block diagonal method, we aggregate the singular values of each block and sort them to create the corresponding curves in Figure 3 (right).
Revised: In the block diagonal method, we aggregate the singular values of each block and sort them to create the corresponding curves in Figure 3 (right).

Original: The individual block spectral distributions are identical to each other.
Revised: The individual block spectral distributions are identical to each other.

Original: The decrease in matrix size by a factor of bcancels with the increase in variance by a factor of b 1 in equation (1), resulting in singular values that all follow the same curve.
Revised: The decrease in matrix size by a factor of bcancels with the increase in variance by a factor of b 1 in equation (1), resulting in singular values that all follow the same curve.

Original: Thus, the ratio of singular values of the trained ip1 layer weight matrix over the singular values of the truly random matrix is ≈ 1 .
Revised: Thus, the ratio of singular values of the trained ip1 layer weight matrix over the singular values of the truly random matrix is ≈ 1 .

Original: Additional supporting ﬁgures comparing the PDFs of the singular values and the Kullback-Leibler divergence can be found in Appendix B.1.
Revised: Additional supporting ﬁgures comparing the PDFs of the singular values and the Kullback-Leibler divergence can be found in Appendix B.1.

Original: If we make ip2 a BD layer as well, the change in variance in ip1 sees minimal effect.
Revised: If we make ip2 a BD layer as well, the change in variance in ip1 sees minimal effect.

Original: For b 1 = 1 , 2 , 5 , 10 , 50 , 100 and b 2 = 1 , 2 , 5 , 10 , the ﬁnal variance in the ip1 layer weights using ( b 1 , b 2 )
Revised: For b 1 = 1 , 2 , 5 , 10 , 50 , 100 and b 2 = 1 , 2 , 5 , 10 , the ﬁnal variance in the ip1 layer weights using ( b 1 , b 2 )

Original: -BD over the ﬁnal variance in the ip1 layer weights using ( b 1 , 1) -BD is ≈ 1 with error ≤ 0 .
Revised: -BD over the ﬁnal variance in the ip1 layer weights using ( b 1 , 1) -BD is ≈ 1 with error ≤ 0 .

Original: 5. CIFAR 
Revised: 5. CIFAR

Original: In our experiments on CIFAR10, the ﬁrst inner product layer weights are initialized using a Gaussian ﬁller with standard deviation 0.1.
Revised: In our experiments on CIFAR10, the ﬁrst inner product layer weights are initialized using a Gaussian ﬁller with standard deviation 0.1.

Original: Thus the ratio of variance of the weights in ip1 in the block diagonal case over that of the fully connected case at initialization is 1.
Revised: Thus the ratio of variance of the weights in ip1 in the block diagonal case over that of the fully connected case at initialization is 1.

Original: Figure 4 (left) indicates that this ratio is a good ﬁrst order estimate for the ﬁnal ratio after 9000 iterations at which time the ratios are 1 .
Revised: Figure 4 (left) indicates that this ratio is a good ﬁrst order estimate for the ﬁnal ratio after 9000 iterations at which time the ratios are 1 .

Original: 02 , 1 . 01 , 1 . 11 , 1 . 21 and 1 .
Revised: 02 , 1 . 01 , 1 . 11 , 1 . 21 and 1 .

Original: 5 for b 1 = 2 , 4 , 8 , 16 , 32 , and 64 respectively.
Revised: 5 for b 1 = 2 , 4 , 8 , 16 , 32 , and 64 respectively.

Original: Again, the relationship deteriorates as the number of blocks grows, and this holds for any random input order.
Revised: Again, the relationship deteriorates as the number of blocks grows, and this holds for any random input order.

Original: We compared the singular values of the weight matrix in layer ip1 for varying numbers of blocks after 9000 training iterations to the singular values of a truly random 1024 × 64 matrix initialized with variance 7 × 10 − 3 , the ﬁnal variance of the FC ip1 layer weights after training.
Revised: We compared the singular values of the weight matrix in layer ip1 for varying numbers of blocks after 9000 training iterations to the singular values of a truly random 1024 × 64 matrix initialized with variance 7 × 10 − 3 , the ﬁnal variance of the FC ip1 layer weights after training.

Original: The singular values of the fully connected ip1 layer weight matrix follow the curve that the singular values of the truly random matrix create.
Revised: The singular values of the fully connected ip1 layer weight matrix follow the curve that the singular values of the truly random matrix create.

Original: As in the MNIST experiments, we aggregate the singular values of each block in the block diagonal method and sort them.
Revised: As in the MNIST experiments, we aggregate the singular values of each block in the block diagonal method and sort them.

Original: In this case, variance of the ip1 layer weights remains relatively constant for varying numbers of blocks, so, by equation (1), the decrease in matrix size by a factor of b 1 must appear in the ratio of singular values as 1 / √ b 1 .
Revised: In this case, variance of the ip1 layer weights remains relatively constant for varying numbers of blocks, so, by equation (1), the decrease in matrix size by a factor of b 1 must appear in the ratio of singular values as 1 / √ b 1 .

Original: Figure 4 (right) shows this relationship.
Revised: Figure 4 (right) shows this relationship.

Original: The ratio of singular values of the trained weight matrix in layer ip1 over the singular values of the random matrix is ≈ 1 / √ b 1 for b 1 blocks with deterioration as b 1 grows.
Revised: The ratio of singular values of the trained weight matrix in layer ip1 over the singular values of the random matrix is ≈ 1 / √ b 1 for b 1 blocks with deterioration as b 1 grows.

Original: Additional supporting ﬁgures comparing the PDFs of the singular values and the Kullback-Leibler divergence can be found in Appendix B.2.
Revised: Additional supporting ﬁgures comparing the PDFs of the singular values and the Kullback-Leibler divergence can be found in Appendix B.2.

Original: If in addition we make the ip2 a BD layer, again, the change in variance in ip1 sees minimal effect.
Revised: If in addition we make the ip2 a BD layer, again, the change in variance in ip1 sees minimal effect.

Original: For b 1 = 1 , 2 , 4 , 8 , 16 , 32 , 64 , the ﬁnal variance in the ip1 layer weights using the ( b 1 , 2) -BD method over the ﬁnal variance in the ip1 layer weights using the ( b 1 , 1)
Revised: For b 1 = 1 , 2 , 4 , 8 , 16 , 32 , 64 , the ﬁnal variance in the ip1 layer weights using the ( b 1 , 2) -BD method over the ﬁnal variance in the ip1 layer weights using the ( b 1 , 1)

Original: -BD method is ≈ 1 with error ≤ 0 . 03 .
Revised: -BD method is ≈ 1 with error ≤ 0 . 03 .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have shown that block diagonal inner product layers can reduce network size, training time and ﬁnal execution time without signiﬁcant harm to the network performance.
Revised: We have shown that block diagonal inner product layers can reduce network size, training time and ﬁnal execution time without signiﬁcant harm to the network performance.

Original: We have also shown that random matrix theory gives informative results about relationships in network structure that are preserved through training.
Revised: We have also shown that random matrix theory gives informative results about relationships in network structure that are preserved through training.

Original: While traditional iterative pruning can reduce storage, the random indices of the surviving weights make sparse computation inefﬁcient, slowing down the training and ﬁnal execution time of the network.
Revised: While traditional iterative pruning can reduce storage, the random indices of the surviving weights make sparse computation inefﬁcient, slowing down the training and ﬁnal execution time of the network.

Original: Our block diagonal methods address this inefﬁciency by conﬁning dense regions to blocks along the diagonal.
Revised: Our block diagonal methods address this inefﬁciency by conﬁning dense regions to blocks along the diagonal.

Original: Without pruning, block diagonal method 1 allows for faster training time.
Revised: Without pruning, block diagonal method 1 allows for faster training time.

Original: Method 2 preserves the learning with focused, structured pruning that reduces computation for speedup during execution.
Revised: Method 2 preserves the learning with focused, structured pruning that reduces computation for speedup during execution.

Original: In our experiments, method 2 saw higher accuracy than the purely block diagonal method for the more complex learning problem, CIFAR10; however, the increase in accuracy came in exchange for slightly more time to train the network.
Revised: In our experiments, method 2 saw higher accuracy than the purely block diagonal method for the more complex learning problem, CIFAR10; however, the increase in accuracy came in exchange for slightly more time to train the network.

Original: There is great deal of ﬂexibility in our block diagonal methods that can be tuned to an individual problem.
Revised: There is great deal of ﬂexibility in our block diagonal methods that can be tuned to an individual problem.

Original: These methods may also make larger network architectures more feasible to train and use since they convert a fully connected layer into a collection of smaller inner product learners working jointly to form a stronger learner.
Revised: These methods may also make larger network architectures more feasible to train and use since they convert a fully connected layer into a collection of smaller inner product learners working jointly to form a stronger learner.

Original: There is a lot of room for additional speedup with block diagonal layers.
Revised: There is a lot of room for additional speedup with block diagonal layers.

Original: Dependency between layers poses a noteworthy bottleneck in network parallelization.
Revised: Dependency between layers poses a noteworthy bottleneck in network parallelization.

Original: With structured sparsity like ours, one no longer needs a full barrier between layers.
Revised: With structured sparsity like ours, one no longer needs a full barrier between layers.

Original: Additional speedup would be seen in software optimized to support weight matrices with organized sparse form, such as blocks, rather than being optimized for dense matrices.
Revised: Additional speedup would be seen in software optimized to support weight matrices with organized sparse form, such as blocks, rather than being optimized for dense matrices.

Original: For example, for many small blocks, one can reach up to 6 fold speedup with specialized batched matrix multiplication (Jhurani & Mullowney, 2015).
Revised: For example, for many small blocks, one can reach up to 6 fold speedup with specialized batched matrix multiplication (Jhurani & Mullowney, 2015).

Original: Hardware has been developing to better support sparse operations.
Revised: Hardware has been developing to better support sparse operations.

Original: Block format may be especially suitable for training on evolving architectures such as neuromorphic systems.
Revised: Block format may be especially suitable for training on evolving architectures such as neuromorphic systems.

Original: These systems, which are far more efﬁcient than GPUs at simulating mammalian brains, have a pronounced 2-D structure and are ill-suited to large dense matrix calculations (Merolla et al., 2014; Boahen, 2014).
Revised: These systems, which are far more efﬁcient than GPUs at simulating mammalian brains, have a pronounced 2-D structure and are ill-suited to large dense matrix calculations (Merolla et al., 2014; Boahen, 2014).

Original: Comparing block diagonal and fully connected layers in sister networks highlighted that the training process for neural networks can be viewed as a random dynamical system.
Revised: Comparing block diagonal and fully connected layers in sister networks highlighted that the training process for neural networks can be viewed as a random dynamical system.

Original: The relationship in the change in variance through training between the block diagonal and fully connected case suggests an invariant pattern that may help evaluate network ﬁtness.
Revised: The relationship in the change in variance through training between the block diagonal and fully connected case suggests an invariant pattern that may help evaluate network ﬁtness.

Original: We emphasize that this is a nontrivial relationship surviving various datasets, network architectures, and activation functions.
Revised: We emphasize that this is a nontrivial relationship surviving various datasets, network architectures, and activation functions.

Original: The group behavior is that of a random matrix with independent entries, but the individual weight updates have complex dependencies.
Revised: The group behavior is that of a random matrix with independent entries, but the individual weight updates have complex dependencies.

Original: Similar random activity occurs in the mammalian brain and suggests looking at random matrix theory to support a network’s plasticity or robustness.
Revised: Similar random activity occurs in the mammalian brain and suggests looking at random matrix theory to support a network’s plasticity or robustness.

Original: This could also be a catalyst for looking at optimal distributions for a trained network.
Revised: This could also be a catalyst for considering optimal distributions for a trained network.

Original: Random matrix theory has been indispensable to the advancement of nuclear physics, quantum physics, neuroscience, and ecology, and has the potential to elevate artiﬁcial neural network analysis in the same manner.
Revised: Random matrix theory has been indispensable to the advancement of nuclear physics, quantum physics, neuroscience, and ecology, and has the potential to elevate artiﬁcial neural network analysis in the same manner.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: All experiments are run on the Bridges’ NVIDIA P100 GPUs through the Pittsburgh Supercomputing Center.
Revised: All experiments are run on the Bridges’ NVIDIA P100 GPUs through the Pittsburgh Supercomputing Center.

Original: A PPENDIX A K ULLBACK -L EIBLER DIVERGENCE
Revised: A PPENDIX A K ULLBACK -L EIBLER DIVERGENCE

Original: The Kullback-Leibler divergence (also called relative entropy) is a measure of how one probability distribution diverges from a second probability distribution (Kullback & Leibler, 2015).
Revised: The Kullback-Leibler divergence (also called relative entropy) is a measure of how one probability distribution diverges from a second probability distribution (Kullback & Leibler, 2015).

Original: The
Revised: The

Original: Kullback-Leibler divergence from Q to P , denoted D KL ( P || Q ) , is the information gain if P is used instead of Q , or the amount of information lost when Q is used to approximate P .
Revised: Kullback-Leibler divergence from Q to P , denoted D KL ( P || Q ) , is the information gain if P is used instead of Q , or the amount of information lost when Q is used to approximate P .

Original: [Equation]
Revised: [Equation]

Original: With a Kullback-Leibler divergence of 0 , one can expect similar, if not identical, behavior of the two different distributions.
Revised: With a Kullback-Leibler divergence of 0 , one can expect similar, if not identical, behavior of the two different distributions.

Original: B R ANDOM MATRIX THEORY ADDITIONAL SUPPORTING MATERIAL B. MNIST
Revised: B R ANDOM MATRIX THEORY ADDITIONAL SUPPORTING MATERIAL B. MNIST

Original: The ratio of the variance and singular values of the ip1 layer weight matrix in block diagonal setting over that of the ip1 layer weight matrix in the fully connected setting after 10000 training iterations on the MNIST dataset using Lenet-5 framework can be seen in Figure 3.
Revised: The ratio of the variance and singular values of the ip1 layer weight matrix in block diagonal setting over that of the ip1 layer weight matrix in the fully connected setting after 10000 training iterations on the MNIST dataset using Lenet-5 framework can be seen in Figure 3.

Original: These ratios  highlight the relationship to random matrix theory best, but here we include the variance and the singular values without taking a ratio because they may also be informative for some readers.
Revised: These ratios best highlight the relationship to random matrix theory, but here we include the variance and the singular values without taking a ratio because they may also be informative for some readers.

Original: See Figure 5.
Revised: See Figure 5.

Original: Figure 6 (left) compares the theoretical probability density function of the singular values of a truly random matrix with independent entries to the measured distribution of the ip1 layer weight matrix singular values for the FC architecture after 10000 training iterations.
Revised: Figure 6 (left) compares the theoretical probability density function of the singular values of a truly random matrix with independent entries to the measured distribution of the ip1 layer weight matrix singular values for the FC architecture after 10000 training iterations.

Original: Figure 6 (right) uses the
Revised: Figure 6 (right) uses the

Original: Kullback-Leibler divergence (Appendix A) to measure the difference between the theoretical probability density function of the singular values of a truly random matrix to the measured distribution of the ip1 layer weight matrix singular values the ( b 1 , 1) -BD architecture10000 training iterations.
Revised: Kullback-Leibler divergence (Appendix A) to measure the difference between the theoretical probability density function of the singular values of a truly random matrix to the measured distribution of the ip1 layer weight matrix singular values the ( b 1 , 1) -BD architecture training iterations.

Original: For b 1 = 1 , 5 , 10 , 20 , 50 , and 100 the Kullback-Leibler divergence stays between 0 .and 0 .
Revised: For b 1 = 1 , 5 , 10 , 20 , 50 , and 100 the Kullback-Leibler divergence stays between 0 .and 0 .

Original: 081 with the highest values when b 1 = 1 , i.e. the FC case.
Revised: 081 with the highest values when b 1 = 1 , i.e. the FC case.

Original: One can see from Figures 3 (right) and 5 (right) that the disparity in the larger singular values decreases as b 1 increases.
Revised: One can see from Figures 3 (right) and 5 (right) that the disparity in the larger singular values decreases as b 1 increases.

Original: For comparison, the average Kullback-Leibler divergence from the discrete measured empirical spectral distribution of 1500 RR (cid:62) where R is an 800 × 500 matrix with standard normally distributed entries to the Marchenko-Pastur distribution µ 0 .
Revised: For comparison, the average Kullback-Leibler divergence from the discrete measured empirical spectral distribution of 1500 RR (cid:62) where R is an 800 × 500 matrix with standard normally distributed entries to the Marchenko-Pastur distribution µ 0 .

Original: 625 as in equation (1) is D KL ( µ 0 .
Revised: 625 as in equation (1) is D KL ( µ 0 .

Original: 625 ||
Revised: 625 ||

Original: λ 1500 RR (cid:62) )
Revised: λ 1500 RR (cid:62) )

Original: = 0 .
Revised: = 0 .

Original: 014 where the average is taken over over 20 initializations.
Revised: 014 where the average is taken over over 20 initializations.

Original: This divergence is the same magnitude as the measured divergence in Figure 6 (right).
Revised: This divergence is the same magnitude as the measured divergence in Figure 6 (right).

Original: It should be mentioned that D KL ( λ 1500 W b 1 W (cid:62) b 1 || µ 0 . 625 ) is inﬁnite because of the difference in support, but this disparity in support reduces as b 1 increases.
Revised: It should be mentioned that D KL ( λ 1500 W b 1 W (cid:62) b 1 || µ 0 . 625 ) is inﬁnite because of the difference in support, but this disparity in support reduces as b 1 increases.

Original: This disparity may be incurred by the relatively small matrix dimensions, as asymptotic random matrix theory often does not represent the extreme singular values well in the ﬁnite case (Rudelson & Vershynin, 2010), but is more likely caused by overtraining.
Revised: This disparity may be incurred by the relatively small matrix dimensions, as asymptotic random matrix theory often does not represent the extreme singular values well in the ﬁnite case (Rudelson & Vershynin, 2010), but is more likely caused by overtraining.

Original: Using FC, the accuracy reaches 98.28 % by iteration 1000.
Revised: Using FC, the accuracy reaches 98.28 % by iteration 1000.

Original: After 1000 iterations, the ratio of the largest singular value of the trained weight matrix in layer ip1 over the largest singular value of a truly random matrix with the same dimensions and variance is 1.16.
Revised: After 1000 iterations, the ratio of the largest singular value of the trained weight matrix in layer ip1 over the largest singular value of a truly random matrix with the same dimensions and variance is 1.16.

Original: Figure(right) shows that this ratio is 1.44 after the full 10000 iterations.
Revised: Figure(right) shows that this ratio is 1.44 after the full 10000 iterations.

Original: D
Revised: D

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: after only 1000 iterations.
Revised: after only 1000 iterations.

Original: The relationship between inner product layer weight matrices and random matrix theory was ﬁrst seen in inner product layer nets without convolutional layers.
Revised: The relationship between inner product layer weight matrices and random matrix theory was ﬁrst seen in inner product layer nets without convolutional layers.

Original: With interest leaning toward convolutional nets we did not include this in the main work, but we include this here for completeness.
Revised: With interest leaning toward convolutional nets we did not include this in the main work, but we include this here for completeness.

Original: The relationship is stronger in deeper layers and for larger weight matrices.
Revised: The relationship is stronger in deeper layers and for larger weight matrices.

Original: In a 3 layer network in which both hidden layers have 500 nodes and ReLu activation, we compare
Revised: In a 3 layer network in which both hidden layers have 500 nodes and ReLu activation, we compare

Original: FC to (1,100,1)- BD on MNIST where in both cases ip2 is initialized with Xavier variance (Glorot & Bengio, 2010) and all other hyperparameters as perscribed by Caffe (Jia et al., 2014).
Revised: FC to (1,100,1)- BD on MNIST where in both cases ip2 is initialized with Xavier variance (Glorot & Bengio, 2010) and all other hyperparameters as perscribed by Caffe (Jia et al., 2014).

Original: Afteriterations, the ratio of variance of weight matrix entries in layer ip2 in block diagonal setting over variance of weight matrix entries in layer ip2 in the fully connected setting is 106.42.
Revised: Afteriterations, the ratio of variance of weight matrix entries in layer ip2 in block diagonal setting over variance of weight matrix entries in layer ip2 in the fully connected setting is 106.42.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Let λ 1500 W 1 W (cid:62) 1 be the measured empirical spectral distribution of 1500 W 1 W (cid:62) 1 where W 1 is the ip layer weight matrix in the FC architecture with only 3 inner product layers after 10000 iterations.
Revised: Let λ 1500 W 1 W (cid:62) 1 be the measured empirical spectral distribution of 1500 W 1 W (cid:62) 1 where W 1 is the ip layer weight matrix in the FC architecture with only 3 inner product layers after 10000 iterations.

Original: The ﬁnal variance of the ip2 layer weights in FC is 0.0012.
Revised: The ﬁnal variance of the ip2 layer weights in FC is 0.0012.

Original: If R is a 500 × 500 truly random matrix with independent entries that have variance 0.0012, then D KL ( λ 1500 RR (cid:62) ||
Revised: If R is a 500 × 500 truly random matrix with independent entries that have variance 0.0012, then D KL ( λ 1500 RR (cid:62) ||

Original: λ 1500 W 1 W (cid:62) 1 ) = 0 .
Revised: λ 1500 W 1 W (cid:62) 1 ) = 0 .

Original: B. CIFAR 
Revised: B. CIFAR

Original: The ratio of the variance and singular values of the ip1 layer weight matrix in block diagonal setting over that of the ip1 layer weight matrix in the fully connected setting after 9000 training iterations on CIFAR10 using cuda-convnet can be seen in Figure 4.
Revised: The ratio of the variance and singular values of the ip1 layer weight matrix in block diagonal setting over that of the ip1 layer weight matrix in the fully connected setting after 9000 training iterations on CIFAR10 using cuda-convnet can be seen in Figure 4.

Original: These ratios  highlight the relationship to random matrix theory best, but here we include the variance and the singular values without taking a ratio because they may also be informative for some readers.
Revised: These ratios best highlight the relationship to random matrix theory, but here we include the variance and the singular values without taking a ratio because they may also be informative for some readers.

Original: See Figure
Revised: See Figure

Original: Figure 8 (left) compares the theoretical probability density function of the singular values of a truly random matrix with independent entries to the measured distribution ofsingular values the ip1 layer weight matrix for the FC architecture after 9000 training iterations.
Revised: Figure 8 (left) compares the theoretical probability density function of the singular values of a truly random matrix with independent entries to the measured distribution of the ip1 layer weight matrix singular values using the FC architecture after 9000 training iterations.

Original: Figure 8 (right) uses the Kullback-Leibler divergence (Appendix A) to measure the difference between the theoretical probability density function of the singular values of a truly random matrix to the measured distribution of singular values the ip1 layer weight matrix for the ( b 1 , 1) -BD architecture aftertraining iterations.
Revised: Figure 8 (right) uses the Kullback-Leibler divergence (Appendix A) to measure the difference between the theoretical prob- ability density function of the singular values of a truly random matrix to the measured distribution of singular values the ip1 layer weight matrix for the ( b 1 , 1) -BD architecture after 9000 training iterations.

Original: For b 1 = 1 , 2 , 4 , 8 , 16 , 32 , and 64 the Kullback-Leibler divergence stays between0 . 03 and0 .
Revised: For b 1 = 1 , 2 , 4 , 8 , 16 , 32 , and 64 the Kullback-Leibler divergence stays between and

Original: For comparison, the average Kullback-Leibler divergence from the discrete measured empirical spectral distribution of 164 RR (cid:62) where R is an 1024 × 64 matrix with standard normally distributed entries to the Marchenko-Pastur distribution µ 
Revised: For comparison, the average Kullback-Leibler divergence from the discrete measured empirical spectral distribution of 164 RR (cid:62) where R is an 1024 × 64 matrix with standard normally distributed entries to the Marchenko-Pastur distribution µ 0 .

Original: 0625 as in equation (1) is D KL ( 
Revised: 0625 as in equation (1) is D KL ( µ 0 . ||

Revised: λ 164 RR (cid:62) )

Original: = 0 .
Revised: = 0 .

Original: 0218 where the average is taken over over 20 initializations.
Revised: 0218 where the average is taken over over 20 initializations.

Original: At its largest discrepancy, this is 5 .times less than the measured divergence in Figure 8 (right).
Revised: At its largest discrepancy, this is 5 .times less than the measured divergence in Figure 8 (right).