Original: N EURAL N ETWORKS WITH B LOCK D IAGONAL I NNER P RODUCT L AYERS
Revised: N EURAL N ETWORKS WITH B LOCK D IAGONAL I NNER P RODUCT L AYERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Two difﬁculties continue to burden deep learning researchers and users: (1) neural networks are cumbersome tools that grow with the complexity of the learning problem, and (2) the activity of the fully connected, or inner product, layers remains mysterious.
Revised: Artiﬁcial neural networks have opened up a world of possibilities in data science and artiﬁcial intelligence, but neural networks are cumbersome tools that grow with the complexity of the learning problem.

Original: We make contributions to these two issues by considering a modiﬁed version of the fully connected layer we call a block diagonal inner product layer.
Revised: We make contributions to this issue by considering a modiﬁed version of the fully connected layer we call a block diagonal inner product layer.

Original: These modiﬁed layers have weight matrices that are block diagonal, turning a single fully connected layer into a set of densely connected neuron groups.
Revised: These modiﬁed layers have weight matrices that are block diagonal, turning a single fully connected layer into a set of densely connected neuron groups.

Original: This method condenses network storage and speeds up the run time without signiﬁcant adverse effect on the testing accuracy, thus offering a new approach to solving the ﬁrst problem. Comparing the change in variance and singular values of the weights through training in a layer when varying the number of blocks gives insight into the second problem.
Revised: This idea is a natural extension of group, or depthwise separable, convolutional layers applied to the fully connected layers.

Original: The ratio of the variance of the weights remains constant throughout training. That is, the relationship in structure is preserved in the ﬁnal parameter distribution. We observe that trained inner product layers have structure similar to that of truly random matrices with iid entries, and that each block in a  block inner product layer behaves like a smaller copy, giving a better understanding of the nature of inner product layers.
Revised: Block diagonal inner product layers can be achieved by either initializing a purely block diagonal weight matrix or by iteratively pruning off diagonal block entries. This method condenses network storage and speeds up the run time without signiﬁcant adverse effect on the testing accuracy, thus offering a new approach to improve network computation efﬁciency.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Today, it is well known that larger neural networks can better represent complex data and hence achieve higher accuracy than smaller networks (Hornik et al., 1989; Simonyan & Zisserman, 2014;
Revised: Today, it is well known that larger neural networks can better represent complex data and hence achieve higher accuracy than smaller networks (Hornik et al., 1989; Simonyan & Zisserman, 2014;

Original: Sermanet et al., 2014).
Revised: Sermanet et al., 2014).

Original: While larger networks are more capable than their smaller counterparts, their size consumes signiﬁcant storage and computational resources.
Revised: While larger networks are more capable than their smaller counterparts, their size consumes signiﬁcant storage and computational resources and memory bandwidth.

Original: Ideally, efforts to reduce memory requirements would also lessen computational demand, but often these competing interests force a trade-off.With the performance gain of large networks, we also renounce some understanding.
Revised: Ideally, efforts to reduce memory requirements would also lessen computational demand, but often these competing interests force a trade-off.

Original: The convolutional layers learn high-level features in the data; adding fully connected layers allows the network to learn non-linear combinations of these features and model more complex global patterns.

Original: While the activation of convolutional layers exhibits the features learned, the knowledge of the fully connected layers remains obscured. The fully connected layers are unwieldy and perplexing, yet they continue to be present in the most successful networks (Krizhevsky et al., 2012; Zeiler & Fergus, 2013; Simonyan & Zisserman, 2014).
Revised: The fully connected layers are unwieldy, yet they continue to be present in the most successful networks (Krizhevsky et al., 2012; Zeiler & Fergus, 2013; Simonyan & Zisserman, 2014).

Original: Our work addresses both memory and computational demand without compromise.
Revised: Our work addresses both memory and computational demand without compromise.

Original: Focusing our attention on the inner product layers, we decrease network complexity and begin to uncover the mechanism of inner product layers.
Revised: Focusing our attention on the inner product layers, we decrease network memory footprint and improve network computational demand.

Original: While larger network architectures achieve higher accuracy, there are a variety of methods to condense them without much harm to the network accuracy.
Revised: While larger network architectures achieve higher accuracy, there are a variety of methods to condense them without much harm to the network accuracy.

Original: One such technique that has gained popularity is pruning (Reed, 1993; Han et al., 2015a;b), but traditional pruning has disadvantages related to network runtime.
Revised: One such technique that has gained popularity is pruning (Reed, 1993; Han et al., 2015a;b), but traditional pruning has disadvantages related to network runtime.

Original: Most existing pruning processes signiﬁcantly slow down network training, and the ﬁnal trained network is usually slower to execute (Han et al., 2015a).
Revised: Most existing pruning processes signiﬁcantly slow down network training, and the ﬁnal trained network is usually slower to execute (Han et al., 2015a).

Original: Sparse format operations require additional overhead that can greatly slow down performance unless one prunes nearly all weight entries, which can damage network accuracy.
Revised: Sparse format operations require additional overhead that can greatly slow down performance unless one prunes nearly all weight entries, which can damage network accuracy.

Original: Localized memory access patterns can be computed faster than non-localized lookups.
Revised: Localized memory access patterns can be computed faster than non-localized lookups.

Original: By implementing block diagonal inner product layers in place of fully connected layers, we condense neural networks in a structured manner that speeds up the ﬁnal runtime and does little harm to the ﬁnal accuracy.
Revised: By implementing block diagonal inner product layers in place of fully connected layers, we condense neural networks in a structured manner that speeds up the ﬁnal runtime and does little harm to the ﬁnal accuracy.

Original: Block diagonal inner product layers can be implemented by either initializing a purely block diagonal weight matrix or by initializing a fully connected layer and focusing pruning efforts off the diagonal blocks to coax the dense weight matrix into structured sparsity.
Revised: Block diagonal inner product layers can be implemented by either initializing a purely block diagonal weight matrix or by initializing a fully connected layer and focusing pruning efforts off the diagonal blocks to coax the dense weight matrix into structured sparsity.

Original: The ﬁrst method also reduces the gradient computation time and hence the overall training time.
Revised: The ﬁrst method also reduces the gradient computation time and hence the overall training time.

Original: The latter method can improve accuracy and supports the robustness of networks to shaping .
Revised: The latter method can improve accuracy and supports the robustness of networks to shaping .

Original: That is, pruning can be used as a mapping between architectures–in particular, a mapping to more convenient architectures.
Revised: That is, pruning can be used as a mapping between architectures–in particular, a mapping to more convenient architectures.

Original: Depending on how many iterations the pruning process takes, this method may also speed up training.
Revised: Depending on how many iterations the pruning process takes, this method may also speed up training.

Original: We have converted a single fully connected layer into a group of smaller inner product learners whose combined efforts form a stronger learner, in essence boosting the layer.
Revised: We have converted a single fully connected layer into a group of smaller inner product learners whose combined efforts form a stronger learner, in essence boosting the layer.

Original: These methods also bring artiﬁcial neural networks closer to the architecture of biological mammalian brains, which have more local connectivity (Herculano-Houzel, 2012).Another link with the mammalian brain is the relationship to random matrix theory.
Revised: These methods also bring artiﬁcial neural networks closer to the architecture of biological mammalian brains, which have more local connectivity (Herculano-Houzel, 2012).

Original: In neuroscience, synaptic connectivity is often represented by a matrix with entries drawn randomly from an appropriate distribution (Rajan & Abbott, 2006; Rajan, 2010).

Original: The distribution of the singular values of a large, random matrix behaves predictably according to the Marchenko-Pastur Law (Marchenko &

Original: Pastur, 1967).

Original: We show that this distribution also represents artiﬁcial neural activity matrices well.

Original: This relationship allows us to compare the behavior of inner product layers in networks that have related structure.

Original: Looking at the distribution of an inner product layer through training, we gain insight into neural networks as random dynamical systems, thereby uncovering a piece of the inner product layer “black box”.

Original: Speciﬁcally, we observe that when varying the number of blocks in a layer, the initial ratio of the variance of the weights is preserved to ﬁrst order throughout training.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Weight pruning comes in many forms including penalty and second derivative methods, sensitivity analysis and cross validation; it can be done iteratively during training or to a trained network before reﬁning the surviving weights (Reed, 1993; Castellano et al., 1997; LeCun et al., 1990; Hassibi &
Revised: Weight pruning comes in many forms including penalty and second derivative methods, sensitivity analysis and cross validation; it can be done iteratively during training or to a trained network before reﬁning the surviving weights (Reed, 1993; Castellano et al., 1997; LeCun et al., 1990; Hassibi &

Original: Stork, 1992; Engelbrecht, 2001).
Revised: Stork, 1992; Engelbrecht, 2001).

Original: With any of these methods, the result is a sparse network that takes less storage space than its fully connected counterpart.
Revised: With any of these methods, the result is a sparse network that takes less storage space than its fully connected counterpart.

Original: Han et al.
Revised: Han et al.

Original: (2015b) iteratively prune a network using the penalty method by adding a mask that disregards pruned parameters for each weight tensor.
Revised: (2015b) iteratively prune a network using the penalty method by adding a mask that disregards pruned parameters for each weight tensor.

Original: This means that the number of required ﬂoating point operations decreases, but the number performed stays the same.
Revised: This means that the number of required ﬂoating point operations decreases, but the number performed stays the same.

Original: Furthermore, masking out updates takes additional time.
Revised: Furthermore, masking out updates takes additional time.

Original: Quantization and Huffman coding can be used to compress a trained network further (Xie & Jabri, 1992;
Revised: Quantization and Huffman coding can be used to compress a trained network further (Xie & Jabri, 1992;

Original: Han et al., 2015a).
Revised: Han et al., 2015a).

Original: Han et al.
Revised: Han et al.

Original: (2015a) report the average time spent on a forward propagation after pruning is complete and the resulting sparse layers have been converted to CSR format; for batch sizes larger than one, the sparse computations are signiﬁcantly slower than the dense calculations.
Revised: (2015a) report the average time spent on a forward propagation after pruning is complete and the resulting sparse layers have been converted to CSR format; for batch sizes larger than one, the sparse computations are signiﬁcantly slower than the dense calculations.

Original: Node pruning (He et al., 2014; Srinivas & Babu, 2015)or group lasso (Yuan & Lin, 2006; Lebedev & Lempitsky, 2016; Wen et al., 2016) could be used to speed up training and ﬁnal execution time more easily than weight pruning since node pruning preserves some structure, but drastic node pruning can harm the network accuracy.
Revised: Node pruning (He et al., 2014; Srinivas & Babu, 2015) could be used to speed up training and ﬁnal execution time more easily than weight pruning since node pruning preserves some structure, but drastic node pruning can harm the network accuracy.

Original: In practice, node pruning requires additional weight ﬁne-tuning to maintain accuracy.
Revised: In practice, node pruning requires additional weight ﬁne-tuning to maintain accuracy.

Revised: Other approaches include storing a low rank approximation for a layer’s weight matrix (Sainath et al., 2013), using speciﬁc parameter sharing mechanisms (Chen et al., 2015), training smaller models on outputs of larger models (distillation) (Hinton et al., 2014), allowing neurons to read stale gradient updates (Ho et al., 2013) and using lower precision (Vanhoucke et al., 2011; M.Courbariaux et al., 2015; Gupta et al., 2015).

Revised: Note that using lower precision or stale gradients are techniques that can be applied to any of the methods considered, including ours.

Revised: More recently, there has been momentum in the direction of structured reduction of network architecture.

Original: Sindhwani et al.
Revised: Sindhwani et al.

Original: (2015) propose structured parameter matrices characterized by low displacement rank that yield high compression rate as well as fast forward and gradient evaluation.
Revised: (2015) propose structured parameter matrices characterized by low displacement rank that yield high compression rate as well as fast forward and gradient evaluation.

Revised: Their work focuses on toeplitz-related transforms of the fully connected layer weight matrix.

Original: However speedup is generally only seen for compression of large weight matrices, and their structured sparsity methods require parameter sharing which may be undesirable in some cases.
Revised: However, speedup is generally only seen for compression of large weight matrices, and their structured sparsity methods require parameter sharing which may be undesirable in some cases.

Original: Block sparsity yields compression with parameter ﬂexibility.
Revised: Block sparsity yields compression with parameter ﬂexibility.

Original: Other approaches include storing a low rank approximation for a layer’s weight matrix (Sainath et al., 2013), using speciﬁc parameter sharing mechanisms (Chen et al., 2015), training smaller models on outputs of larger models (distillation) (Hinton et al., 2014), allowing neurons to read stale gradient updates (Ho et al., 2013) and using lower precision (Vanhoucke et al., 2011; M.Courbariaux et al., 2015; Gupta et al., 2015).

Original: Note that using lower precision or stale gradients are techniques that can be applied to any of the methods considered, including ours.

Original: Methods to condense and speedup convolutional layers, like predicting redundant parameters (Denil et al., 2013) and Group Lasso (Yuan & Lin, 2006; Lebedev & Lempitsky, 2016; Wen et al., 2016), can also be used in conjunction with our methods.
Revised: Group lasso has made contributions to the compression of convolutional layers (Yuan & Lin, 2006; Lebedev & Lempitsky, 2016; Wen et al., 2016).

Original: There is less work considering the distribution of weights in artiﬁcial neural networks.
Revised: Group lasso expands the concept of node pruning to convolutional ﬁlters.

Original: Initialization distributions to combat vanishing gradients are supported by theoretical variances for back propagation gradients under the assumption that the weights are independent, which is not valid beyond the ﬁrst iteration (Glorot & Bengio, 2010; Sun, 2015).
Revised: That is, group lasso applies L 1 norm regularization to entire ﬁlters.

Original: Random weights have been looked at as good predictors of successful network architecture (Saxe et al., 2011). More recently, N.Tishby (2017) discussed the trend of the distribution of weight updates as they relate to the Mutual Information
Revised: Likewise, group, or depthwise separable, convolutions have been used in recent CNN architectures with great success (Xie et al., 2016; Zhang et al., 2017;

Original: Plane.
Revised: Chollet, 2017; Howard et al., 2017).

Original: To the best of our knowledge, the effects of architecture on the change in distribution of the weights through training and the connection between trained inner product layer weights and random matrices have not been explored. In nuclear physics, the behavior of individual nuclei is the result of predictable forces, but the behavior of a collection of interacting particles becomes chaotic. Wigner (1995) proposes a statistical description of such systems that is still widely used for studying physical systems with particle-particle interactions, like thermal conductivity.
Revised: In group convolutions, the number of nonzero weights does not change, but the number of connections changes; a particular ﬁlter does not see all of the channels of the previous layer. Block diagonal inner product layers apply this idea of separable neuron groups to the fully connected layers, which decreases the number of weights as well as the number of connections.

Original: Wigner claimed that variation in the positions of compound nuclei resonances is well represented in terms of the eigenvalues of a large symmetric matrix with iid real entries.
Revised: Decoupling channels in images is natural because channels often carry very similar information. When converting a fully connected layer to a block diagonal inner product layer, blocks may not even see a whole channel.

Original: Brain activity behaves similarly at the macro level; in theoretical neuroscience, random matrices are used to model synaptic connections and to study brain plasticity (Sompolinsky et al., 1988; Rajan & Abbott, 2006; Rajan, 2010). Random matrices also appear in the study of large ecological systems (Allesina & Tang, 2015). Knowing that inner product layers in artiﬁcial neural networks are well modeled by random matrices opens the ﬁeld to a new range of analytical tools that may support a speciﬁc network’s robustness or plasticity.
Revised: This method transforms a fully connected layer into an ensemble of smaller fully connected neuron groups that boost the layer. Methods to condense and speedup convolutional layers like group lasso or group convolutions, can also be used in conjunction with our methods.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: We consider two methods for implementing block diagonal inner product layers:
Revised: We consider two methods for implementing block diagonal inner product layers:

Original: We initialize a layer with a purely block diagonal weight matrix and keep the number of connections constant throughout training.We initialize a fully connected layer and iteratively prune entries off the diagonal blocks to achieve a block substructure.
Revised: We initialize a layer with a purely block diagonal weight matrix and keep the number of connections constant throughout training.

Original: Within a layer, all blocks have the same size.
Revised: Within a layer, all blocks have the same size.

Original: Method 2 is accomplished in three phases: a dense phase, an iterative pruning phase and a block diagonal phase.
Revised: Method 2 is accomplished in three phases: a dense phase, an iterative pruning phase and a block diagonal phase.

Original: In the dense phase a fully connected layer is initialized and trained in the standard way.
Revised: In the dense phase a fully connected layer is initialized and trained in the standard way.

Original: During the iterative pruning phase, focused pruning is applied to entries off the diagonal blocks using the weight decay method with L 1 -norm.
Revised: During the iterative pruning phase, focused pruning is applied to entries off the diagonal blocks using the weight decay method with L 1 -norm.

Original: That is, if W is the weight matrix for a fully connected layer we wish to push toward block diagonal, we add α (cid:80) i,j | 1 i,j W i,j | to the loss function during the iterative pruning phase, where α is a tuning parameter and 1 i,j indicates whether W i,j is off the diagonal blocks in W .
Revised: That is, if W is the weight matrix for a fully connected layer we wish to push toward block diagonal, we add [Equation] to the loss function during the iterative pruning phase, where α is a tuning parameter and 1 i,j indicates whether W i,j is off the diagonal blocks in W .

Revised: The frequency of regularization and pruning during this phase are additional hyperparameters.

Original: During this phase, masking out updates for pruned entries is more efﬁcient than maintaining sparse format.
Revised: During this phase, masking out updates for pruned entries is more efﬁcient than maintaining sparse format.

Original: When pruning is complete, to maximize speedup it is best to reformat the weight matrix once such that the blocks are condensed and adjacent in memory.
Revised: When pruning is complete, to maximize speedup it is best to reformat the weight matrix once such that the blocks are condensed and adjacent in memory.

Original: 1 Batched smaller dense calculations for the blocks use cuBLAS strided batched multiplication (Nickolls et al., 2008).
Revised: 1 Batched smaller dense calculations for the blocks use cuBLAS strided batched multiplication (Nickolls et al., 2008).

Original: There is a lot of ﬂexibility in method 2 that can be tuned for speciﬁc user needs.
Revised: There is a lot of ﬂexibility in method 2 that can be tuned for speciﬁc user needs.

Original: More pruning iterations may increase the total training time but can yield higher accuracy and reduce overﬁtting.
Revised: More pruning iterations may increase the total training time but can yield higher accuracy and reduce overﬁtting.

Original: It should be mentioned that as the number of blocks increases, the overhead to perform cuBLAS strided batched multiplication can become noticeable; this library is not yet well optimized for performing many small matrix products (Masliah et al., 2016).

Original: However, with specialized batched multiplications for many small matrices, Jhurani & Mullowney (2015) attain up to 6 fold speedup.

Original:  E XPERIMENTS : SPEEDUP AND ACCURACY
Revised:  E XPERIMENTS : SPEEDUP AND ACCURACY

Original: Our goal is to reduce memory storage of the inner product layers while maintaining or reducing the ﬁnal execution time of the network with minimal loss in accuracy.
Revised: Our goal is to reduce memory storage of the inner product layers while maintaining or reducing the ﬁnal execution time of the network with minimal loss in accuracy.

Original: We will also see the reduction of total training time in some cases.
Revised: We will also see the reduction of total training time in some cases.

Original: To test this, we ran experiments on the MNIST (LeCun et al.)
Revised: To test this, we ran experiments on the MNIST (LeCun et al.),

Original: and CIFAR10 (Krizhevsky, 2009)  datasets.
Revised: CIFAR10 (Krizhevsky, 2009) and ImageNet (Russakovsky et al., 2015) datasets.

Original: All experiments are run on the Bridges’ NVIDIA P GPUs through the Pittsburgh Supercomputing Center.
Revised: All experiments are run on the Bridges’ NVIDIA P100 GPUs through the Pittsburgh Supercomputing Center.

Original: We implement our work in Caffe (Jia et al., 2014) which provides suggested network architectures for these datasets.
Revised: We implement our work in Caffe (Jia et al., 2014) which provides suggested network architectures for these datasets.

Original: Training is done with batched gradient descent using the cross-entropy loss function on the softmax of the output layer.
Revised: Training is done with batched gradient descent using the cross-entropy loss function on the softmax of the output layer.

Original: We report the forward time per inner product layer when the layers are purely block diagonal; this time only measures the matrix multiplication in the forward step and does not include the time to layers, one should alter the output format of the previous layer and the expected input format of the following layer accordingly, in particular to row major ordering.
Revised: We report the forward time per inner product layer when the layers are purely block diagonal; this time only measures the matrix multiplication in the forward step and does not include the time to prune.

Original: prune. We compare these times to the runtime of sparse matrix multiplication with random entries in CSR format using cuSPARSE (Nickolls et al., 2008).
Revised: We compare these times to the runtime of sparse matrix multiplication with random entries in CSR format using cuSPARSE (Nickolls et al., 2008).

Original: We also report the combined forward and backward time to do the three matrix products involved in gradient descent training when the layers are purely block diagonal.
Revised: We also report the combined forward and backward time to do the three matrix products involved in gradient descent training when the layers are purely block diagonal.

Original: For brevity we refer to the block diagonal method as ( b 1 , . . .
Revised: For brevity we refer to the block diagonal method as ( b 1 , . . .

Original: , b n ) -BD; b i = 1 indicates that layer i is fully connected.
Revised: , b n ) -BD; b i = 1 indicates that layer i is fully connected.

Original: FC is short for all inner product layers being fully connected.
Revised: FC is short for all inner product layers being fully connected.

Original: We refer to networks that differ only by the number of blocks in a single inner product layer as sister networks.
Revised: We refer to networks that differ only by the number of blocks in a single inner product layer as sister networks.

Original: 4. MNIST
Revised: Figure 1 shows the speedup when performing matrix multiplication using an n × n weight matrix and batch size 100 when the weight matrix is purely block diagonal. The speedup when performing only the forward-pass matrix product is shown in the left pane, and the speedup when performing all gradient descent products is shown in the right pane. As the number of blocks increases, the overhead to perform cuBLAS strided batched multiplication can become noticeable; this library is not yet well optimized for performing many small matrix products (Masliah et al., 2016). However, with specialized batched multiplications for many small matrices, Jhurani & Mullowney (2015) attain up to 6 fold speedup.

Revised: Zhang et al.

Revised: (2017) note that when multiple group convolutions are stacked together, outputs from a particular group only see the inputs within the group.

Revised: They suggest that this property blocks information ﬂow between channel groups and weakens representation.

Revised: To correct for this, they suggest dividing the channels in each group into subgroups, and feeding each group in the next layer with a mixture of subgroups from various parent groups in the previous layer.

Revised: One drawback of applying this approach to block inner product layers is that it either requires moving entries in memory or doing more, smaller matrix products.

Revised: As mentioned, performing many small matrix products can hurt performance; this can be seen in Figure 1.

Revised: Using pruning to achieve the block diagonal structure, as in method 2, also addresses this issue.

Revised: Pruning does add some work to the training iterations, but does not add work to the ﬁnal execution of the trained network unlike the shufﬂing method found in ShufﬂeNet (Zhang et al., 2017).

Revised: After pruning is complete, the learned weights are the result of a more complete picture; while the information ﬂow has been constrained, it is preserved like a ghost in the remaining weights.

Revised: Another alternative to the ﬁxed channel shufﬂe offered in ShufﬂeNet, is a random shufﬂe of the whole blocks similar to the “random sparse convolution” layer in the CNN library cuda-convnet (Krizhevsky, 2012a).

Revised: We will compare results using all of these methods.

Original: We experimented on the MNIST dataset with the LeNet-5 framework (LeCun et al., 1998).
Revised: We experimented on the MNIST dataset with the LeNet-5 framework (LeCun et al., 1998).

Original: LeNethas two convolutional layers with pooling followed by two inner product layers with ReLU activation.
Revised: LeNethas two convolutional layers with pooling followed by two inner product layers with ReLU activation.

Original: The ﬁrst inner product layer, ip1, has a 500 × 800 weight matrix, and the output inner product layer, ip2, has a 10 × 500 weight matrix.
Revised: The ﬁrst inner product layer, ip1, has a 500 × 800 weight matrix, and the output inner product layer, ip2, has a 10 × 500 weight matrix.

Original: We initialize the inner product layers using the Xavier weight ﬁller (Glorot & Bengio, 2010), which samples a uniform distribution with variance 1 /n in , where n in is the number of neurons feeding into a node.
Revised: We initialize the inner product layers using the Xavier weight ﬁller (Glorot & Bengio, 2010), which samples a uniform distribution with variance 1 /n in , where n in is the number of neurons feeding into a node.

Original: We used a training batch size of 64 and all other hyperparameters provided by Caffe are left unchanged.
Revised: We used a training batch size of 64 and all other hyperparameters provided by Caffe are left unchanged.

Original: Figure 1 shows time and accuracy results for block diagonal method 1 without pruning.
Revised: Figure 2 shows time and accuracy results for block diagonal method 1 without pruning.

Original: The ( b 1 , b 2 ) -
Revised: The ( b 1 , b 2 ) -

Original: BD architecture has (800 × 500) /b
Revised: BD architecture has (800 × 500) /b

Original: 1 + (500 × 10) /b 2 nonzero weights across both inner product layers.
Revised: 1 + (500 × 10) /b 2 nonzero weights across both inner product layers.

Original: The points at which the forward sparse and forward block curves meet in Figure 1 (left) indicate the fully connected dense forward runtimes for each layer.
Revised: The points at which the forward sparse and forward block curves meet in Figure 2 (left) indicate the fully connected dense forward runtimes for each layer; these are made clearer with dotted, black, vertical lines.

Original: There is ≥ 1 . 4 × speedup for b 1 ≤ 50 , or 8000 nonzero entries, when timing both forward and backward matrix products, and 1 . 6 × speedup when b 1 = 100 , or 4000 nonzero entries, in the forward only case.
Revised: There is ≥ 1 . 4 × speedup for b 1 ≤ 50 , or 8000 nonzero entries, when timing both forward and backward matrix products, and 1 . 6 × speedup when b 1 = 100 , ornonzero entries, in the forward only case.

Original: Sparse format times are slow until there are less than 50 nonzero entries.
Revised: Sparse format times are slow until there are less than 50 nonzero entries.

Original: Figure 1 (right) shows a slight decline in accuracy as the number of nonzero entries decreases.
Revised: Figure 2 (right) shows a slight decline in accuracy as the number of nonzero entries decreases.

Original: FC achieves a ﬁnal accuracy of 99.11 % .
Revised: FC achieves a ﬁnal accuracy of 99.11 % .

Original: Without pruning, (100 , 10) -BD has a ﬁnal accuracy of 98.52 % .
Revised: Without pruning, (100 , 10) -BD has a ﬁnal accuracy of 98.52 % .

Original: In all cases testing accuracy remains within 1 % of FC accuracy.
Revised: In all cases testing accuracy remains within 1 % of FC accuracy.

Original: Using traditional iterative pruning with L 2 regularization, as suggested in (Han et al., 2015b), pruning every ﬁfth iteration until 4000 and 500 nonzero entries survived in ip1 and ip2 respectively gave an accuracy of 98.55 % , but the forward multiplication was more than 8 times slower than the dense fully connected case.
Revised: Using traditional iterative pruning with L 2 regularization, as suggested in (Han et al., 2015b), pruning every ﬁfth iteration until 4000 and 500 nonzero entries survived in ip1 and ip2 respectively gave an accuracy of 98.55 % , but the forward multiplication was more than 8 times slower than the dense fully connected case.

Original: Implementing (100 , 10) -BD method 2 with pruning using 15 dense iterations and 350 pruning iterations gave a ﬁnal accuracy of 98.65 % .
Revised: Implementing (100 , 10) -BD method 2 with pruning using 15 dense iterations and 350 pruning iterations gave a ﬁnal accuracy of 98.65 % .

Original: Thus we saw no great beneﬁt to using pruning over initializing pure block diagonal inner product layers for the MNIST dataset.
Revised: Thus we saw no great beneﬁt to using pruning over initializing pure block diagonal inner product layers for the MNIST dataset.

Revised: In this setting, using neither random shufﬂing of whole blocks in ip1, nor ﬁxed sub-block shufﬂing in the manner of Zhang et al.

Revised: (2017) delivered any noticeable improvement.

Original: In (Sindhwani et al., 2015), Toeplitz (3) has error rate 2.09 % using a single hidden layer net with 1000 hidden nodes on MNIST.
Revised: In (Sindhwani et al., 2015), Toeplitz (3) has error rate 2.09 % using a single hidden layer net with 1000 hidden nodes on MNIST.

Original: This method yields 63.32 fold compression over the fully connected setting.
Revised: This method yields 63.32 fold compression over the fully connected setting.

Original: However from their Figure 3, this slows down theforward pass by around 1 .
Revised: However from their Figure 3, this slows down the pass by around 1 .

Original: 5 × and thebackward pass by around5 .
Revised: 5 × and the pass by around

Original: 5 ×.
Revised: 5 ×

Original: Our netwith one hidden layer, 980 hidden nodes using (49 , 1) -BD on MNIST has 29.43 fold compressionand error rate 4.37 % using method 2 with pruning.
Revised: Our net one hidden layer 980 hidden nodes using 43 fold compression method hidden only the

Original: Our speedup is 1.53 for forward only and 1.04 when combining the forward and backward runtime.
Revised: As mentioned we left suggested hyperpameters by Caffe unchanged for hyperparameters like learning rate, momentum and weight decay.

Original: This is curious because the blocks of neurons in the hidden layer can only see a portion of the input images.
Revised: In our experiments on the MNIST dataset we performed only manual hyperparameter tuning of new hyperparameters introduced by method 2 like the coefﬁcient of the new regularization term (see equation 1) and the frequency of pruning iterations.

Original: 4. CIFAR 
Revised: 4. CIFAR

Original: We experimented on the CIFAR10 dataset with Krizhevsky’s cuda-convnet (2012).
Revised: We experimented on the CIFAR10 dataset with Krizhevsky’s cuda-convnet (2012b).

Original: Cuda-convnet has three convolutional layers with ReLu activation and pooling, followed by two fully connected layers with no activation.
Revised: Cuda-convnet has three convolutional layers with ReLu activation and pooling, followed by two fully connected layers with no activation.

Original: The ﬁrst inner product layer, ip1, has a 64 × 1024 weight matrix, and the second has a 10 × 64 weight matrix.
Revised: The ﬁrst inner product layer, ip1, has a 64 × 1024 weight matrix, and the second has a 10 × 64 weight matrix.

Original: In all methods the inner product layer weights are initialized using a Gaussian ﬁller with standard deviation 0.1, as suggested by Caffe.
Revised: In all methods the inner product layer weights are initialized using a Gaussian ﬁller with standard deviation 0.1, as suggested by Caffe.

Original: We used a training batch size of 100, and all other hyperparameters provided by Caffe’s “quick” model are left unchanged.
Revised: We used a training batch size of 100, and all other hyperparameters provided by Caffe’s “quick” model are left unchanged.

Original: Caffe reports ≈ 75% accuracy with the FC architecture.
Revised: Caffe reports ≈ 75% accuracy with the FC architecture.

Original: Figure 2 shows time and accuracy results for the block diagonal method 1 without pruning.
Revised: Figure 3 shows time and accuracy results for the block diagonal method 1 without pruning.

Original: The ( b 1 , b 2 ) -BD architecture has (1024 × 64) /b 1 + (64 × 10) /b 2 nonzero entries across both inner product layers.
Revised: The ( b 1 , b 2 ) -BD architecture has (1024 × 64) /b 1 + (64 × 10) /b 2 nonzero entries across both inner product layers.

Original: In the ip1 layer, there is  speedup for b 1 ≤ 32 , or 2048 nonzero entries, when timing both forward and backward matrix products, and  b 1 ≤ 64 , or 1024 nonzero entries, in the forward only case.
Revised: In the ip1 layer, there is ≥ 1 . 26 × speedup for b 1 ≤ 32 , or 2048 nonzero entries, when timing both forward and backward matrix products, and ≥ 1 . 65 × speedup for b 1 ≤ 64 , or 1024 nonzero entries, in the forward only case.

Original: Again, sparse format performs poorly until there are less than 50 nonzero entries.
Revised: Again, sparse format performs poorly until there are less than 50 nonzero entries.

Original: Figure 
Revised: Figure 3 (right) shows a decline in accuracy as the blocks increase. FC achieves a ﬁnal accuracy of 76.29 % . Without pruning, (64 , 2) -BD has a ﬁnal accuracy of 72.49 % .

Original: Using traditional iterative pruning with L 2 regularization pruning every ﬁfth iteration untiland 320 nonzero entries survived in the ﬁnal two inner product layers gave an accuracy of 75.18 % , but again the forward multiplication was more than 8 times slower than the dense fully connected computation.
Revised: Using traditional iterative pruning with L 2 regularization pruning every ﬁfth iteration untiland 320 nonzero entries survived in the ﬁnal two inner product layers gave an accuracy of 75.18 % , but again the forward multiplication was more than 8 times slower than the dense fully connected computation.

Original: On the other hand, implementing (64 , 2) -BD method 2 with pruning, which has corresponding numbers of nonzero entries, with 500 dense iterations and 1000 pruning iterations gave a ﬁnal accuracy of 74.81 % .
Revised: On the other hand, implementing (64 , 2) -BD method 2 with pruning, which has corresponding numbers of nonzero entries, with 500 dense iterations and 1000 pruning iterations gave a ﬁnal accuracy of 74.81 % .

Original: This is a 35.97 fold compression of the inner product layer parameters with only a 1.5 % drop in accuracy.
Revised: This is a 35.97 fold compression of the inner product layer parameters with only a 1.5 % drop in accuracy.

Original: The total forward runtime of  (64 , 2) -BD is 1.6 times faster than
Revised: The total forward runtime of ip1 and ip2 in (64 , 2) -BD is 1.

Original:  FC.
Revised: times faster than in FC.

Original: To achieve comparable speed with sparse format we used traditional iterative pruning to leaveand 40 nonzero entries inthe ﬁnal inner productlayers givingan accuracy of 73.01 % .
Revised: To achieve comparable speed with sparse format we used traditional itera- tive pruning to leave 37 and 40 nonzero entries in inner product giving of 73.01 % .

Original: Thus implementing block diagonal layers with pruning yields comparable accuracy and memory condensation to traditional iterative pruning with faster ﬁnal execution time.
Revised: Thus implementing block diagonal layers with pruning yields comparable accuracy and memory condensation to traditional iterative pruning with faster ﬁnal execution time.

Original: Whole node pruning decreases the accuracy more than corresponding reductions in the block diagonal setting.
Revised: Whole node pruning decreases the accuracy more than corresponding reductions in the block diagonal setting.

Original: Node pruning until ip1 had only 2 outputs, i.e. a 1024 × 2 weight matrix, and ip2 had a 2 × 10 weight matrix for a total of 2068 weights between the two layers gave a ﬁnal accuracy 0f 59.67 % .
Revised: Node pruning until ip1 had only 2 outputs, i.e. a 1024 × 2 weight matrix, and ip2 had a 2 × 10 weight matrix for a total of 2068 weights between the two layers gave a ﬁnal accuracy of 59.67 % .

Original: (64,2)-BD has a total of 1344 weights between the two inner product layers and had a ﬁnal accuracy 15.14 % higher with pruning.
Revised: (64,2)-BD has a total of 1344 weights between the two inner product layers and had a ﬁnal accuracy 15.14 % higher with pruning.

Original: The ﬁnal accuracy on an independent test set was 76.29 % on CIFAR10 using the FC net while the ﬁnal accuracy on the training set itself was 83.32 % .
Revised: The ﬁnal accuracy on an independent test set was 76.29 % on CIFAR10 using the FC net while the ﬁnal accuracy on the training set itself was 83.32 % .

Original: Using the (64,2)-BD net without pruning, the accuracy on an independent test set was 72.49 % , but on the training set was 75.63 % .
Revised: Using the (64,2)-BD net without pruning, the accuracy on an independent test set was 72.49 % , but on the training set was 75.63 % .

Original: With pruning, the accuracy of (64,2)-BD on an independent test set was 74.81 % , but on the training set was 76.85 % .
Revised: With pruning, the accuracy of (64,2)-BD on an independent test set was 74.81 % , but on the training set was 76.85 % .

Original: Both block diagonal methods decrease overﬁtting; the block diagonal method with pruning decreases overﬁtting slightly more.R ANDOM MATRIX THEORY OBSERVATIONS
Revised: Both block diagonal methods decrease overﬁtting; the block diagonal method with pruning decreases overﬁtting slightly more.

Original: The Marchenko-Pastur distribution describes the asymptotic behavior of the singular values of large random matrices with iid entries (1967).

Original: Let X be an m × n matrix with iid entries x ij such that

Original: E[ x ij ] = 0 and Var[ x ij ] = σ 2 .

Original: The Marchenko-Pastur theorem states that as n, m → ∞ such that m/n → y > 0 , with probability 1 the empirical spectral distribution of 1 n XX (cid:62) converges in distribution to the density

Original: [Equation]

Original: otherwise (1) with point mass 1 − 1 /y at the origin if y > 1 where a = σ 2 (1 − y 2 ) and b = σ 2 (1 + y 2 ) . Network weights not independent [Equation] in an iteration, where λ is the learning rate, b is the batch size, L is the loss function and x i is sampled from the data distribution.
Revised: We saw no signiﬁcant affect on the ﬁnal accuracy after 9000 iterations when using random shufﬂing of the ip2 layer blocks during training.

Original: One can easily verify,

Original: [Equation] using estimators for the right side of the equation, where a simple Hadamard (Schur) product provides a good estimate for the second term. In our experience, the covariance quickly became the dominating term. However, for a large enough weight matrix, the singular values of an inner product layer weight matrix behave according to the Marchenko-Pastur distribution even after thousands of training iterations.
Revised: Similarly, performing ﬁxed sub-block shufﬂe as suggested by Zhang et al.

Original: That is, after thousands of correlated updates, the weight matrix behaves like a matrix with iid entries. The assumption of independence fails at the micro level when calculating the change in variance of the weights, but is accurate at the macro level when considering the behavior ofthe weight matrix as an operator. In this section we discuss only the ﬁrst method for implementing block diagonal layers without pruning. While a relationship between fully connected layer weights and the corresponding block diagonal layer weights in trained sister networks is not evident from equation (2), indeed a relationship can be seen in the singular values of the weight matrices, and, in particular, in the change in the variance of the weight matrix entries throughout training.
Revised: (2017) on the (2 , 2) -BD architecture using method 1 where shufﬂing is applied to sub-blocks of layer ip2 did not yield improvement over (2 , 2) -BD without ﬁxed sub-block shufﬂing.

Original: The initialization ratio of variances incorresponding layer weight matrices of sister networks persists through thousands of training iterations. This ﬁnding may provide a good mechanism for examining related network architectures and may support claims about a network’s malleability or ﬁtness. 5. MNIST As mentioned, inour experiments on the MNIST dataset the  inner product layer weights are initialized using the Xavier algorithm
Revised: These methods may be useful in networks with many stacked block inner product layers or networks with sparse convolution as a result of group lasso or group convolution, but saw no beneﬁt in the cuda-convnet architecture with two inner product layers.

Original: (Glorot & Bengio, 2010). Thus the initialization variance of the weights in ip1 is b 1 / 800 in the block diagonal case, where b 1 is the number of blocks, and the ratio of this variance over that of the fully connected case at initialization is b 1 . Figure 3 (left) shows that this ratio is a good ﬁrst order estimate for the ﬁnal ratio after 10000 iterations. The ﬁnal ratios are 5 .
Revised: Using method 2 to achieve a block substructure with pruning did yield improvement over method 1.

Original: 32 and 96 . for b 1 = 5 , 10 , 20 , 50 , and 100 respectively. We can see that the relationship deteriorates as the number of blocks increases. This holds for any random input order and when the sigmoid activation function is used for layer ip1 as long as the activation function is consistent across sister networks.
Revised: 4. I MAGE N ET : INCOMPLETE

Original: [-When using the sigmoid activation function, the ratio seemed to deteriorate less quickly; e.g. the ﬁnal variance ratio was 101 . 
 for b 1 = 100 . Figure 3 (right) compares the singular values of the ip1 layer weight matrix for varying numbers of blocks after 10000 training iterations-] to the singular values of a truly random 800 × 500 matrix initialized with variance 6 . 6 × 10 − 4 , the ﬁnal variance of the FC ip1 layer weights.
Revised: [-When using the sigmoid activation function, the ratio seemed to deteriorate less quickly; e.g. the ﬁnal variance ratio was 101 . 
 for b 1 = 100 . Figure 3 (right) compares the singular values of the ip1 layer weight matrix for varying numbers of blocks after 10000 training iterations-]In this section we will put accuracy results when we are able to get them.

Original: The singular values of the ip1 weight matrix follow the curve that the singular values of the truly random matrix create with some error in the largest and smallest singular values. We believe the disparity in the extreme singular values is the result of over training (See Appendix B.1); using FC, the accuracy reaches 98.28 % by iteration 1000. In the block diagonal method, weaggregate the singular values of each block and sort them to create the corresponding curves in Figure 3 (right). The individual block spectral distributions are identical to each other.
Revised: Currently we are unable to run ImageNet experiments using AlexNet because we need more memory on the Bridges Supercomputer.

Original: The decrease in matrix size by a factor of bcancels with the increase in variance by a factor of b 1 in equation (1), resulting in singular values that all follow the same curve.

Original: Thus, the ratio of singular values of the trained ip1 layer weight matrix over the singular values of the truly random matrix is ≈ 1 . Additional supporting ﬁgures comparing the PDFs of the singular values and the Kullback-Leibler divergence can be found in Appendix B.1. If we make ip2 a BD layer as well, the change in variance in ip1 sees minimal effect. For b 1 = 1 , 2 , 5 , 10 , 50 , 100 and b 2 = 1 , 2 , 5 , 10 , the ﬁnal variance in the ip1 layer weights using ( b 1 , b 2 )
Revised: We have placed a request for more memory and we are waiting for that request to be granted.

Original: -BD over the ﬁnal variance in the ip1 layer weights using ( b 1 , 1) -BD is ≈ 1 with error ≤ 0 .
Revised: We experimented on the ImageNet dataset with the AlexNet framework (Krizhevsky et al., 2012).

Original: 5. CIFAR
Revised: AlexNet has ﬁve convolutional layers followed by three inner product layers with ReLU and dropout activation.

Original: In our experiments on CIFAR10, the ﬁrst inner product layer weights are initialized using a Gaussian ﬁller with standard deviation 0.1. Thus the ratio of variance of the weights in ip1 in the block diagonal case over that of the fully connected case at initialization is 1. Figure 4 (left) indicates that this ratio is a good ﬁrst order estimate for the ﬁnal ratio after 9000 iterations at which time the ratios are 1 .
Revised: The ﬁrst inner product layer, ip1, has a 4096 × 9216 weight matrix, the second inner product layer, ip2, has a 4096 × 4096 weight matrix, and the output inner product layer, ip3, has a 1000 × 4096 weight matrix. We initialize the inner product layers using a gaussian weight ﬁller.

Original: 02 , 1 . 01 , 1 . 11 , 1 . 21 and 1 . 5 for b 1 = 2 , 4 , 8 , 16 , 32 , and 64 respectively. Again, the relationship deteriorates as the number of blocks grows, and this holds for any random input order. We compared the singular values of the weight matrix in layer ip1 for varying numbers of blocks after 9000 training iterations to the singular values of a truly random 1024 × 64 matrix initialized with variance 7 × 10 − 3 , the ﬁnal variance of the FC ip1 layer weights after training.
Revised: We used a training batch size of 256 and all other hyperparameters provided by Caffe are left unchanged.

Original: The singular values of the fully connected ip1 layer weight matrix follow the curve that the singular values of the truly random matrix create. As in the MNIST experiments, we aggregate the singular values of each block in the block diagonal methodand sort them. In this case, variance of the ip1 layer weights remains relatively constant for varying numbers of blocks, so, by equation (1), the decrease in matrix size by a factor of b 1 must appear in the ratio of singular values as 1 / √ b 1 . Figure 4 (right) shows this relationship.
Revised: Figure 4 shows time and accuracy results for block diagonal method 1 without pruning.

Original: The ratio of singular values of the trained weight matrix in layer ip1 over the singular values of the random matrix is ≈ 1 / √ b 1 for b 1 blocks with deterioration as b  1 grows.
Revised: The ( b 1 , b 2 , b 3 ) -BD architecture has (4096 × 9216) /b 1 +(4096 × 4096) /b 2 +(1000 × 4096) /b 3 nonzero weights across all inner product layers.

Original: Additional supporting ﬁgures comparing the PDFs of the singular values and the Kullback-Leibler divergence can be found in Appendix B.2. If in addition we make the ip2 a BD layer, again, the change in variance in ip1 sees minimal effect. For b 1 = 1 , 2 , 4 , 8 , 16 , 32 , 64 , the ﬁnal variance in the ip1 layer weights using the ( b 1 , 2) -BD method over the ﬁnal variance in the ip1 layer weights using the ( b 1 , 1) -BD method is ≈ 1 with error ≤ 0 . 03 .
Revised: The points at which the forward sparse and forward block curves meet in Figure 4 (left) indicate the fully connected dense forward runtimes for each layer.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have shown that block diagonal inner product layers can reduce network size, training time and ﬁnal execution time without signiﬁcant harm to the network performance.We have also shown that random matrix theory gives informative results about relationships in network structure that are preserved through training.
Revised: We have shown that block diagonal inner product layers can reduce network size, training time and ﬁnal execution time without signiﬁcant harm to the network performance.

Original: While traditional iterative pruning can reduce storage, the random indices of the surviving weights make sparse computation inefﬁcient, slowing down the training and ﬁnal execution time of the network.
Revised: While traditional iterative pruning can reduce storage, the random indices of the surviving weights make sparse computation inefﬁcient, slowing down the training and ﬁnal execution time of the network.

Original: Our block diagonal methods address this inefﬁciency by conﬁning dense regions to blocks along the diagonal.
Revised: Our block diagonal methods address this inefﬁciency by conﬁning dense regions to blocks along the diagonal.

Original: Without pruning, block diagonal method 1 allows for faster training time.
Revised: Without pruning, block diagonal method 1 allows for faster training time.

Original: Method 2 preserves the learning with focused, structured pruning that reduces computation for speedup during execution.
Revised: Method 2 preserves the learning with focused, structured pruning that reduces computation for speedup during execution.

Original: In our experiments, method 2 saw higher accuracy than the purely block diagonal method for the more complex learning problem, CIFAR10; however, the increase in accuracy came in exchange for slightly more time to train the network.
Revised: In our experiments, method 2 saw higher accuracy than the purely block diagonal method for the more complex learning problem, CIFAR10; however, the increase in accuracy came in exchange for slightly more time to train the network.

Original: There is great deal of ﬂexibility in our block diagonal methods that can be tuned to an individual problem.
Revised: There is great deal of ﬂexibility in our block diagonal methods that can be tuned to an individual problem.

Original: These methods may also make larger network architectures more feasible to train and use since they convert a fully connected layer into a collection of smaller inner product learners working jointly to form a stronger learner.
Revised: These methods may also make larger network architectures more feasible to train and use since they convert a fully connected layer into a collection of smaller inner product learners working jointly to form a stronger learner.

Revised: In particular, GPU memory constraints become less constricting.

Original: There is a lot of room for additional speedup with block diagonal layers.
Revised: There is a lot of room for additional speedup with block diagonal layers.

Original: Dependency between layers poses a noteworthy bottleneck in network parallelization.
Revised: Dependency between layers poses a noteworthy bottleneck in network parallelization.

Original: With structured sparsity like ours, one no longer needs a full barrier between layers.
Revised: With structured sparsity like ours, one no longer needs a full barrier between layers.

Original: Additional speedup would be seen in software optimized to support weight matrices with organized sparse form, such as blocks, rather than being optimized for dense matrices.
Revised: Additional speedup would be seen in software optimized to support weight matrices with organized sparse form, such as blocks, rather than being optimized for dense matrices.

Original: For example, for many small blocks, one can reach up to 6 fold speedup with specialized batched matrix multiplication (Jhurani & Mullowney, 2015).
Revised: For example, for many small blocks, one can reach up to 6 fold speedup with specialized batched matrix multiplication (Jhurani & Mullowney, 2015).

Original: Hardware has been developing to better support sparse operations.
Revised: Hardware has been developing to better support sparse operations.

Original: Block format may be especially suitable for training on evolving architectures such as neuromorphic systems.
Revised: Block format may be especially suitable for training on evolving architectures such as neuromorphic systems.

Original: These systems, which are far more efﬁcient than GPUs at simulating mammalian brains, have a pronounced 2-D structure and are ill-suited to large dense matrix calculations (Merolla et al., 2014; Boahen, 2014). Comparing block diagonal and fully connected layers in sister networks highlighted that the training process for neural networks can be viewed as a random dynamical system.
Revised: These systems, which are far more efﬁcient than GPUs at simulating mammalian brains, have a pronounced 2-D structure and are ill-suited to large dense matrix calculations (Merolla et al., 2014; Boahen, 2014)

Original: The relationship in the change in variance through training between the block diagonal and fully connected case suggests an invariant pattern that may help evaluate network ﬁtness.

Original: We emphasize that this is a nontrivial relationship surviving various datasets, network architectures, and activation functions.

Original: The group behavior is that of a random matrix with independent entries, but the individual weight updates have complex dependencies.

Original: Similar random activity occurs in the mammalian brain and suggests looking at random matrix theory to support a network’s plasticity or robustness.

Original: This could also be a catalyst for considering optimal distributions for a trained network.

Original: Random matrix theory has been indispensable to the advancement of nuclear physics, quantum physics, neuroscience, and ecology, and has the potential to elevate artiﬁcial neural network analysis in the same manner.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: All experiments are run on the Bridges’ NVIDIA P100 GPUs through the Pittsburgh Supercomputing Center.A PPENDIX A K ULLBACK -L EIBLER DIVERGENCE
Revised: All experiments are run on the Bridges’ NVIDIA P100 GPUs through the Pittsburgh Supercomputing Center.

Original: The Kullback-Leibler divergence (also called relative entropy) is a measure of how one probability distribution diverges from a second probability distribution (Kullback & Leibler, 2015).

Original: The

Original: Kullback-Leibler divergence from Q to P , denoted D KL ( P || Q ) , is the information gain if P is used instead of Q , or the amount of information lost when Q is used to approximate P .

Original: [Equation]

Original: With a Kullback-Leibler divergence of 0 , one can expect similar, if not identical, behavior of the two different distributions.

Original: B R ANDOM MATRIX THEORY ADDITIONAL SUPPORTING MATERIAL B. MNIST

Original: The ratio of the variance and singular values of the ip1 layer weight matrix in block diagonal setting over that of the ip1 layer weight matrix in the fully connected setting after 10000 training iterations on the MNIST dataset using Lenet-5 framework can be seen in Figure 3.

Original: These ratios best highlight the relationship to random matrix theory, but here we include the variance and the singular values without taking a ratio because they may also be informative for some readers.

Original: See Figure 5.

Original: Figure 6 (left) compares the theoretical probability density function of the singular values of a truly random matrix with independent entries to the measured distribution of the ip1 layer weight matrix singular values for the FC architecture after 10000 training iterations.

Original: Figure 6 (right) uses the

Original: Kullback-Leibler divergence (Appendix A) to measure the difference between the theoretical probability density function of the singular values of a truly random matrix to the measured distribution of the ip1 layer weight matrix singular values the ( b 1 , 1) -BD architecture training iterations.

Original: For b 1 = 1 , 5 , 10 , 20 , 50 , and 100 the Kullback-Leibler divergence stays between 0 .and 0 .

Original: 081 with the highest values when b 1 = 1 , i.e. the FC case.

Original: One can see from Figures 3 (right) and 5 (right) that the disparity in the larger singular values decreases as b 1 increases.

Original: For comparison, the average Kullback-Leibler divergence from the discrete measured empirical spectral distribution of 1500 RR (cid:62) where R is an 800 × 500 matrix with standard normally distributed entries to the Marchenko-Pastur distribution µ 0 .

Original: 625 as in equation (1) is D KL ( µ 0 .

Original: 625 ||

Original: λ 1500 RR (cid:62) )

Original: = 0 .

Original: 014 where the average is taken over over 20 initializations.

Original: This divergence is the same magnitude as the measured divergence in Figure 6 (right).

Original: It should be mentioned that D KL ( λ 1500 W b 1 W (cid:62) b 1 || µ 0 . 625 ) is inﬁnite because of the difference in support, but this disparity in support reduces as b 1 increases.

Original: This disparity may be incurred by the relatively small matrix dimensions, as asymptotic random matrix theory often does not represent the extreme singular values well in the ﬁnite case (Rudelson & Vershynin, 2010), but is more likely caused by overtraining.

Original: Using FC, the accuracy reaches 98.28 % by iteration 1000.

Original: After 1000 iterations, the ratio of the largest singular value of the trained weight matrix in layer ip1 over the largest singular value of a truly random matrix with the same dimensions and variance is 1.16.

Original: Figure(right) shows that this ratio is 1.44 after the full 10000 iterations.

Original: D

Original: [Equation]

Original: [Equation]

Original: after only 1000 iterations.

Original: The relationship between inner product layer weight matrices and random matrix theory was ﬁrst seen in inner product layer nets without convolutional layers.

Original: With interest leaning toward convolutional nets we did not include this in the main work, but we include this here for completeness.

Original: The relationship is stronger in deeper layers and for larger weight matrices.

Original: In a 3 layer network in which both hidden layers have 500 nodes and ReLu activation, we compare

Original: FC to (1,100,1)- BD on MNIST where in both cases ip2 is initialized with Xavier variance (Glorot & Bengio, 2010) and all other hyperparameters as perscribed by Caffe (Jia et al., 2014).

Original: Afteriterations, the ratio of variance of weight matrix entries in layer ip2 in block diagonal setting over variance of weight matrix entries in layer ip2 in the fully connected setting is 106.42.

Original: [Figure]

Original: [Figure]

Original: Let λ 1500 W 1 W (cid:62) 1 be the measured empirical spectral distribution of 1500 W 1 W (cid:62) 1 where W 1 is the ip layer weight matrix in the FC architecture with only 3 inner product layers after 10000 iterations.

Original: The ﬁnal variance of the ip2 layer weights in FC is 0.0012.

Original: If R is a 500 × 500 truly random matrix with independent entries that have variance 0.0012, then D KL ( λ 1500 RR (cid:62) ||

Original: λ 1500 W 1 W (cid:62) 1 ) = 0 .

Original: B. CIFAR

Original: The ratio of the variance and singular values of the ip1 layer weight matrix in block diagonal setting over that of the ip1 layer weight matrix in the fully connected setting after 9000 training iterations on CIFAR10 using cuda-convnet can be seen in Figure 4.

Original: These ratios best highlight the relationship to random matrix theory, but here we include the variance and the singular values without taking a ratio because they may also be informative for some readers.

Original: See Figure

Original: Figure 8 (left) compares the theoretical probability density function of the singular values of a truly random matrix with independent entries to the measured distribution of the ip1 layer weight matrix singular values using the FC architecture after 9000 training iterations.

Original: Figure 8 (right) uses the

Original: Kullback-Leibler divergence (Appendix A) to measure the difference between the theoretical prob- ability density function of the singular values of a truly random matrix to the measured distribution of singular values the ip1 layer weight matrix for the ( b 1 , 1) -BD architecture after 9000 training iterations.

Original: For b 1 = 1 , 2 , 4 , 8 , 16 , 32 , and 64 the Kullback-Leibler divergence stays between and

Original: For comparison, the average Kullback-Leibler divergence from the discrete measured empirical spectral distribution of 164 RR (cid:62) where R is an 1024 × 64 matrix with standard normally distributed entries to the Marchenko-Pastur distribution µ 0 .

Original: 0625 as in equation (1) is D KL ( µ 0 .

Original: ||

Original: λ 164 RR (cid:62) )

Original: = 0 .

Original: 0218 where the average is taken over over 20 initializations.

Original: At its largest discrepancy, this is 5 .times less than the measured divergence in Figure 8 (right).