  Recently, researchers proposed pruning deep neural network weights (DNNs)
using an $N:M$ fine-grained block sparsity mask. In this mask, for each block
of $M$ weights, we have at least $N$ zeros. In contrast to unstructured
sparsity, $N:M$ fine-grained block sparsity allows acceleration in actual
modern hardware. So far, this was used for DNN acceleration at the inference
phase. First, we suggest a method to convert a pretrained model with
unstructured sparsity to a $N:M$ fine-grained block sparsity model, with little
to no training. Then, to also allow such acceleration in the training phase, we
suggest a novel transposable-fine-grained sparsity mask where the same mask can
be used for both forward and backward passes. Our transposable mask ensures
that both the weight matrix and its transpose follow the same sparsity pattern;
thus the matrix multiplication required for passing the error backward can also
be accelerated. We discuss the transposable constraint and devise a new measure
for mask constraints, called mask-diversity (MD), which correlates with their
expected accuracy. Then, we formulate the problem of finding the optimal
transposable mask as a minimum-cost-flow problem and suggest a fast linear
approximation that can be used when the masks dynamically change while
training. Our experiments suggest 2x speed-up with no accuracy degradation over
vision and language models. A reference implementation can be found at
https://github.com/papers-submission/structured_transposable_masks.
