  The unstructured sparsity after pruning poses a challenge to the efficient
implementation of deep learning models in existing regular architectures like
systolic arrays. On the other hand, coarse-grained structured pruning is
suitable for implementation in regular architectures but tends to have higher
accuracy loss than unstructured pruning when the pruned models are of the same
size. In this work, we propose a model compression method based on a novel
weight permutation scheme to fully exploit the fine-grained weight sparsity in
the hardware design. Through permutation, the optimal arrangement of the weight
matrix is obtained, and the sparse weight matrix is further compressed to a
small and dense format to make full use of the hardware resources. Two pruning
granularities are explored. In addition to the unstructured weight pruning, we
also propose a more fine-grained subword-level pruning to further improve the
compression performance. Compared to the state-of-the-art works, the matrix
compression rate is significantly improved from 5.88x to 14.13x. As a result,
the throughput and energy efficiency are improved by 2.75 and 1.86 times,
respectively.
