Rank and run-time aware compression of NLP Applications (2020)

Abstract: Sequence model based NLP applications canbe large. Yet, many applications that benefit from them run on small devices with very limited compute and storage capabilities, while still having run-time constraints. As a result, there is a need for a compression technique that can achieve significant compression without negatively impacting inference run-time and task accuracy. This paper proposes a new compression technique called Hybrid Matrix Factorization (HMF) that achieves this dual objective. HMF improves low-rank matrix factorization (LMF) techniques by doubling the rank of the matrix using an intelligent hybrid-structure leading to better accuracy than LMF. Further, by preserving dense matrices, it leads to faster inference run-timethan pruning or structure matrix based compression technique. We evaluate the impact of this technique on 5 NLP benchmarks across multiple tasks (Translation, Intent Detection,Language Modeling) and show that for similar accuracy values and compression factors, HMF can achieve more than 2.32x faster inference run-time than pruning and 16.77% better accuracy than LMF.

## introduction sequence based (lstms/grus) nlp applications are being increasingly run on mobile phones and smart watches. they are typically enabled by querying a cloud-based system to do most of the computation. the energy, latency, and privacy implications associated with running a query on the cloud is changing where users run a neural network application. we should, therefore, expect an increase in the number of nlp applications running on embedded devices. due to the energy and power constraints of edge devices, embedded socs frequently use lower-bandwidth memory technologies and smaller caches compared to desktop and server processors. thus, there is a need for good compression techniques to enable large nlp models to fit into an smaller edge device or ensure that they run efficiently on devices with smaller caches @xcite . additionally, compressing models should not negatively impact the inference run-time as these tasks may have real-time deadlines to provide a good user experience. in order to choose a compression scheme for a particular network, one needs to consider 3 different axes -the compression factor, the inference run-time speedup over the baseline, and the accuracy. ideally, a good compression algorithm should not sacrifice improvement along one axis for improvement along another. for example, network pruning @xcite has shown to be an effective compression technique, but pruning creates a sparse matrix representation that is inefficient to execute on most modern cpus. our analysis shows that pruned networks can achieve a faster run-time than the baseline only for significantly high compression factors. low-rank matrix factorization (lmf) is another popular compression technique that can achieve speedup proportional to the compression factor. however, lmf has had mixed results in maintaining model accuracy @xcite @xcite . this is because lmf reduces the rank of a matrix significantly, reducing its expressibility @xcite . lastly, structured matrices @xcite can also be used to compress neural networks. while these techniques show a significant reduction in computation, this reduction only translates to a realized run-time improvement for large matrices @xcite or while using specialized hardware @xcite . for benchmarks evaluated in this paper, hlf gets 30× speed-up improvement over structured matrix based technique @xcite . given lmf's good run-time characteristics, it can potentially act as an alternative to pruning. however, lmf leads to an accuracy loss. to overcome the problem of finding an alternative to pruning, which preserves the run-time benefit of dense structures of lmf and the accuracy benefits of pruned networks, we introduce a new compression technique called hybrid matrix factorization (hlf). hlf can act as an effective compression technique for nlp edge use cases on embedded cpus. the results are very promising -hlf achieves iso-accuracy for a large compression factor (2× to 4×), improves the cpu run-time over pruning by a factor of 2.32× and can achieve 16.77% better model accuracy than lmf and 9% better accuracy than smaller baselines. ## related work pruning @xcite @xcite has been the most successful compression technique for all types of neural networks. poor hardware characteristics of pruning has led to research in block based pruning technique @xcite . however, block based pruning technique also requires certain amount of block sparsity to achieve faster run-time than baseline. having a strict compression factor requirement to get better run-time is a stringent constraint that hlf manages to avoid. structured matrices have shown significant potential for compression of nn @xcite @xcite @xcite . block circular compression is an extension of structured matrix based compression technique, converting every block in a matrix into structured matrix. we will show in this paper that hlf is a superior technique than block circular decomposition. tensor decomposition (cp decomposition, kronecker, tucker decomposition etc) based methods have also shown significant reduction in parameters @xcite . matrix factorization @xcite @xcite can be categorized under this topic. we will show in this paper that hlf can lead to better accuracy than lmf compressed rnns. quantization is another popular technique for compression @xcite @xcite @xcite @xcite . networks compressed using hlf can be further compressed using quantization. dynamic techniques are used to improve inference run-time of rnns by skipping certain rnn state updates @xcite @xcite . these techniques are based on the assumption that not all inputs to a rnn are needed for final classification task. thus we can learn a small and fast predictor that can learn to skip certain inputs and its associated computation. hlf technique is orthogonal to this technique and networks compressed using hlf can be further optimized using this technique. design of efficient structures for lstm/gru cells like sru @xcite , qrnn @xcite and pru @xcite have also led to networks with faster inference run-time benefits or lesser number of parameters. these structures are different from structured matrices and are hand-crafted after better understanding the application domain. hlf can be further used to optimize the matrices in these architectures to make the resultant network more parameter and run-time efficient. finally, any technique used to reduce the parameter footprint of embedding matrices in nlp can further optimize rnn networks optimized using hlf @xcite . in this paper, we show that hlf can compress networks with compressed word embedding layers. ## results we compare hlf with lmf and 3 other compression techniques -model pruning, small baseline and a structured matrix based technique called block circular decomposition. these techniques and why they need to be considered are discussed below: given the significant slow-down of inference of bcd compressed networks, we do not discuss the results sing bcd compression in the rest of the paper. ## discussion effectively, hlf acts as an alternative to lmf whenever compression using pruning does not lead to the required run-time benefit and lmf leads to loss in accuracy. hlf has a better accuracy than lmf for most evaluation points, validating the assumption in the paper that rank of a matrix in a rnn is important for better task accuracy in nlp applications. additionally, hlf has a better accuracy than smaller baseline. this validates the assumption of the importance of constrained features in addition to the richer features in a small baseline network. ## limitations while hlf provides significant benefits over lmf, there are two limitations associated with the technique: ## conclusion choosing the right compression technique requires looking at three criteria -compression factor, accuracy, and run-time. pruning is an effective compression technique, but can sacrifice speedup over baseline for certain compression factors. lmf achieves better speedup than baseline for all compression factors, but leads to accuracy degradation. this paper introduces a new compression scheme called hlf, which preserves the dense structures of lmf while effectively doubling the rank of the matrix using an intelligent structure by design. this leads to 2× faster inference run-time than pruning and up-to 16% better accuracy than lmf.