In recent years, the fields of natural language processing (NLP) and information retrieval (IR) have made tremendous progress thanks to deep learning models like Recurrent Neural Networks (RNNs), Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTMs)networks, and Transformer  based models like Bidirectional Encoder Representations from Transformers (BERT)  [ [ [ [ [ [ . But these models are humongous in size. On the other hand,real world applications demand small model size, low response times and low computational power wattage. In this survey, we discuss six different types of methods (Pruning, Quantization, Knowledge Distillation, Parameter Sharing, Tensor Decomposition, and Linear Transformer based methods) for compression of such models to enable their deployment in real industry NLP projects.Given the critical need of building applications with efficient and small models, and the large amount of recently published work in this area, we believe that this URLanizes the plethora of work done by the 'deep learning for NLP' community in the past few years and presents it as a coherent story.
In recent years, the fields of natural language processing (NLP) and information retrieval (IR) have made tremendous progress thanksto deep learning models like Recurrent Neural Networks (RNNs), Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTMs)networks, and Transformer 120 based models like Bidirectional Encoder Representations from Transformers (BERT) 24 , GenerativePre-training Transformer (GPT-2)[94 , Multi-task Deep Neural Network (MT-DNN)[73 , Extra-Long Network (XLNet)[134 , Text-to-text transfer transformer (T5)[95 , T-NLG[98 and GShard[63 . But these models are humongous in size. On the other hand,real world applications demand small model size, low response times and low computational power wattage. In this survey, wediscuss six different types of methods (Pruning, Quantization, Knowledge Distillation, Parameter Sharing, Tensor Decomposition, andSub-quadratic Transformer based methods) for compression of such models to enable their deployment in real industry NLP projects.Given the critical need of building applications with efficient and small models, and the large amount of recently published work inthis area, we believe that this URLanizes the plethora of work done by the 'deep learning for NLP' community in the past fewyears and presents it as a coherent story.
R
thanks to
thanksto
A
120
A
24
A
, GenerativePre-training Transformer (GPT-2)
A
94
A
, Multi-task Deep Neural Network (MT-DNN)
A
73
A
, Extra-Long Network (XLNet)
A
134
A
, Text-to-text transfer transformer (T5)
A
95
A
, T-NLG
A
98
A
and GShard
A
63
R
we discuss
wediscuss
R
and Linear
andSub-quadratic
R
in this
inthis
R
few years
fewyears