In this work, we present a novel approach for simultaneous knowledge transfer and model compression called Weight Squeezing. With this method, we perform knowledge transfer from a pre-trained teacher model by learning the mapping from its weights to smaller student model weights, without significant loss of model accuracy. We applied Weight Squeezing combined with Knowledge Distillation to a pre-trained text classification model , and compared it to various knowledge transfer and model compression methods on several downstream text classification tasks  . We observed that our approach produces better results than Knowledge Distillation methods without any loss in inference speed. We also compared Weight Squeezing with Low Rank Factorization methods and observed that our method is significantly faster at inference while being competitive in terms of accuracy.
In this work, we present a novel approach for simultaneous knowledge transfer and model compression called Weight Squeezing. With this method, we perform knowledge transfer from a pre-trained teacher model by learning the mapping from its weights to smaller student model weights, without significant loss of model accuracy. We applied Weight Squeezing to a pre-trained text classification model and compared our method to various other knowledge transfer and model compression methods on several downstream text classification tasks based on the GLUE dataset . We observed that our approach produces better results than other methods for training student models without any loss in inference speed. We also compared Weight Squeezing with Low-Rank Factorization approach and observed that our method is significantly faster at inference while being competitive in terms of accuracy.
D
combined with Knowledge Distillation
R
, and compared it to various
and compared our method to various other
A
based on the GLUE dataset
R
Knowledge Distillation methods
other methods for training student models
R
Low Rank Factorization methods
Low-Rank Factorization approach