  Deep learning techniques have achieved great success in many fields, while at
the same time deep learning models are getting more complex and expensive to
compute. It severely hinders the wide applications of these models. In order to
alleviate this problem, model distillation emerges as an effective means to
compress a large model into a smaller one without a significant drop in
accuracy. In this paper, we study a related but orthogonal issue, data
distillation, which aims to distill the knowledge from a large training dataset
down to a smaller and synthetic one. It has the potential to address the large
and growing neural network training problem based on the small dataset. We
develop a novel data distillation method for text classification. We evaluate
our method on eight benchmark datasets. The results that the distilled data
with the size of 0.1% of the original text data achieves approximately 90%
performance of the original is rather impressive.
