Distilling knowledge from a well-trained cumbersome network to a small one has recently become a new research topic, as lightweight neural networks with high performance are particularly in need in various resource-restricted systems. This paper addresses the problem of distilling word embeddings for NLP tasks. We propose an encoding approach to distill task-specific knowledge from a set of highdimensional embeddings, so that we can reduce model complexity by a large margin as well as retain high accuracy, achieving a good compromise between efficiency and performance. Experiments reveal the phenomenon that distilling knowledge from cumbersome embeddings is better than directly training neural networks with small embeddings. Model compression; neural networks; word embeddings As said, existing approaches to knowledge transfer between two neural networks mainly follow a two-step strat-
In natural language processing (NLP), REF distill task-specific knowledge from word embeddings.
Distilling Word Embeddings: An Encoding Approach
CIKM '16
Computer Science