This paper explores the use of knowledge distillation to improve a Multi-Task Deep Neural Network (MT-DNN) for learning text representations across multiple natural language understanding tasks. Although ensemble learning can improve model performance, serving an ensemble of large DNNs such as MT-DNN can be prohibitively expensive. Here we apply the knowledge distillation method (Hinton et al., 2015) in the multi-task learning setting. For each task, we train an ensemble of different MTDNNs (teacher) that outperforms any single model, and then train a single MT-DNN (student) via multi-task learning to distill knowledge from these ensemble teachers. We show that the distilled MT-DNN significantly outperforms the original MT-DNN on 7 out of 9 GLUE tasks, pushing the GLUE benchmark (single model) to 83.7% (1.5% absolute improvement 1 ). The code and pre-trained models will be made publicly available at https://github.com/namisan/mt-dnn.
REF train an ensemble of large DNNs and then distill their knowledge to a single DNN via multi-task learning to ensemble its teacher performance.
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
ArXiv
ArXiv
Computer Science