The M icrosoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding (2020)

Abstract: We present MT-DNN, an open-source natural language understanding (NLU) toolkit that makes it easy for researchers and developers to train customized deep learning models. Built upon PyTorch and Transformers, MT-DNN is designed to facilitate rapid customization for a broad spectrum of NLU tasks, using a variety of objectives (classification, regression, structured prediction) and text encoders (e.g., RNNs, BERT, RoBERTa, UniLM). A unique feature of MT-DNN is its built-in support for robust and transferable learning using the adversarial multi-task learning paradigm. To enable efficient production deployment, MT-DNN supports multi-task knowledge distillation, which can substantially compress a deep neural model without significant performance drop. We demonstrate the effectiveness of MT-DNN on a wide range of NLU applications across general and biomedical domains. The software and pre-trained models will be publicly available at https://github.com/namisan/mt-dnn .

## introduction nlp model development has observed a paradigm shift in recent years, due to the success in using pretrained language models to improve a wide range of nlp tasks @xcite . unlike the traditional pipeline approach that conducts annotation in stages using primarily supervised learning, the new paradigm features a universal pretraining stage that trains a large neural language model via self-supervision on a large unlabeled text corpus, followed by a fine-tuning step that starts from the pretrained contextual representations and conducts supervised learning for individual tasks. the pretrained language models can effectively model textual variations and distributional similarity. therefore, they can make subsequent task-specific training more sample efficient and often significantly boost performance in downstream tasks. however, these models are quite large and pose significant challenges to production deployment that has stringent memory or speed requirements. as a result, knowledge distillation has become another key feature in this new learning paradigm. an effective distillation step can often substantially compress a large model for efficient deployment @xcite @xcite . in the nlp community, there are several well designed frameworks for research and commercial purposes, including toolkits for providing conventional layered linguistic annotations @xcite , platforms for developing novel neural models @xcite and systems for neural machine translation @xcite . however, it is hard to find an existing tool that supports all features in the new paradigm and can be easily customized for new tasks. for example, @xcite provides a number of popular transformerbased @xcite text encoders in a nice unified interface, but does not offer multitask learning or adversarial training, state-of-the-art techniques that have been shown to significantly improve performance. additionally, most public frameworks do not offer knowledge distillation. a notable exception is distillbert @xcite , but it provides a standalone compressed model and does not support task-specific model compression that can further improve performance. we introduce mt-dnn, a comprehensive and easily-configurable open-source toolkit for building robust and transferable natural language understanding models. mt-dnn is built upon pytorch @xcite and the popular transformer-based text-encoder interface @xcite . it supports a large inventory of pretrained models, neural architectures, and nlu tasks, and can be easily customized for new tasks. a key distinct feature for mt-dnn is that it provides out-of-box adversarial training, multi-task learning, and knowledge distillation. users can train a set of related tasks jointly to amplify each other. they can also invoke adversarial training @xcite @xcite , which helps improve model robustness and generalizability. for production deployment where large model size becomes a practical obstacle, users can use mt-dnn to compress the original models into substantially smaller ones, even using a completely different architecture (e.g., compressed bert or other transformer-based text encoders into lstms @xcite ). the distillation step can similarly leverage multitask learning and adversarial training. users can also conduct pretraining from scratch using the masked language model objective in mt-dnn. moreover, in the fine-tuning step, users can incorporate this as an auxiliary task on the training text, which has been shown to improve performance. mt-dnn provides a comprehensive list of stateof-the-art pre-trained nlu models, together with step-by-step tutorials for using such models in general and biomedical applications. ## application in this section, we present a comprehensive set of examples to illustrate how to customize mt-dnn for new tasks. we use popular benchmarks from general and biomedical domains, including glue @xcite , @xcite , scitail @xcite , squad @xcite , anli @xcite , and biomedical named entity recognition (ner), relation extraction (re) and question answering (qa) @xcite . to make the experiments reproducible, we make all the configuration files publicly available. we also provide a quick guide for customizing a new task in jupyter notebooks. ## conclusion microsoft mt-dnn is an open-source natural language understanding toolkit which facilitates researchers and developers to build customized deep learning models. its key features are: 1) support for robust and transferable learning using adversarial multi-task learning paradigm; 2) enable knowledge distillation under the multi-task learning setting which can be leveraged to derive lighter models for efficient online deployment. we will extend mt-dnn to support natural language generation tasks, e.g. question generation, and incorporate more pre-trained encoders, e.g. t5 @xcite in future.