GenerationDistillation for Efficient Natural Language Understanding in LowData Settings. Over the past year, the emergence of transfer learning with largescale language models (LM) has led to dramatic performance improvements across a broad range of natural language understanding tasks. However, the size and memory footprint of these large LMs makes them difficult to deploy in many scenarios (e.g. on mobile phones). Recent research points to knowledge distillation as a potential solution, showing that when training data for a given task is abundant, it is possible to distill a large (teacher) LM into a small taskspecific (student) network with minimal loss of performance. However, when such data is scarce, there remains a significant performance gap between large pretrained LMs and smaller taskspecific models, even when training via distillation. In this paper, we bridge this gap with a novel training approach, called generationdistillation, that leverages large finetuned LMs in two ways: (1) to generate new (unlabeled) training examples, and (2) to distill their knowledge into a small network using these examples. Across three lowresource text classification datsets, we achieve comparable performance to BERT while using 300x fewer parameters, and we outperform prior approaches to distillation for text classification while using 3x fewer parameters.

Promoting the Knowledge of Source Syntax in Transformer NMT Is Not Needed The utility of linguistic annotation in neural machine translation seemed to had been established in past papers. The experiments were however limited to recurrent sequencetosequence architectures and relatively small data settings. We focus on the stateoftheart Transformer model and use comparably larger corpora. Specifically, we try to promote the knowledge of sourceside syntax using multitask learning either through simple data manipulation techniques or through a dedicated model component. In particular, we train one of Transformer attention heads to produce sourceside dependency tree. Overall, our results cast some doubt on the utility of multitask setups with linguistic information. The data manipulation techniques, recommended in previous works, prove xc2xa0ineffective in large data settings. The treatment of selfattention as dependencies seems much more promising: it helps in translation and reveals that Transformer model can very easily grasp the syntactic structure. An important but curious result is, however, that identical gains are obtained by using trivial linear treesu0027u0027 instead of true dependencies. The reason for the gain thus may not be coming from the added linguistic knowledge but from some simpler regularizing effect we induced on selfattention matrices.