Distilling Neural Networks for Greener and Faster Dependency Parsing (2020)

Abstract: The carbon footprint of natural language processing research has been increasing in recent years due to its reliance on large and inefficient neural network implementations. Distillation is a network compression technique which attempts to impart knowledge from a large model to a smaller one. We use teacher-student distillation to improve the efficiency of the Biaffine dependency parser which obtains state-of-the-art performance with respect to accuracy and parsing speed (Dozat and Manning, 2017). When distilling to 20% of the original model’s trainable parameters, we only observe an average decrease of ∼1 point for both UAS and LAS across a number of diverse Universal Dependency treebanks while being 2.30x (1.19x) faster than the baseline model on CPU (GPU) at inference time. We also observe a small increase in performance when compressing to 80% for some treebanks. Finally, through distillation we attain a parser which is not only faster but also more accurate than the fastest modern parser on the Penn Treebank.

## introduction ethical nlp research has recently gained attention @xcite . for example, the environmental cost of ai research has become a focus of the community, especially with regards to the development of deep neural networks @xcite . beyond developing systems to be greener, increasing the efficiency of models makes them more costeffective, which is a compelling argument even for people who might downplay the extent of anthropogenic climate change. in conjunction with this push for greener ai, nlp practitioners have turned to the problem of developing models that are not only accurate but also efficient, so as to make them more readily deployable across different machines with varying computational capabilities @xcite @xcite . this is in contrast with the recently popular principle of make it bigger, make it better @xcite . here we explore teacher-student distillation as a means of increasing the efficiency of neural network systems used to undertake a core task in nlp, dependency parsing. to do so, we take a state-of-the-art biaffine parser from @xcite . the biaffine parser is not only one of the most accurate parsers, it is the fastest implementation by almost an order of magnitude among state-of-the-art performing parsers. contribution we utilise teacher-student distillation to compress biaffine parsers trained on a diverse subset of universal dependency (ud) treebanks. we find that distillation maintains accuracy performance close to that of the full model and obtains far better accuracy than simply implementing equivalent model size reductions by changing the parser's network size and training normally. furthermore, we can compress a parser to 20% of its trainable parameters with minimal loss in accuracy and with a speed 2. @xcite faster than that of the original model on cpu (gpu). ## dependency parsing dependency parsing is a core nlp task where the syntactic relations of words in a sentence are encoded as a well-formed tree with each word attached to a head via a labelled arc. figure 1 shows an example of such a tree. the syntactic information attained from parsers has been shown to benefit a number of other nlp tasks such as relation extraction @xcite , machine translation @xcite , and sentiment analysis @xcite . ## network compression model compression has been under consideration for almost as long as neural networks have been utilised, e.g. @xcite introduced a pruning technique which removed weights based on a locally predicted contribution from each weight so as to minimise the perturbation to the error function. more recently, @xcite introduced a means of pruning a network up to 40 times smaller with minimal affect on performance. @xcite and @xcite utilised magnitude-based pruning to increase network generalisation. more specific to nlp, @xcite used absolute-magnitude pruning to compress neural machine translation systems by 40% with minimal loss in performance. however, pruning networks leaves them in an irregularly sparse state which cannot be trivially recast into less sparse architectures. sparse tensors could be used for network layers to obtain real-life decreases in computational complexity, however, current deep learning libraries lack this feature. @xcite introduced structured pruning to account for this, but this kernel-based technique is restricted to convolutional networks. more recently @xcite pruned the heads of the attention mechanism in their neural machine translation system and found that the remaining heads were linguistically salient with respect to syntax, suggesting that pruning could also be used to undertake more interesting analyses beyond merely compressing models and helping generalisation. @xcite and @xcite developed distillation as a means of network compression from the work of @xcite , who compressed a large ensemble of networks into one smaller network. similar and more recent work, used this method of compressing many models into one to achieve state-of-the-art parsing performance @xcite . teacherstudent distillation is the process of taking a large network, the teacher, and transferring its knowledge to a smaller network, the student. teacherstudent distillation has successfully been exploited in nlp for machine translation, language modelling, and speech recognition @xcite 2016; @xcite . beyond that it has also been successfully used in conjunction with exploring structured linguistic prediction spaces @xcite . latterly, it has also been used to distill task-specific knowledge from bert @xcite . other compression techniques have been used such as low-rank approximation decomposition @xcite , vector quantisation @xcite , and huffman coding @xcite . for a more thorough survey of current neural network compression methods see @xcite . ## teacher-student distillation the essence of model distillation is to train a model and subsequently use the patterns it learnt to influence the training of a smaller model. for teacher-student distillation, the smaller model, the student, explicitly uses the information learnt by the larger original model, the teacher, by comparing the distribution of each model's output layer. we use the kullback-leibler divergence to calculate the loss between the teacher and the student: for our implementation, there are two probability distributions for each model, one for the arc prediction and one for the label prediction. by using the distributions of the teacher rather than just using the predicted arc and label, the student can learn more comprehensively about which arcs and labels are very unlikely in a given context, i.e. if the teacher makes a mistake in its prediction, the distribution might still carry useful information such as having a similar probability for y g and y p which can help guide the student better rather than just learning to copy the teacher's predictions. in addition to the loss with respect to the teacher's distributions, the student model is also trained using the loss on the gold labels in the training data. we use categorical cross entropy to calculate the loss on the student's predicted head classifications: ## methodology we train biaffine parsers and apply the teacherstudent distillation method to compress these models into a number of different sizes for a number of universal treebanks v2.4 (ud) @xcite . we use the hyperparameters from @xcite , but use a pytorch implementation for our experiments which obtains the same parsing results and runs faster than the reported speed of the original (see data we use the subset of ud treebanks suggested by de lhoneux et al. (2017) from v2.4, so as to cover a wide range of linguistic features, linguistic typologies, and different dataset sizes. we make some changes as this set of treebanks was chosen from a previous ud version. we exchange kazakh with uyghur because the kazakh data does not include a development set and uyghur is a closely related language. we also exchange ancient-greek-proiel for ancient-greek-perseus because it contains more non-projective arcs (the number of arcs which cross another arc in a given tree) as this was the original justification for including ancient greek. further, we follow smith et al. (2018b) and exchange czech-pdt with russian-gsd. we also included wolof as african languages were wholly unrepresented in the original collection of suggested treebanks (dione, 2019). details of the treebanks pertinent to parsing can be seen in hardware for evaluating the speed of each model when parsing the test sets of each treebank we set the number of cpu cores to be one and either ran the parser using that solitary core or using a gpu (using a single cpu core too). the cpu used was an intel core i7-7700 and the gpu was an nvidia geforce gtx 1080. 3 experiment we compare the performance of each model on the aforementioned ud treebanks with respect to the unlabelled attachment score (uas) which evaluates the accuracy of the arcs, and the labelled attachment score (las) which also includes the accuracy of the arc labels. we also evaluate the differences in inference time for each model on cpu and gpu with respect to sentences per second and tokens per second. we report sentences per second as this has been the measurement traditionally used in most of the literature, but we also use tokens per second as this more readily captures the difference in speed across parsers for different treebanks where the sentence length varies considerably. we also report the number of trainable parameters of each distilled model and how they compare to the baseline, as this is considered a good measure of how green a model is in lieu of the number of floating point operations (fpo) @xcite . 4 6 results and discussion figure 2 shows the average attachment scores across all test treebanks (all results presented in this section are on the test treebanks) for the distilled models and the equivalent-sized base models against the size of the model relative to the original full model. there is a clear gap in performance between these two sets of models with roughly 2 points of uas and las more for the distilled models. this shows that the distilled models do actually manage to leverage the information from the original full model. the full model's scores are also shown and it is clear that on average the model can be distilled to 60% with no loss in performance. when compressing to 20% of the full model, the performance only decreases by about 1 point for both uas and las. figures 3a and 3b show the differences in uas and las for the models distilled to 20% and 80% respectively for each treebank when compared to the equivalent sized baseline model and the full baseline model. the distilled models far outperform the equivalent-sized baselines for all treebanks. it is clear that for the smaller model some treebanks suffer more when compressed to 20% than others when compared to the full baseline model, e.g. finnish-tdt and ancient-greek-perseus. these two treebanks have the largest percentage of non-projective arcs (as can be seen in however, the two smallest treebanks, tamil-ttb and wolof-wtb, actually increase in accuracy when using distillation, especially tamil-ttb, which is by far the smallest treebank, with an increase in uas and las of about 4 points over the full base model. this is likely the result of over-fitting when using the larger, more powerful model, so that reducing the model size actually helps with generalisation. these observations are echoed in the results for the model distilled to 80%, where most treebanks lose less than a point for uas and las against the full baseline, but have a smaller increase in performance over the equivalent-sized baseline. this makes sense as the model is still close in size to the full baseline and still similarly powerful. the increase in performance for tamil-ttb and wolof- ## conclusion we have obtained results that suggest using teacher-student distillation for ud parsing is an effective means of increasing parsing efficiency. the baseline parser used for our experiments was not only accurate but already fast, meaning it was a strong baseline from which to see improvements. we obtained parsing speeds 2.30x (1.19x) faster on cpu (gpu) while only losing ∼1 point for both uas and las when compared to the original sized model. furthermore, the smallest model which obtains these results only has 20% of the original model's trainable parameters, vastly reducing its environmental impact.