Knowledge Distillation for S wedish NER models: A Search for Performance and Efficiency (2021)

Abstract: The current recipe for better model performance within NLP is to increase model size and training data. While it gives us models with increasingly impressive results, it also makes it more difficult to train and deploy state-of-the-art models for NLP due to increasing computational costs. Model compression is a field of research that aims to alleviate this problem. The field encompasses different methods that aim to preserve the performance of a model while decreasing the size of it. One such method is knowledge distillation. In this article, we investigate the effect of knowledge distillation for named entity recognition models in Swedish. We show that while some sequence tagging models benefit from knowledge distillation, not all models do. This prompts us to ask questions about in which situations and for which models knowledge distillation is beneficial. We also reason about the effect of knowledge distillation on computational costs.

## introduction currently, most research that pushes the boundary for state-of-the-art performance within natural language processing involves the increase of number of model parameters as well as the computations needed for training @xcite @xcite . the trend seems to be that the larger the model, the better the performance. as noted by @xcite these state-of-the-art models require significant computational resources during training as well as deployment. while it certainly is a good thing that state-of-the-art performance within nlp is continuously improving, there is work to be done on model efficiency. more efficient models are needed both for the sake of the environment and for the sake of equal research opportunities. here we define an "efficient model" based on both performance and computational cost, such that a model is more efficient if it has better performance or lower computational cost, and vice versa. knowledge distillation @xcite is one way to improve model efficiency during deployment. there are several works on successful application of knowledge distillation both for pretraining tasks and for specific downstream tasks. @xcite show that knowledge distillation can be used to improve deployment efficiency of models for the downstream task of document classification in english. in this article we investigate the effect of knowledge distillation on models for named entity recognition (ner) in swedish. 1 the intention is to shed some light on how well knowledge distillation performs for different sequence tagging models and in the swedish language. our main goal is to contribute to better model efficiency within nlp. naturally, this entails that we also focus on measuring the efficiency of each model investigated. hopefully, this work will facilitate the development of more efficient models for both the english and the swedish language. our work focuses on the task of named entity recognition, on model efficiency and on improving model efficiency. these topics are hardly new to the nlp arena and we will use this section to describe some of the previous work. ## swedish ner dataset we use the manual ner annotations based on the suc 3.0 dataset @xcite swedish ner task. before training, we reshape the data to a more suitable format for our task. firstly, the manual annotations in the suc 3.0 data contain annotations for the entities person, animal, myth (for example "god"), place, institution, product, work, event and other. these entity categories are not found in ner datasets for other languages. in order to make better comparisons to other languages, we map the entity types in the dataset to the same types as those that can be found in the conll-2003 data, as described by tables 2 and 3 list some of the features of the reshaped dataset. from these tables, we can observe that the swedish dataset is about three times larger than the conll-2003 dataset, while the latter has a higher density of entities. it is worth remarking that while the english dataset was developed for ner, the suc dataset was originally compiled for the purpose of part-of-speech tagging, with the entity annotation added later. additionally, we can resource loc misc org per #examples suc 3.0 train 6,705 4,551 6,005 16,030 51,971 dev 955 549 885 2,135 7,351 test 1,857 1,402 1,574 4,662 14,923 total 9,517 6,502 8,464 22,827 74, 245 ## method the goal of this work is contribute to better model efficiency within nlp by investigating the effect of kd on different ner models in swedish. to this end, we utilize the method for kd as presented in section 4.1 and investigate the ner models seen in section 4.2. the efficiency of our models is then measured as described in section 4.3. ## results and discussion the model scores on the swedish ner test data are split into tables 5 and 6 . the results in the former table are of the simpler models that were not regularized, while the results in the latter are of the models that were regularized. we split the analysis of the results with respect to our aspects of interest. consequently, we start off with a general analysis of the model results for swedish ner, after which we examine the effect of kd on model scores and then study the effect of kd on model efficiency. ## conclusion and future work our work indicates that different models may differ in whether they benefit from kd. thus, we cannot make the assumption that kd should benefit the performance of every model. adding to the question of why some models seem to benefit from kd, we may also ask ourselves in which situations the soft targets of a teacher model may benefit a student model. we observe three different situations for which it is worth to further investigate the effect of kd; 1) when the student model is in need of regularization, 2) when we want more data for the student model to train on and 3) when the data for training is of poor quality. the two latter situations have not been covered in this work, while they have been mentioned by other researchers @xcite . the former situation has already been observed by @xcite , and we have found additional support for it in our work. for this situation we may further investigate how kd works in combination with existing regularization techniques and whether it is a better such technique. from our work we can also conclude that kd may provide us with more efficient models at deployment, while the cost of training these models is high due to the need of a trained teacher. this prompts us to reason about when kd is worth the effort, with regard to how we value an increase in f1 score in terms of computational costs. we also reason about situations when kd may be a good investment for models that will be used heavily during deployment. to fully measure the benefits of kd with respect to model efficiency we conclude that we need to investigate better tools for judging these trade-offs and different deployment situations. future work may also investigate other types of kd, such as extracting more layers than the embedding layer from the teacher model and providing teacher signals to more layers of the student model. potentially, these kd variations could further improve the performance of a model without requiring more computational costs. moreover, it still remains to investigate the benefits of data augmentation for the student models during kd. the question is whether we could attain even better kd results with this approach. this could also be taken one step further to the region of completely unsupervised training on unlabeled data, merely by providing the student model with the labels generated by the teacher. lastly we can conclude that, unsurprisingly, kd works for the swedish language as well. one interesting next step which may benefit the swedish industry would be to develop a swedish distil-bert.