Distilling Multilingual Transformers into CNN s for Scalable Intent Classification (2022)

Abstract: We describe an application of Knowledge Distillation used to distill and deploy multilingual Transformer models for voice assistants, enabling text classification for customers globally. Transformers have set new state-of-the-art results for tasks like intent classification, and multilingual models exploit cross-lingual transfer to allow serving requests across 100+ languages. However, their prohibitive inference time makes them impractical to deploy in real-world scenarios with low latency requirements, such as is the case of voice assistants. We address the problem of cross-architecture distillation of multilingual Transformers to simpler models, while maintaining multilinguality without performance degradation. Training multilingual student models has received little attention, and is our main focus. We show that a teacher-student framework, where the teacher’s unscaled activations (logits) on unlabelled data are used to supervise student model training, enables distillation of Transformers into efficient multilingual CNN models. Our student model achieves equivalent performance as the teacher, and outperforms a similar model trained on the labelled data used to train the teacher model. This approach has enabled us to accurately serve global customer requests at speed (18x improvement), scale, and low cost.

## introduction for nearly all natural language understanding tasks, e.g. superglue @xcite , state-of-theart results are obtained using pre-trained transformer models. their performance is dependent on their size and the amount of pre-training data, typically billions of tokens @xcite . intent classification (ic), the task of understanding a user's intent from an utterance, is a core component of all voice assistants such as siri or alexa. ic is challenging due to the hundreds of intents and contexts that such systems must support, and ic performance has benefited greatly from transformers @xcite . as voice systems have expanded support to new languages, the benefits of transformers have multiplied with the advent of multilingual versions such as xlm-roberta @xcite . despite the advantages, deploying transformers at scale is not always feasible, mainly due to: (i) large memory footprint (hundreds of gb), foot_0 and (ii) long inference time 2 that is prohibitive for applications processing millions of inputs per minute. while approaches to reducing memory footprint -such as quantization @xcite or pruning @xcite ) -have been proposed, minimizing inference time is more challenging. pruning can speed up inference, but there are limitations to how many self-attention layers can be pruned without loss of performance. knowledge distillation (kd) @xcite is another approach for transferring knowledge across model architectures, e.g. from transformers to lstms @xcite , to optimize performance. however, cross-architecture distillation of multilingual transformers to multilingual non-transformer architectures has received almost no attention in the community. in this work we present the first exposition of this task. specifically, we describe an approach used to deploy multilingual ic models for voice assistants allowing accurate inference at scale, speed, and low-cost. we face two key challenges: (i) meeting low inference latency requirements, allowing us to globally serve customers in real time (millions per minute), and (ii) supporting multi-linguality, here we support 11 locales with 7 languages. example utterances are shown below, which represent ecommerce questions issued in different languages. we use the teacher-student distillation paradigm, and show the optimal kd strategy for multilingual ic can leverage teacher logits alone @xcite . utterances for ic are typically 10-40 tokens, allowing us to exploit an efficient convnet architecture, and assess how they can obtain multilingual and pretrained knowledge from models like xlm-r via distillation. while there have been previous attempts on distilling transformer models into convnets @xcite , our work is the first to explore crossarchitecture multilingual kd on real-world applications with strict requirements for latency and accuracy. we make the following contributions. ## related work we now review some of the popular approaches for distilling and compressing transformer models. model finetuning. @xcite propose an efficient way to fine tune monolingual models on multilingual tasks by simply using the output of cross-lingual transformer models as pseudolabels. their approach is based on the ulmfit model @xcite , where instead of the stacked lstm networks @xcite , they rely on quasi recurrent neural networks @xcite . qrnn are similar to cnn, with the difference that the convoluational operators are done at each timestep, however, due to parallelization, they can be computed much more efficiently than lstms. qrnns are up to 16x faster than lstms, however, for our case, we find that convnets are more efficient than qrnns, as they do not perform stepwise computations as qrnns do. we compare the inference time of qrnns and our proposed student model, and conclude that simple convnets have significantly lower inference time. model compression. @xcite systematically review approaches for compressing transformers. to reduce memory usage, quantization is often applied @xcite . quantization reduces the amount of bits required to store network parameters. for example, parameters represented using float32, can instead be stored using only 16 or fewer bits, reducing memory usage significantly. this allows deploying larger models in compute infrastructure with limited resources. model pruning is a widely explored research direction for compression, mainly consisting of two techniques. first, in unstructured pruning, weights are zeroed out using different strategies @xcite . second, in structured pruning either the self-attention heads @xcite or the encoder layers @xcite are pruned. quantization and pruning facilitate usage of large transformers without the requirement of very high memory capacity (gpu or cpu) machines. quantization, and unstructured model pruning, mainly reduce memory usage. structured pruning, where encoder and self-attention layers are dropped, can improve efficiency. yet, for many real-world applications the latency needs cannot be met (with few milliseconds, as is our case). for instance, pruning more than 50% of attention heads can lead to performance loss @xcite . knowledge distillation (kd). @xcite discuss the trade-offs between model size and performance. training a larger model, and distilling its knowledge to a smaller model, either using the same training data or unsupervised training data, yields identical performance. the contrary cannot be said when training a small model directly, where the performance is significantly worse than its bigger counterpart. kd works under the teacherstudent paradigm, where the teacher's output is used to train the student model such that it mimics the teacher model in terms of the output. there are several efforts in distilling transformers into recurrent @xcite and convolutional architectures @xcite . while recurrent models like lstms can significantly re-duce memory footprint and latency, the step-wise sequential computation induces a large latency overhead that cannot be overcome. conversely, convnets are highly efficient for text classification, both in terms of performance and latency. our approach is similar to that of @xcite , in that we use cnns as the main building block of the student model, however, we differ in several fundamental aspects and make contributions that further push the application of knowledge distillation. first, we deal with a multilingual task, which increases the complexity of the knowledge transfer from the teacher to the student model. second, our convnet architecture is different to account for the multilingual requirement. thirdly, we rely on unsupervised data for distillation, where we show how much data is necessary across different languages to have identical performance between the teacher and student models. ## multilingual distillation method we now describe the kd approach: the ic task, the teacher/student models, and the learning objective. ## experimental setup we now describe the datasets used to train the teacher model, and for distillation. we also define the evaluation metrics used to asses how well the student model mimics its teacher. ## conclusions we described an approach for distilling knowledge from transformer into a single multilingual cnn. to our knowledge this is the first detailed exposition of cross-architecture kd to multilingual student models. we leverage the outlined framework to accurately serve predictions for our customers at speed, scale, low-cost, and across all languages. empirically we showed how such a kd framework can be utilized in practice: on a held out test set (unseen during training and distillation), the output probabilities have a very low kl divergence (cf. appendix a). with increasing amounts of distillation data, the probability distributions become highly similar. figure 4 : ncq/cq confidence distribution (x-axis) as a function of how likely an utterance is to be cq. for ncq, this probability ideally should be close to zero, and vice-versa for cq (close to one). the skewedness score g1 measures the concentration of the probability mass for ncq and cq, respectively. for ncq the higher score the better (in the positive range), whereas for cq, the lower the score the better (negative range). the results are shown for the student models: s 1% , s 10% , s 50% (distilled with 1%, 10%, and 50% of the data, respectively), and for the teacher model t.