Extreme Model Compression for On-device Natural Language Understanding (2020)

Abstract: In this paper, we propose and experiment with techniques for extreme compression of neural natural language understanding (NLU) models, making them suitable for execution on resource-constrained devices. We propose a task-aware, end-to-end compression approach that performs word-embedding compression jointly with NLU task learning. We show our results on a large-scale, commercial NLU system trained on a varied set of intents with huge vocabulary sizes. Our approach outperforms a range of baselines and achieves a compression rate of 97.4% with less than 3.7% degradation in predictive performance. Our analysis indicates that the signal from the downstream task is important for effective compression with minimal degradation in performance.

## introduction spoken language understanding (slu) is the task of extracting meaning from a spoken utterance. a typical approach to slu consists of two modules: an automatic speech recognition (asr) module that transcribes the audio into a text transcript, followed by a natural language understanding (nlu) module that predicts the semantics (domain, intent and slots) from the asr transcript. the last few years have seen an increasing application of deep learning approaches to both asr @xcite @xcite and nlu @xcite @xcite , making them more reliable, accurate and efficient. this has led to an increasing popularity of feature-rich commercial voice assistants (vas) -like amazon alexa, google assistant, apple's siri and microsoft's cortana. vas were used in over 3 billion devices in the world in 2019, and are estimated to reach 8 billion devices by 2023 1 . with a growing number of users relying on vas for their day-to-day activities, voice interfaces have become ubiquitous, and are employed in a range of devices, including smart tvs, mobile phones, smart appliances, home assistants and wearable devices. the slu processing for vas is often offloaded to the cloud, where high-performance, compute-rich hardware is used to serve complex machine learning models. however, on-device slu is growing in popularity due to its wide applicability and attractive benefits @xcite @xcite . first, it enables vas to work offline, without an active internet connection, allowing their use in remote areas and on devices with poor or intermittent internet connectivity, for eg. in automobiles. second, on-device processing reduces latency by eliminating communication over the network, and results in an improved user experience. and third, processing utterances on the edge decreases the load on cloud-services, resulting in reduced cloud hardware requirements and associated costs. nlu is the task of extracting intents and semantics from user queries. nlu in vas typically consists of the following sub-tasks -domain classification (dc), intent classification (ic) and named entity recognition @xcite . prior work has shown the effectiveness of recurrent neural models, that jointly model these tasks in a multi-task setup @xcite @xcite . these models typically are made up of large word embeddings, sometimes accounting for more than 90% of the model parameters, and hence require compression for their deployment on resource constrained devices. generic model compression approaches such as quantization @xcite are ineffective for compressing large word-embeddings, as they do not achieve the required performance at high compression rates. prior approaches for word-embedding compression @xcite tackle comparatively smaller vocabulary sizes and are typically post-processing approaches, where compression is performed after the downstream task models are trained. post-processing compression for large vocabulary sizes is not effective as the compression is lossy and task-agnostic. under higher compression rates, post-processing word embedding compression can lead to a significant degradation in downstream performance. in this paper, we present a principled approach for compressing neural models targeted to perform nlu on resource-constrained devices. we tackle a large number of intents and huge vocabularies (∼ 200k), which are typical in a large-scale, commercial nlu system. to overcome the limitations of prior task-agnostic embedding compression approaches, we propose an end-to-end compression technique, where the compression layers are jointly trained with the downstream task (nlu) model. joint training allows for both task-aware compression and compression-aware task learning. task-aware compression enables the compression model to learn better reconstructions for words that are more important to the downstream task. at the same time, compression-aware task learning enables the downstream task model to adapt itself to the errors in embedding reconstructions. we further combine word embedding compression with recurrent layer compression using quantization to compress our model to just a few mb, achieving a compression @xmath0. neural model compression: due to its many practical applications, research on neural model compression has received massive interest in recent years. existing approaches for general neural model compression include low-precision computation @xcite @xcite , quantization @xcite , network pruning @xcite , svd-based weight matrix decomposition @xcite and knowledge distillation @xcite . for neural nlp models, however, larger focus has been on compressing huge word embedding matrices. embedding compression approaches include quantization @xcite , binarization @xcite , dimensionality reduction and matrix factorization methods such as pca @xcite and svd @xcite ). an alternative post-training compression approach using deep compositional code learning (dccl) was also proposed by @xcite . this approach learns compressed embedding representations based on additive quantization @xcite and forms the basis of our task-aware compression approach. in contrast to @xcite , we propose a task-aware compression approach, where embedding compression is performed during the task model training, instead of as a post-processing step. 3 method problem setup: nlu consists of three component tasks -domain classification (dc), intent classification (ic) and named entity recognition (ner). dc and ic are sentence classification tasks and determine the domain (e.g. music) and the intent (e.g. playmusic) of the input utterance. ner is a sequence tagging task, where each word in the utterance is assigned a slot tag (e.g. albumname, song-name etc). the combination of the domain, intent and slots represents the semantic interpretation for the given utterance and is passed on to the downstream application. our goal is to compress the nlu models, to fit within extreme disk space constraints with minimal degradation in predictive performance. furthermore, low-latency and inference support for the models are desirable. ## experiments in this section we describe the datasets used and our experimental setup for model compression. while our approach is generically applicable to any nlp task that uses word embeddings, we show the effectiveness of our approach on the three nlu tasks -dc, ic, and ner. we show our results on a large scale commercial nlu system trained across a large number of intents with huge vocabularies. dataset. we use annotated live traffic data of a large-scale, cloud based, commercial va system to train our nlu models. utterances from the live traffic are randomly sampled and anonymized to remove any customer specific information. they are then annotated by skilled annotators for the nlu labels corresponding to the domain, intent and slot labels for each utterance. the training set chosen for our experiments contains millions of utterances spanning 5 domains, and over 150 intents and slots. one of these domains is the 'out of domain' (or ood) domain, consisting of utterances not supported by the nlu system. the intent for these utterances is labeled as the 'oodintent' and the words are given the 'other' slot tag. our held-out test set is prepared by randomly sampling 1 million utterances from the live-production traffic, following a similar process. in order to facilitate optimization and early stopping, we also use a validation set of a similar scale. evaluation metrics. we use the following metrics for evaluating the performance on the nlu tasks: intent recognition error rate (irer): this is the ratio of number of incorrect interpretations to the total number of utterances. a correct interpretation is when the predicted domain, intent and all slots for an utterances are correct. we compute the irer only on non-ood utterances. intent classification error rate (icer): this is the ratio of number of incorrect intent predictions to the total number of utterances. ## results and analysis in general, we find that task-aware approaches perform better than task-agnostic post-processing approaches. this is because the task-aware end-to-end compression tunes the compression to the downstream task, while also adjusting the task model parameters to recover performance due to lossy reconstructions. from task-aware dccl outperforms even the best task-agnostic compression baseline (tag. dccl + nlu fine-tuning) by 39-44% at each of the different compression rates. this shows that the loss signal from the downstream task helps performance by not only adapting the task model to the compression, but also by improving compression quality. moreover, our model at 120× compression rate performs better than the best baseline even at 60× compression rate. in other words, our models are 2× smaller than even the best baseline for a similar performance. we also find that the embedding reconstruction loss added to the downstream task loss helps improve the downstream performance, especially when the compression rate is lower i.e. when the gumbel-sampling layers are larger or more in number. in order to understand the importance of task-aware compression, we plot the word embedding reconstruction loss (figure 4 ) for the top most frequent words in our dataset. as seen in figure 4 , the average reconstruction loss for task-agnostic dccl remains approximately constant irrespective of frequency of the words, indicating that all words are treated equally. in contrast, task-aware compression reduces the average reconstruction loss for more frequent words indicating that the network capacity is spent to learn better reconstructions for words more important for the downstream task. note that the model used for the graph is the task-aware dccl model without the reconstruction loss term. we also find that dccl-based approaches consistently performed better than their svd counterparts, in both task-aware and task-agnostic variants. svd-based approaches do not perform well beyond a specific compression rate (+7.99% for 1.7× compression). on investigating, we found that word embeddings were full rank matrices, with high singular values for all components, indicating that these components captured high variance. ## conclusion in this paper, we present approaches for extreme model compression for performing natural language understanding on resource-constrained device. we use a unified multi-domain, multi-task neural model that performs dc, ic and ner for all supported domains. we discuss model compression approaches to compress the bulkiest components of our models -the word embeddings, and propose a task-aware end-to-end compression method based on deep compositional code learning where we jointly train the compression layers with the downstream task. this approach reduced word embeddings sizes to just a few mb, achieving a word-embedding compression rate of 98.4% and outperforms all other taskagnostic and task-aware embedding compression baselines. we further apply post-training 8-bit linear quantization to compress the recurrent layers of the model. these approaches together result in a net model compression rate of 97.5%, with a minimal performance degradation of 3.64% when compared to the uncompressed model baseline. dccl approaches are complementary to other compression approaches such as knowledge distillation and model pruning. while our work demonstrates the effectiveness of task-aware dccl on the classification and tagging tasks in nlu, the approach itself is generic and can be applied to other nlp tasks that rely on large word-embeddings. as part of future work, we would like to explore the effectiveness of task-aware dccl on nlp tasks such as machine translation and language modeling. we would also like to explore compression of models with advanced architectures using contextual embeddings.