Accelerating Natural Language Understanding in Task-Oriented Dialog (2020)

Abstract: Task-oriented dialog models typically leverage complex neural architectures and large-scale, pre-trained Transformers to achieve state-of-the-art performance on popular natural language understanding benchmarks. However, these models frequently have in excess of tens of millions of parameters, making them impossible to deploy on-device where resource-efficiency is a major concern. In this work, we show that a simple convolutional model compressed with structured pruning achieves largely comparable results to BERT on ATIS and Snips, with under 100K parameters. Moreover, we perform acceleration experiments on CPUs, where we observe our multi-task model predicts intents and slots nearly 63x faster than even DistilBERT.

## introduction the advent of smart devices like amazon alexa, facebook portal, and google assistant has increased the necessity of resource-efficient taskoriented systems @xcite @xcite . these systems chiefly perform two natural language understanding tasks, intent detection and slot filling, where the goals are to understand what the user is trying to accomplish and the metadata associated with the request, respectively @xcite . however, there remains a disconnect between state-of-theart task-oriented systems and their deployment in real-world applications. recent top performing systems have largely saturated performance on atis @xcite and snips @xcite by leveraging complex neural architectures and large-scale, pre-trained language models @xcite , but their usability in on-device settings remains suspect @xcite . mobile phones, for example, have sharp hardware constraints and limited memory capacities, implying systems must optimize for both accuracy and resource-efficiency as possible to be able to run in these types of environments @xcite . in this work, we present a vastly simplified, single-layer convolutional model @xcite that is highly compressible but nonetheless achieves competitive results on task-oriented natural language understanding benchmarks. in order to compress the model, we use structured magnitude-based pruning @xcite , a two-step approach where (1) entire convolutional filters are deleted according to their ℓ 2 norms; and (2) remaining portions of the underlying weight matrix are spliced together. the successive reduction in the number of convolutional output connections permits downstream weight matrices to reduce their number of input connections as well, collectively resulting in a smaller model. structured pruning and re-training steps are then interleaved to ensure the model is able to reconstruct lost filters that may contribute valuable information. during test-time, however, we use the pruned model as-is without further fine-tuning. our simple convolutional model with structured pruning obtains strong results despite having less than 100k parameters. on atis, our multi-task model achieves 95% intent accuracy and 94% slot f1, only about 2% lower than bert @xcite . structured pruning also admits significantly faster inference: on cpus, we show our model is 63× faster than distilbert. unlike compression methods based on unstructured pruning (frankle and carbin, 2019), our model enjoys speedups without having to rely on a sparse tensor library at test-time @xcite , thus we demonstrate the potential for usage in resource-constrained, ondevice settings. our code is publicly available at https://github.com/oja/pruned-nlu . ## convolutional model convolutions for on-device modeling. stateof-the-art task-oriented models are largely based on recurrent neural networks (rnns) @xcite or transformers @xcite . however, these models are often impractical to deploy in lowresource settings. recurrent models must sequentially unroll sequences during inference, and selfattention mechanisms in transformers process sequences with quadratic complexity @xcite . high-performing, pre-trained transformers, in particular, also have upwards of tens of millions of parameters, even when distilled @xcite . convolutional neural networks (cnns), in contrast, are highly parallelizable and can be significantly compressed with structured pruning @xcite , while still achieving competitive performance on a variety of nlp tasks @xcite . furthermore, the core convolution operation has enjoyed speedups with dedicated digital signal processors (dsps) and field programmable gate arrays (fpgas) @xcite . model compatibility with on-device hardware is one of the most important considerations for practitioners as, even if a model works well on high throughput gpus, its components may saturate valuable resources like memory and power @xcite . ## structured pruning structured vs. unstructured pruning. pruning is one compression technique that removes weights from an over-parameterized model @xcite , often relying on a heuristic function that ranks weights (or groups of weights) by their importance. methods for pruning are broadly categorized as unstructured and structured: unstructured pruning allows weights to be removed haphazardly without geometric constraints, but structured pruning induces well-defined sparsity patterns, for example, dropping entire filters in a convolutional layer according to their norm @xcite @xcite . critically, the model's true size is not diminished with unstructured pruning, as without a sparse tensor library, weight matrices with scattered zero elements must still be stored @xcite . in contrast, structurally pruned models do not rely on such libraries at test-time since non-zero units can simply be spliced together. ## tasks and datasets we build convolutional models for intent detection and slot filling, two popular natural language understanding tasks in the task-oriented dialog stack. intent detection is a multi-class classification problem, whereas slot filling is a sequence labeling problem. formally, given utterance tokens @xcite @xcite and our systems on atis and snips. we experiment with single-task and multi-task models. number of model parameters are shown in parentheses where applicable; multi-task models use the format (atis/snips). following previous work, we evaluate on atis @xcite and snips @xcite , both of which are single-turn dialog benchmarks with intent detection and slot filling. atis has 4,478/500/893 train/validation/test samples, respectively, with 21 intents and 120 slots. snips has 13,084/700/700 samples with 7 intents and 72 slots. our setup follows the same preparation as @xcite . ## experiments and results we evaluate the performance, compression, and acceleration of our structured pruning approach against several baselines. note that we do not employ post-hoc compression methods like quantization @xcite , as they are orthogonal to our core method, and can be utilized at no additional cost to further improve performance on-device. ## conclusion in this work, we show that structurally pruned convolutional models achieve competitive performance on intent detection and slot filling at only a fraction of the size of state-of-the-art models. our method outperforms popular compression methods, such as knowledge distillation, and results in large cpu speedups compared to bert and distilbert.