Efficient Intent Detection with Dual Sentence Encoders (2020)

Abstract: Building conversational systems in new domains and with added functionality requires resource-efficient models that work under low-data regimes (i.e., in few-shot setups). Motivated by these requirements, we introduce intent detection methods backed by pretrained dual sentence encoders such as USE and ConveRT. We demonstrate the usefulness and wide applicability of the proposed intent detectors, showing that: 1) they outperform intent detectors based on fine-tuning the full BERT-Large model or using BERT as a fixed black-box encoder on three diverse intent detection data sets; 2) the gains are especially pronounced in few-shot setups (i.e., with only 10 or 30 annotated examples per intent); 3) our intent detectors can be trained in a matter of minutes on a single CPU; and 4) they are stable across different hyperparameter settings. In hope of facilitating and democratizing research focused on intention detection, we release our code, as well as a new challenging single-domain intent detection dataset comprising 13,083 annotated examples over 77 intents.

## introduction task-oriented conversational systems allow users to interact with computer applications through conversation in order to solve a particular task with well-defined semantics, such as booking restaurants, hotels and flights @xcite @xcite , providing tourist information @xcite , or automating customer support @xcite . intent detection is a vital component of any taskoriented conversational system @xcite . in order to understand the user's current goal, the system must leverage its intent detector to classify the user's utterance (provided in varied natural language) into one of several * equal contribution. tt is now at the oxford university. transfer learning on top of pretrained sentence encoders @xcite , inter alia) has now established as the mainstay paradigm aiming to mitigate the bottleneck with scarce in-domain data. however, directly applying the omnipresent sentence encoders such as bert to intent detection may be sub-optimal. 1) as shown by @xcite , pretraining on a general language-modeling (lm) objective for conversational tasks is less effective than conversational pretraining based on the response selection task and conversational data @xcite . 2) fine-tuning bert and its variants is very resource-intensive as it assumes the adaptation of the full large model. moreover, in few-shot setups fine-tuning may result in overfitting. from a commercial perspective, these properties lead to extremely slow, cumbersome, and expensive development cycles. therefore, in this work we propose to use efficient dual sentence encoders such as universal sentence encoder (use) @xcite and convert @xcite to support intent detection. these models are in fact neural architectures tailored for modeling sentence pairs @xcite , and are trained on a conversational response selection task. as such, they inherently encapsulate conversational knowledge needed for (few-shot) intent detection. we discuss their advantage over lm-based encoders, and empirically validate the usefulness of conversational pretraining for intent detection. we show that intent detectors based on fixed use and convert encodings outperform bert-backed intent detectors across the board on three diverse intent detection datasets, with prominent gains especially in few-shot scenarios. another advantage of dual models is their compactness: foot_2 we demonstrate that our state-of-the-art use+convert intent detectors can be trained even on a regular laptop's cpu in only several minutes. we also show that intent classifiers based on dual sentence encoders are largely invariant to hyperparameter changes. this finding is extremely important for real-life low-data regimes: due to the invariance, the expensive hyperparameter tuning step can be bypassed, and a limited number of annotated examples can be used directly as additional training data (instead of held-out validation data). another contribution of this work is a new and challenging intent detection dataset in the banking domain, dubbed banking77. it follows the very recent endeavor of procuring high-quality intent detection data @xcite , but is very different in nature than the other datasets. unlike prior work which scatters a set of coarsegrained intents across a multitude of domains (i.e., 10+ domains, see ## methodology: intent detection with dual sentence encoders pretrained sentence encoders. large-scale pretrained models have benefited a wide spectrum of nlp applications immensely @xcite @xcite . their core strength lies in the fact that, through consuming large general-purpose corpora during pretraining, they require smaller amounts of domain-specific training data to adapt to a particular task and/or domain @xcite . the adaptation is typically achieved by adding a task-specific output layer to a large pretrained sentence encoder, and then fine-tuning the entire model @xcite . however, the fine-tuning process is computationally intensive @xcite , and still requires sufficient taskspecific data @xcite . as such, the standard full fine-tuning approach is both unsustainable in terms of resource consumption @xcite , as well as suboptimal for few-shot scenarios. ## new dataset: banking77 in spite of the crucial role of intent detection in any task-oriented conversational system, publicly available intent detection datasets are still few and far between, even for english. the previous standard datasets such as web apps, ask ubuntu, the chatbot corpus @xcite or snips @xcite are limited to only a small number of classes (< 10), which oversimplifies the intent detection task and does not emulate the true environment of commercial systems. therefore, more recent work has recognized the need for improved and more challenging intent detection datasets. 1) the dataset of @xcite , dubbed hwu64, contains 25,716 examples for 64 intents in 21 domains. however, the two recent english datasets are multi-domain, and the examples per each domain may not sufficiently capture the full complexity of each domain as encountered "in the wild". therefore, to complement the recent effort on data collection for intent detection, we propose a new singledomain dataset: it provides a very fine-grained set of intents in a banking domain, not present in hwu64 and clinc150. the new banking77 dataset comprises 13,083 customer service queries labeled with 77 intents. its focus on fine-grained single-domain intent detection makes it complementary to the two other datasets: we believe that any comprehensive intent detection evaluation should involve both coarser-grained multi-domain datasets such as hwu64 and clinc150, and a finegrained single-domain dataset such as banking77. the data statistics are summarized in the single-domain focus of banking77 with a large number of intents makes it more challenging. some intent categories partially overlap with others, 4 experimental setup few-shot setups. we conduct all experiments on the three intent detection datasets described in §3. we are interested in wide-scale few-shot intent classification in particular: we argue that this setup most closely resembles the development process of a commercial conversational system, which typically starts with only a small number of data points when expanding to a new domain or task. we simulate such low-data settings by sampling smaller subsets from the full data. we experiment with setups where only 10 or 30 examples are available for each intent, while we use the same standard test sets for each experimental run. 4 mlp design. unless stated otherwise (e.g., in experiments where we explicitly vary hyperparameters), for the mlp classifier, we use a single 512dimensional hidden layer. we train with stochastic gradient descent (sgd), with the learning rate of 0.7 and linear decay. we rely on very aggressive dropout (0.75) and train for 500 iterations to reach convergence. we show how this training regime can improve the model's generalization capability, and we also probe its (in)susceptibility to diverse hyperparameter setups later in §5. low-data settings are balanced, which is especially easy to guarantee in few-shot scenarios. ## results and discussion few-shot scenarios. the focus of this work is on low-data few-shot scenarios often met in production, where only a handful of annotated examples per intent are available. the usefulness of dual sentence encoders comes to the fore especially in this setup: 1) the results indicate gains over the fine-tuned bert model especially for few-shot scenarios, and the gains are more pronounced in our "fewest-shot" setup (with only 10 annotated examples per intent). the respective improvements of use+convert over bert-tuned are +1.77, +1.33, and +0.97 for banking77, clinc150, and hwu64 (10 examples per intent), and we also see banking77 clinc150 hwu64 model 10 full 10 full 10 full bert-fixed 64.9 (67.8) [57.0] 86.2 (88.4) [74.9] 78.1 (80.6) [70.2] 91.2 (92.6) [84.7] 71.5 (72.8) [68.0] 85.9 (86.8) [81.5] use 83.9 (84.4) [83.0] 92.6 (92.9) [91.4] 90.6 (91.0) [89.9] 95.0 (95.3) [93.9] 83.6 (83.9) [83.0] 91.6 (92.1) [90.7] convert 83.1 (83.4) [82.4] 92.6 (93.0) [91.6] 92.4 (92.8) [92.0] 97.1 (97.2) [96.3] 82.5 (83.1) [82.0] 91.3 (91.6) [90.8] use+convert 85.2 (85.5) [84.8] 93.3 (93.5) [92.8] 93.2 (93.5) [92.8] 97.0 (97.2) [96.5] 85.9 (86.2) [85.7] 92.5 (92.8) [91.6] invariance to hyperparameters. a prominent risk in few-shot setups concerns overfitting to small data sets @xcite . another issue concerns the sheer lack of training data, which gets even more pronounced if a subset of the (already scarce) data must be reserved for validation and hyper-parameter tuning. therefore, a desirable property of any few-shot intent detector is its invariance to hyperparameters and, consequently, its off-the-shelf usage without further tuning on the validation set. this effectively means that one could use all available annotated examples directly for training. in order to increase the reliability of the intent detectors and prevent overfitting in few-shot scenarios, we suggest to use the aggressive dropout regularization (i.e., the dropout rate is 0.75), and a very large number of iterations (500), see §4. we now demonstrate that the intent detectors based on dual encoders are very robust with respect to different hyper-parameter choices, starting from this basic assumption that a high number of iterations and high dropout rates r are needed. for each classifier, we fix the base/pivot configuration from §4: the number of hidden layers is @xmath0, its dimensionality is @xmath1, the sgd optimizer is used with the learning rate of 0.75. starting from the pivot configuration, we create other configurations by altering one hyper-parameter at the time from the pivot. we probe the following values: @xmath2.75, 0.5, 0.25}, @xmath3, 1, 2}, @xmath4, 256, 512, 1024}, and we also try out all the configurations with another optimizer: adam with the linearly decaying learning rate of 4×10 -4 . the results with all hyperparameter configs are summarized in resource efficiency. besides strong performance established in further discussion. the results from tables 3 and 4 show that transferring representations from conversational pretraining based on the response selection task (and conversational data) is useful for conversational tasks such as intent detection. this corroborates the main findings from prior work @xcite . the results also suggest that using the current pretrained bert as an off-the-shelf sentence encoder is sub-optimal for an application such as intent detection: bert is much more powerful when used in the fine-tuning mode instead of the less expensive "feature-based" mode @xcite . this might be due to its pretraining lm objective: while both use and convert are forced to reason at the level of full sentences during the re-sponse selection pretraining, bert is primarily a (local) language model. it seems that the next sentence prediction objective is not sufficient to learn a universal sentence encoder which can be applied off-the-shelf to unseen sentences in conversational tasks @xcite . however, bert's competitive performance in the fine-tuning mode, at least in the full data scenarios, suggests that it still captures knowledge which is useful for intent detection. given strong performance of both finetuned bert and dual models in the intent detection task, in future work we plan to investigate hybrid strategies that combine dual sentence encoders and lm-based encoders. note that it is also possible to combine bert-fixed with the two dual encoders, but such ensembles, besides yielding reduced performance, also substantially increase training times (see again we also believe that further gains can be achieved by increasing the overall size and depth of dual models such as convert, but this comes at the expense of its efficiency and training speed: note that the current architecture of convert relies on only 6 transformer layers and embedding dimensionality of 512 (cf., bert-large with 24 layers and 1024-dim embeddings). ## conclusion we have presented intent classification models that rely on sentence encoders which were pretrained on a conversational response selection task. we have demonstrated that using dual encoder models such as use and convert yield state-of-the-art intent classification results on three diverse intent classification data sets in english. one of these data sets is another contribution of this work: we have proposed a fine-grained single-domain data set spanning 13,083 annotated examples across 77 intents in the banking domain. the gains with the proposed models over fully fine-tuned bert-based classifiers are especially pronounced in few-shot scenarios, typically encountered in commercial systems, where only a small set of annotated examples per intent can be guaranteed. crucially, we have shown that the proposed intent classifiers are extremely lightweight in terms of resources, which makes them widely usable: they can be trained on a standard laptop's cpu in several minutes. this property holds promise to facilitate the development of intent classifiers even without access to large computational resources, which in turn also increases equality and fairness in research @xcite . in future work we will port the efficient intent detectors based on dual encoders to other languages, leveraging multilingual pretrained representations @xcite . this work has also empirically validated that there is still ample room for improvement in the intent detection task especially in low-data regimes. therefore, similar to recent work @xcite @xcite , we will also investigate how to transfer intent detectors to low-resource target languages in few-shot and zero-shot scenarios. we also plan to extend the models to handle out-of-scope prediction @xcite . we have released the code and the data sets online at: github.com/polyai-ldn/ polyai-models.