Benchmarking Commercial Intent Detection Services with Practice-Driven Evaluations (2021)

Abstract: Intent detection is a key component of modern goal-oriented dialog systems that accomplish a user task by predicting the intent of users’ text input. There are three primary challenges in designing robust and accurate intent detection models. First, typical intent detection models require a large amount of labeled data to achieve high accuracy. Unfortunately, in practical scenarios it is more common to find small, unbalanced, and noisy datasets. Secondly, even with large training data, the intent detection models can see a different distribution of test data when being deployed in the real world, leading to poor accuracy. Finally, a practical intent detection model must be computationally efficient in both training and single query inference so that it can be used continuously and re-trained frequently. We benchmark intent detection methods on a variety of datasets. Our results show that Watson Assistant’s intent detection model outperforms other commercial solutions and is comparable to large pretrained language models while requiring only a fraction of computational resources and training data. Watson Assistant demonstrates a higher degree of robustness when the training and test distributions differ.

## introduction intent detection and entity recognition form the basis of the natural language understanding (nlu) components of a task-oriented dialog system. the intents and entities identified in a given user utterance help trigger the appropriate conditions defined in a dialog tree which guides the user through a predetermined dialog-flow. these task-oriented dialog systems have gained popularity for designing applications around customer support, personal assistants, and opinion mining, etc. the conversational ai market is expected to grow to an estimated usd 13.9 billion by 2025 as figure 1 : accuracy of commercial solutions on the hint3 collection of datasets. results are averaged across the full versions of the three datasets and their subset versions. the in-scope accuracy is reported on a threshold of 0.1. watson assistant achieves the best results on average. results for all methods except watson assistant are obtained from @xcite . there are three primary challenges in designing intent detection models that power real-world dialog systems: (1) limitations in training data: while typical machine learning models are trained on large, balanced, labeled datasets, practical intent detection systems rely on customer provided data. these datasets are usually small, probably noisy, unbalanced, and contain classes with overlapping semantics, etc. the relatively poor quality of training data makes it hard to train accurate models. (2) robustness to non-standard user inputs: when the intent detection models are deployed in realworld settings, they often operate on test data that differs significantly from the training data. the mismatch in train and test data distributions mainly comes from the free-form of input user queries. these real world queries express the same intents with their non-standard paraphrases, which are difficult to fully cover during training. the lack of large and clean training data makes this problem worse. in this work, taking the aforementioned three realistic challenges into consideration, we evaluate multiple intent detection models and focus on their accuracy, data efficiency, robustness, and computational efficiency. we compare the performance of various commercial intent detection models on three datasets in the hint3 collection @xcite . we also evaluate pretrained language models (lm) on three commonly used public datasets for benchmarking intent detection -clinc150 @xcite , banking77 @xcite , and huw64 @xcite . in addition, we create few-shot learning settings from these datasets, to better match real world low-resource scenarios. furthermore, we measure the "in the wild" robustness of the systems via creating difficult test subsets from existing test sets. finally, we evaluate the classification accuracy and training time of these models because it directly affects the usability and development lifecycle of an conversational ai solution. we build upon the existing study in @xcite which benchmarked commercial solutions aside from ibm watson assistant (i.e., dialogflow, luis, and rasa). we extend this study by adding watson assistant and recent large-scale pretrained lms. we also explore few-shot and robustness settings, and compare the resource efficiency and training times of different commercial solutions as well as pretrained lms. among these solutions, watson assistant's new intent detection algorithm performs better than other commercial solutions (figure 1 ), and achieves comparable accuracy when 2 inference time is usually dependent on service-level agreements between the provider and the user which determine the response time upper bounds of the apis. this is hard to measure and compare across services in a reliable way for the purpose of this study. compared to large-scale pretrained lms (figure 2 ) while being much more efficient. ## related work several datasets have been released to test the performance of intent detection for task-oriented dialog systems such as web apps, ask ubuntu and chatbot corpus from @xcite ; atis dataset @xcite and snips @xcite . the atis and snips datasets have been created with focus on voice interactive chatbots. voice modality has some specific characters, i.e., it does not contain typos and it is less noisy than text-based communication. thus, these datasets are oversimplified version of the text-based intent detection task "in the wild" due to well-constructed dataset and limited number of classes. recently, clinc150 @xcite , banking77 @xcite , and hwu64 @xcite have been used to benchmark the performance of intent detection systems. these datasets cover a large number of intents across a wider range of domains, which captures more real-world complexity of doing finegrained classification. arora et al. (2020) proposed a new collection of datasets called hint3, containing a noisy and diverse set of intents and examples across three domains sourced from domain experts and real users. prior work from arora et al. (2020), braun et al. (2017), and liu et al. (2019a) study the performance of different conversational ai services using the datasets mentioned above. casanueva et al. (2020), larson et al. (2019), arora et al. (2020), bunk et al. (2020) and others have benchmarked several state-of-the-art (sota) pretrained lms such as bert @xcite on the aforementioned datasets. we aim to standardize the benchmarking tests that need to be run while developing an industry scale intent detection system. the tests should cover a variety of real-world datasets, settings such as few-shot scenarios and testing on semantically dissimilar test examples. additionally, the tests should also cover resource efficiency and training time -since they affect the overall deployment costs of a virtual assistant cloud service. a carefully chosen trade-off between accuracy and efficiency is the decision making factor in choosing the algorithm for the real-world intent detection system. ## experiment i: comparison with pretrained lms pretrained lms finetuned for intent detection have been shown to perform very well in recent literature, such as @xcite . users can modify and adapt pretrained lms to serve them as part of a scalable solution. however, this often requires a complex solution design, an example of which can be found in @xcite . in this work we evaluate and compare the commercial services with the following pretrained lms: use base , i.e., universal sentence encoder @xcite ; distilbert base @xcite ; bert base , bert large @xcite ; and roberta base @xcite . we compare watson assistant, rasa, and the aforementioned pretrained-lms on the datasets and settings described in section 3, and measure the training time as well as accuracy. ## experimental ii: comparison among commercial solutions finally, we conduct comparison studies among commercial services. commercial solutions are more suitable for enterprise customers and are designed for users who have limited knowledge of machine learning and natural language processing. one of the challenges in comparing the performance of commercial services and designing experiments lies in the fact that most service providers have terms of use prohibiting any type of benchmarking on their services. to overcome this challenge, we use the prior benchmarking study from @xcite to obtain the performance of existing commercial solutions. in this benchmark, hint3 dataset collection is used which contains three tasks with small amounts of training data. we extended the study by including the results on the watson assistant service. ## conclusion we proposed a new methodology to assess the performance of intent detection "in the wild" in task-oriented dialog systems. in practice, the platforms developed for building and deploying virtual assistants have to consider several scenarios and trade-offs. these systems have to train the best performing models in few-shot settings, strike a compromise between training time and accuracy, and adapt seamlessly to a wide range of domains. we compare the performance of leading commercial services which are designed to develop task-oriented dialog systems on the publicly available datasets and also compared their performance against popular pretrained lms. our results demonstrate that watson assistant outperforms mar-ket competitors on the hint3 dataset collection, which comprises real-world queries. our results also show that watson assistant is competitive with pretrained lms across a wide range of datasets and settings but trains much faster -which is a key factor in usability of a commercial conversational ai solution.