HINT 3: Raising the bar for Intent Detection in the Wild (2020)

Abstract: Intent Detection systems in the real world are exposed to complexities of imbalanced datasets containing varying perception of intent, unintended correlations and domain-specific aberrations. To facilitate benchmarking which can reflect near real-world scenarios, we introduce 3 new datasets created from live chatbots in diverse domains. Unlike most existing datasets that are crowdsourced, our datasets contain real user queries received by the chatbots and facilitates penalising unwanted correlations grasped during the training process. We evaluate 4 NLU platforms and a BERT based classifier and find that performance saturates at inadequate levels on test sets because all systems latch on to unintended patterns in training data.

## introduction over the last few years, task-oriented dialogue systems have gained increasing traction for applications like personal assistants, automated customer support agents, etc. this has led to the availability of several commercialised and/or open conversational bot building platforms. most popular systems today involve intent detection as a vital part of their natural language understanding (nlu) pipeline. recent advances in transfer learning @xcite @xcite has enabled systems that perform quite well on existing benchmarking datasets @xcite . definitions of intent often vary across users, tasks and domains. perception of intent could range from a generic abstraction such as "ordering a product" to extreme granularity such as "enquiring for a discount on a specific product if ordered using a specific card". additionally, factors such as imbalanced data distribution in the training set, assumptions during training data generation, diverse background of domain experts involved in defining the classes make this task more challenging. during inference, these systems may be deployed to users with diverse cultural backgrounds who might frame their queries differently even when communicating in the same language. furthermore, during inference, apart from correctly identifying in-scope queries, the system is expected to accurately reject out-of-scope @xcite queries, adding on to the challenge. most existing datasets for intent detection are generated using crowdsourcing services. to accurately benchmark in real-world settings, we release 3 new single-domain datasets, each spanning multiple coarse and fine grain intents, with the test sets being drawn entirely from actual user queries on the live systems at scale instead of being crowdsourced. on these datasets, we find that the performance of existing systems saturates at unsatisfactory levels because they end up learning spurious patterns from the training dataset instead of generalising to the perceived meanings of intents. we evaluate 4 nlu platforms -dialogflow foot_0 , luis foot_1 , rasa nlu foot_2 , haptik 45 and a bert @xcite based classifier on all 3 datasets and highlight gaps in language understanding. we further probe into queries where all the current systems fail and question the efficacy of the current approach of learning. additionally, we repeat all our experiments on the subset of training data and show a performance drop in all the systems despite retaining relevant and sufficient utterances in the training subset. we've made our datasets and code freely accessible on github to promote dataset example intents example queries type label train test in-scope out-of-scope sof mattress generic offers what are the available offers any other offers if i order now do i get 20% discount in lockdown period give me some discount specific 100 night trial offer what is the 100-night offer free 100 days trial trial offer on customisation i need try curekart generic recommend product sir i want to fast gain weight. i need beginners hair multivitamin role of electrolytes powder i'm beginner in gym which is the best whey protein is it help for sperm count for biceps and tricep muscle growth supplements can i get rivamal 120 ml at my home can diabetic patient have it power play11 generic chat with an agent connect with agent chat with customer service agent application is not responding during team joining my transaction is incorrect powerplay11 rummy issue my sign up bonus is incorrect suddenly balance gone why my current basketball teams being shown?? did not receive my amount why it is showing wrong balance ## prior work despite intent detection being an important component of most dialogue systems, very few datasets have been collected from real users. web apps, ask ubuntu and chatbot datasets from @xcite contain a limited number of intents (<10), oversimplifying the task. more recent datasets like hwu64 from @xcite and clinc150 from @xcite span a large number of intents in multiple domains but are generated using crowd sourcing services hence are limited in diversity in user expressions which arise from but not limited to domain specific presumptions, context from how and where the bot is made available, paraphrases emerging from cultural and ethnic diversity of user base, conversational slang, etc. our work has some similarity with clinc150, in that they also highlight the problem of out-ofscope intent detection and with banking77 from @xcite ) that focuses on a single domain. however, all three -hwu64, clinc150, banking77 offer relatively large and well balanced training set which might not be always feasible to collect for every new domain. for all datasets mentioned so far, recent works have reported a reasonably high performance (>90% average) for in-scope queries. despite this, gaps in language understanding become apparent when such systems are deployed. datasets introduced in this paper and further analysis of results attempts to recognise critical gaps in language understanding and calls for further research into more robust methods. ## datasets we introduce hint3, a collection of datasets shown in ## benchmark evaluation we evaluated the performance of our datasets on platforms like dialogflow, luis, rasa and haptik in addition to evaluating performance on bert. all layers of bert were fine-tuned with a learning rate of 4e-5 for up to 50 epochs with a warmup period of 0.1 and early stopping. ## results figure 1 presents results for all systems, for both full and subset variations of the dataset. best accuracy on all the datasets is in the early 70s. best mcc for the datasets varies from 0.4 to 0.6, suggesting the systems are far from perfectly understanding natural language. in analyzing few example queries which failed on all platforms in ## conclusion this paper analyzed intent detection on 3 new datasets consisting of both in-scope and out-ofscope queries received on 3 live chat bots over a period of 15 days. our findings indicate that there's a significant gap in performance on crowdsourced datasets vs in a real world setup. nlu systems don't seem to be actually "understanding" language or capturing "meaning". we believe our analysis and dataset will lead to developing better, more robust dialogue systems.