Intent Detection with W iki H ow (2020)

Abstract: Modern task-oriented dialog systems need to reliably understand users’ intents. Intent detection is even more challenging when moving to new domains or new languages, since there is little annotated data. To address this challenge, we present a suite of pretrained intent detection models which can predict a broad range of intended goals from many actions because they are trained on wikiHow, a comprehensive instructional website. Our models achieve state-of-the-art results on the Snips dataset, the Schema-Guided Dialogue dataset, and all 3 languages of the Facebook multilingual dialog datasets. Our models also demonstrate strong zero- and few-shot performance, reaching over 75% accuracy using only 100 training examples in all datasets.

## introduction task-oriented dialog systems like apple's siri, amazon alexa, and google assistant have become pervasive in smartphones and smart speakers. to support a wide range of functions, dialog systems must be able to map a user's natural language instruction onto the desired skill or api. performing this mapping is called intent detection. intent detection is usually formulated as a sentence classification task. given an utterance (e.g. "wake me up at 8"), a system needs to predict its intent (e.g. "set an alarm"). most modern approaches use neural networks to jointly model intent detection and slot filling @xcite @xcite . in response to a rapidly growing range of services, more attention has been given to zero-shot intent detection (ferreira et al., 2015a,b; yazdani and henderson, 2015; chen et al., 2016; kumar et al., 2017; gangadharaiah and 1 the data and models are available at https:// github.com/zharry29/wikihow-intent . narayanaswamy, 2019). while most existing research on intent detection proposed novel model architectures, few have attempted data augmentation. one such work @xcite showed that models can learn much knowledge that is important for intent detection from massive online resources such as wikipedia. we propose a pretraining task based on wiki-how, a comprehensive instructional website with over 110,000 professionally edited articles. their topics span from common sense such as "how to download music" to more niche tasks like "how to crochet a teddy bear." we observe that the header of each step in a wikihow article describes an action and can be approximated as an utterance, while the title describes a goal and can be seen as an intent. for example, "find good gas prices" in the article "how to save money on gas" is similar to the utterance "where can i find cheap gas?" with the intent "save money on gas." hence, we introduce a dataset based on wikihow, where a model predicts the goal of an action given some candidates. although most of wikihow's domains are far beyond the scope of any present dialog system, models pretrained on our dataset would be robust to emerging services and scenarios. also, as wikihow is available in 18 languages, our pretraining task can be readily extended to multilingual settings. using our pretraining task, we fine-tune transformer language models, achieving state-of-the-art results on the intent detection task of the snips dataset @xcite , the schema-guided dialog (sgd) dataset @xcite , and all 3 languages (english, spanish, and thai) of the facebook multilingual dialog datasets @xcite , with statistically significant improvements. as our accuracy is close to 100% on all these datasets, we further experiment with zero-or few-shot settings. our models achieve over 70% accuracy with no in-domain training data on snips and sgd, and over 75% with only 100 training examples on all datasets. this highlights our models' ability to quickly adapt to new utterances and intents in unseen domains. ## experiments we fine-tune a suite of off-the-shelf language models pretrained on our wikihow data, and evaluate them on 3 major intent detection benchmarks. ## discussion and future work as our model performances exceed 99% on snips and fb-en, the concern arises that these intent detection datasets are "solved". we address this by performing error analysis and providing future outlooks for intent detection. ## conclusion by pretraining language models on wikihow, we attain state-of-the-art results in 5 major intent detection datasets spanning 3 languages. the wideranging domains and languages of our pretraining resource enable our models to excel with few labelled examples in multilingual settings, and suggest open-domain intent detection is now feasible.