Example-Driven Intent Prediction with Observers (2021)

Abstract: A key challenge of dialog systems research is to effectively and efficiently adapt to new domains. A scalable paradigm for adaptation necessitates the development of generalizable models that perform well in few-shot settings. In this paper, we focus on the intent classification problem which aims to identify user intents given utterances addressed to the dialog system. We propose two approaches for improving the generalizability of utterance classification models: (1) observers and (2) example-driven training. Prior work has shown that BERT-like models tend to attribute a significant amount of attention to the [CLS] token, which we hypothesize results in diluted representations. Observers are tokens that are not attended to, and are an alternative to the [CLS] token as a semantic representation of utterances. Example-driven training learns to classify utterances by comparing to examples, thereby using the underlying encoder as a sentence similarity model. These methods are complementary; improving the representation through observers allows the example-driven model to better measure sentence similarities. When combined, the proposed methods attain state-of-the-art results on three intent prediction datasets (banking77, clinc150, hwu64) in both the full data and few-shot (10 examples per intent) settings. Furthermore, we demonstrate that the proposed approach can transfer to new intents and across datasets without any additional training.

## introduction task-oriented dialog systems aim to satisfy a user goal in the context of a specific task such as booking flights @xcite , providing transit information @xcite , or acting as a tour guide @xcite . task-oriented dialog systems must first understand the user's goal * work done while shikib was at amazon by extracting meaning from a natural language utterance. this problem is known as intent prediction and is a vital component of task-oriented dialog systems @xcite . given the vast space of potential domains, a key challenge of dialog systems research is to effectively and efficiently adapt to new domains @xcite . rather than adapting to new domains by relying on large amounts of domainspecific data, a scalable paradigm for adaptation necessitates the development of generalizable models that perform well in few-shot settings @xcite . the task of intent prediction can be characterized as a two step process: (1) representation (mapping a natural language utterance to a semantically meaningful representation) and (2) prediction (inferring an intent given a latent representation). these two steps are complementary and interdependent, thereby necessitating that they be jointly improved. therefore, to enhance the domain adaptation abilities of intent classification systems we propose to (1) improve the representation step through observers and (2) improve the prediction step through example-driven training. while bert @xcite ) is a strong model for natural language understanding tasks @xcite , prior work has found a significant amount of bert's attention is attributed to the [cls] and [sep] tokens, though these special tokens do not attribute much attention to the words of the input until the last layer @xcite . motivated by the concern that attending to these tokens is causing a dilution of representations, we introduce observers. rather than using the latent representation of the [cls] token, we instead propose to have tokens which attend to the words of the input but are not attended to. in this manner, we disentangle bert's attention with the objective of improving the semantic content captured by the utterance representations. a universal goal of language encoders is that inputs with similar semantic meanings have similar latent representations @xcite . to maintain consistency with this goal, we introduce example-driven training wherein an utterance is classified by measuring similarity to a set of examples corresponding to each intent class. while standard approaches implicitly capture the latent space to intent class mapping in the learned weights (i.e., through a classification layer), example-driven training makes the prediction step an explicit nonparametric process that reasons over a set of examples. by maintaining consistency with the universal goal of language encoders and explicitly reasoning over the examples, we demonstrate improved generalizability to unseen intents and domains. by incorporating both observers and exampledriven training on top of the convbert model 1 @xcite , we attain state-of-the-art results on three intent prediction datasets: bank-ing77 @xcite , clinc150 @xcite , and hwu64 @xcite in both full data and few-shot settings. to measure the generalizability of our proposed models, we carry out experiments evaluating their ability to transfer to new intents and across datasets. by simply modifying the set of examples during evaluation and without any additional training, our example-driven approach attains strong results on both transfer to unseen intents and across datasets. this speaks to the generalizability of the approach. further, to demonstrate that observers mitigate the problem of diluted representations, we carry out probing experiments and show that the representations produced by observers capture more semantic information than the [cls] token. the contributions of this paper are as follows: ## methods in this section, we describe several methods for the task of intent prediction. we begin by describing two baseline models: a standard bert classifier @xcite and convbert with task-adaptive masked language modelling @xcite . the proposed model extends the con-vbert model of @xcite through observers and example-driven training. given the aforementioned two step characterization of intent prediction, observers aim to improve the representation step while example-driven training improves the prediction step. ## analysis this section describes several experiments that were carried out to show the unique benefits of observers and example-driven training, as well as to validate our hypothesis regarding the two methods. first, we show that with the example-driven formulation for intent prediction, we can attain strong performance on intents unseen during training. next, we show that the generalization to new intents transfers across datasets. next, we carry out a probing experiment that demonstrates that the latent representation of the observers contains greater semantic information about the input. finally, we discuss an ablation over the number of observers used which demonstrates that the benefit of observers is primarily a consequence of the disentangled attention. ## conclusion in order to enhance the generalizability of intent prediction models, we introduce (1) observers and (2) example-driven training. we attain sota results on three datasets in both full data and the few shot settings. furthermore, our proposed approach exhibits the ability to transfer to unseen intents and across datasets without any additional training, highlighting its generalizability. we carry out a probing experiment that shows the representations produced by observers to better capture the semantic information in the input. there are several avenues for future work. (1) observers and example-driven training can be extended beyond intent prediction to tasks like slot filling and dialog state tracking. (2) since observers are disentangled from the attention graph, it is worth exploring whether it possible to force each of the observers to capture a different property of the input (i.e., intent, sentiment, domain, etc.). (3) our mechanism for measuring sentence similarity in our example-driven formulation can be improved.