On the Effectiveness of Sentence Encoding for Intent Detection Meta-Learning (2022)

Abstract: Recent studies on few-shot intent detection have attempted to formulate the task as a meta-learning problem, where a meta-learning model is trained with a certain capability to quickly adapt to newly specified few-shot tasks with potentially unseen intent categories. Prototypical networks have been commonly used in this setting, with the hope that good prototypical representations could be learned to capture the semantic similarity between the query and a few labeled instances. This intuition naturally leaves a question of whether or not a good sentence representation scheme could suffice for the task without further domain-specific adaptation. In this paper, we conduct empirical studies on a number of general-purpose sentence embedding schemes, showing that good sentence embeddings without any fine-tuning on intent detection data could produce a non-trivially strong performance. Inspired by the results from our qualitative analysis, we propose a frustratingly easy modification, which leads to consistent improvements over all sentence encoding schemes, including those from the state-of-the-art prototypical network variants with task-specific fine-tuning.

## introduction the task of intent detection aims at classifying user queries, typically in the form of short sentences, into intent categories. it has been widely adopted as one crucial component inside various applications such as dialogue systems, virtual assistants, and search engines. the domain-specific nature of those applications makes intent detection rather challenging because of the difficulty to acquire high-quality labeled data at scale. in particular, the sets of intents could vary a lot in different realworld scenarios. such scenarios motivate the research of few-shot intent detection, which aims to classify utterances with new intent labels given very few labeled examples. recently, there exists a popular stream of research efforts @xcite @xcite ) that models the few-shot intent detection task as a metalearning problem -a general machine learning paradigm which has already been successfully applied in other tasks of natural language processing @xcite @xcite . under this formulation, the target is to train a good meta-learner that could be used to quickly adapt to any fewshot intent classification task with very few labeled examples. one of the most popular methods of meta-learning is the prototypical network @xcite , which learns an embedding of the input data, and then constructs a prototypical representation for every class via averaging over the input embeddings. each query will be classified as the class with the minimum distance between the query embedding and the class prototype. earlier empirical findings @xcite suggest that the prototypical networks could reach the stateof-the-art performance on most intent detection datasets when a text encoder based on fine-tuned bert @xcite is used for sentence representation. however, one notable challenge for prototypical networks, or basically most of the current metalearning approaches is that the models could easily overfit the sparse and biased data distribution from merely a few training instances @xcite . given that the goal of prototypical networks is essentially to learn proper encoding functions for nearest-neighbor classification, one natural question arises: is it possible to utilize other general-purpose sentence representation schemes without fine-tuning on intent detection data? in this way, the cost of collecting and fine-tuning on domain-specific labeled data might be mitigated, while reducing the risk of domain-specific overfitting. in this paper, we conduct an empirical study to verify the utility and the effectiveness of recent popular sentence encoding schemes for intent detection meta-learning. specifically, we make the following contributions: ## results and analysis ## conclusion and future work motivated by the nature of prototypical networks for intent detection meta-learning, in this paper, we empirically compare some modern popular sentence encoders on multiple intent detection benchmarks, observing non-trivially strong performance with better cross-dataset generalization capability than the fine-tuned sentence encoders. inspired by our follow-up analysis, we propose a simple modification that has consistently and substantially brought performance gain over all systems: adding the intent label names into the support set. this strategy not only improves over the performance from general-purpose sentence encoding, but also the state-of-the-art results from the fine-tuned pro-taugment framework. one limitation of our study for now is that the sentence representation in use is mostly based on bert variants. it could be technically interesting to experiment with models pre-trained in a sequence-to-sequence fashion @xcite , which might have better captured semantic paraphrase representations via denoising or other training objectives.