Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognition (2024)

Abstract: We present Conformal Intent Classification and Clarification (CICC), a framework for fast and accurate intent classification for task-oriented dialogue systems. The framework turns heuristic uncertainty scores of any intent classifier into a clarification question that is guaranteed to contain the true intent at a pre-defined confidence level.By disambiguating between a small number of likely intents, the user query can be resolved quickly and accurately. Additionally, we propose to augment the framework for out-of-scope detection.In a comparative evaluation using seven intent recognition datasets we find that CICC generates small clarification questions and is capable of out-of-scope detection.CICC can help practitioners and researchers substantially in improving the user experience of dialogue agents with specific clarification questions.

## introduction intent classification (ic) is a crucial step in the selection of actions and responses in task-oriented dialogue systems. to offer the best possible experience with such systems, ic should accurately map user inputs to a predefined set of intents. a widely known challenge of language in general, and ic specifically, is that user utterances may be incomplete, erroneous, and contain linguistic ambiguities. although ic is inherently challenging, a key strength of the conversational setting is that disambiguation or clarification questions (cqs) can be posed @xcite . posing the right cq at the right time results in a faster resolution of the user query, a more natural conversation, and higher user satisfaction @xcite @xcite . cqs have been considered in the context of information retrieval @xcite but have received little attention in the context of task-oriented dialogue. deciding when to ask a cq and how to pose it are challenging tasks @xcite . first, it is not clear when the system can safely proceed under the assumption that the true intent was correctly identified. second, it is not clear when the model is too uncertain to formulate a cq @xcite . finally, it is unclear what the exact information content of the clarification question should be. we present conformal intent classification and clarification (cicc), a framework for deciding when to ask a cq, what its information content should be, and how to formulate it. the framework uses conformal prediction to turn a models' predictive uncertainty into prediction sets that contain the true intent at a predefined confidence level @xcite . the approach is agnostic to the intent classifier, does not require re-training of this model, guarantees that the true intent is in the cq, allows for rejecting the input as too ambiguous if the model is too uncertain, has interpretable hyperparameters, generates clarification questions that are small and is amenable to the problem of detecting out-of-scope inputs. in a comparative evaluations with seven data sets and three ic models, we find that cicc outperforms heuristic approaches to predictive uncertainty quantification in all cases. the benefits of cicc are most prominent for ambiguous inputs, which arise naturally in real-world dialogue settings @xcite . ## related work we discuss related work on ambiguity and uncertainty detection within ic and cp with language models. clarification questions various works acknowledge the problem of handling uncertainty in intent classification and to address it with cqs. dhole (2020) proposes a rule-based approach for asking discriminative cqs. the approach is limited to cqs with two intents, lacks a theoretical foundation, and provides no intuitive way of balancing coverage with cq size. @xcite survey ambiguous queries in the context of conversational search and list sources of ambiguity. they mention that clarification questions should be short, specific, and based on system uncertainty. we propose a principled approach to asking short and specific questions based on uncertainty of any underlying intent classifier for the purposes of taskoriented dialogue. @xcite propose an approach for asking a cq containing a fixed top-k most likely intents with intent-specific uncertainty thresholds. this approach does not come with any theoretical guarantees and its hyperparameters need to be tuned on an additional data set whereas our approach comes with guarantees on coverage of the true intent and with intuitively interpretable hyperparameters that can be tuned on the same calibration set. we do not compare directly to this method but include top-k selection in our benchmark. cqs have been studied in other domains, including information retrieval @xcite , product description improvement @xcite , and open question-answering @xcite . in contrast to the task-specific domain investigated in this work, these domains leave more room for asking generic questions for clarification and do not easily allow for incorporating model uncertainty. furthermore, the proposed methods require ad hoc tuning of scores based on heuristic metrics of model uncertainty, and do not provide ways to directly balance model uncertainty with cq size. ## methodology we address the problem of asking cqs in task oriented dialogue systems in the following way. we take a user utterance and a pre-trained intent classifier, and then return an appropriate response based on the predictive uncertainty of the model. algorithm 1 lists these steps, and an example input is presented in figure 1 . in this section we describe and detail the components of cicc. we start by providing a background on conformal prediction. ## experimental setup this section lists the experiments performed to comparatively evaluate cicc across seven data sets and on three ic models foot_1 . ## results the third column lists the average size of the cq. we see that cicc yields the smallest cqs and that the number of inputs that is deemed too ambiguous is relatively small for cicc. the last column denotes the relative number of inputs that is rejected as too ambiguous. cicc rejects a relatively low number of inputs. upon inspection, many of these inputs could be classified as different intents based ## conclusion we have proposed a framework for detecting and addressing uncertainty in intent classification with conformal prediction. the framework empirically determines when to ask a clarification question and how that question should be formulated. the framework uses a moderately sized calibration set and comes with intuitively interpretable parameters. we have evaluated the framework in eight settings, and have found that the framework strictly outperforms baselines across all metrics in six out of eight cases and performs competitively in the other. the framework additionally handles inputs that are too ambiguous for intent classification naturally. we have additionally proposed and evaluated the usage of cicc for out-of-scope detection and found that it is suitable for this. we finally believe that the framework opens promising avenues for future work, including the usage of intent groups for better adaptivity, an extension to bayesian models to address data drift and unsupervised oos with cicc @xcite , to determine conversation stopping rules based on subsequent questions to rephrase or clarify and to combine it with reinforcement learning for, e.g., personalization @xcite . we believe that cicc and/or conformal prediction may also prove useful in various other tasks, including entity recognition, detecting label errors @xcite and to empirically identify similar intents.