Detecting Ambiguous Utterances in an Intelligent Assistant (2024)

Abstract: In intelligent assistants that perform both chatting and tasks through dialogue, like Siri and Alexa, users often make ambiguous utterances such as “I’m hungry” or “I have a headache,” which can be interpreted as either chat or task intents. Naively determining these intents can lead to mismatched responses, spoiling the user experience. Therefore, it is desirable to determine the ambiguity of user utterances. We created a dataset from an actual intelligent assistant via crowdsourcing and analyzed tendencies of ambiguous utterances. Using this labeled data of chat, task, and ambiguous intents, we developed a supervised intent classification model. To detect ambiguous utterances robustly, we propose feeding sentence embeddings developed from microblogs and search logs with a self-attention mechanism. Experiments showed that our model outperformed two baselines, including a strong LLM-based one. We will release the dataset.

## introduction with the rise of ai-powered devices, intelligent assistants such as siri and alexa have gained popularity. these assistants interact with users in ways that allow them to search for information, operate devices, and even engage in human-like conversations @xcite . when responding to a user request, intelligent assistants must recognize its intent and trigger appropriate modules to fulfill the request. in recent years, while various methods have been utilized to determine intent, there are still challenges in handling ambiguous intents (figure 1 ). for example, the utterance "tokyo station" can be taken as either a route search for a train or a map search (both of which belong to task-oriented utterances), and "i'm hungry" can be taken as a casual conversation starter (non-task-oriented) or as a request for restaurant information (task-oriented). such ambiguity of intent is particularly noticeable in intelligent assistants, where task-oriented and non-task-oriented utterances are mixed, and most utterances are short due to the characteristics of devices. to address these challenges, researchers have made efforts to generate responses that help clarify the intent @xcite @xcite . however, it is crucial first to identify which utterances require such clarification since generating clarification for every utterance is unrealistic. furthermore, these efforts focused on task-oriented dialogue systems, and it remains unclear which types of utterances would exhibit ambiguous intents in intelligent assistants, which encompass a combination of task-oriented and non-task-oriented interactions. considering these, we set up the task of identifying utterances with ambiguous intents in intelligent assistants. to analyze and detect such ambiguity, we collected pairs of user utterances and system responses from the dialogue logs of a commercial intelligent assistant and labeled them using crowdsourcing. we referred to an existing dataset of intelligent assistants (akasaki and kaji, 2017) and assigned three labels: 'chat,' 'task,' and 'ambiguous.' this allows us to simplify the problem and flexibly consider the later process of the system. using the dataset, we conducted an analysis to identify trends in the types of utterances that lead to ambiguous intents. we developed the bert-based classifier using the constructed dataset. to classify noisy utterances robustly, we fed sentence embeddings derived from large-scale search query logs and microblog logs corresponding to task intent and chat intent, respectively, into the model. we weighted those embeddings using a self-attention mechanism to consider which embedding is effective for the target utterance. in the experiments, our method outperformed other classification models, including the resourcepowerful llm-based model, and accurately detected ambiguous utterances. ## detecting ambiguous utterances in intelligent assistants this section describes the intelligent assistant handled in this paper and the task settings. ## dataset this section details the dataset construction and our analysis of the ambiguous utterances. ## detection method here, we describe our detection method. we use bert @xcite as a base classifier bert (•) and perform fine-tuning on the developed corpus to conduct multi-class classification. however, since the utterances of intelligent assistants are often short and noisy, it is difficult to classify them accurately. to address this problem, akasaki and kaji (2017) improved the accuracy of an intent classification model by feeding the likelihood of the at the detection phase, we apply the selfattention mechanism to simcse(u 0 ) query , simcse(u 0 ) post and the output of bert ((u 0 , r -1 , u -1 , r -2 , u -2 )) as: this method first obtains the similarity x ij between e i and e j , where each e represents a bert embedding and sentence embedding. we use additive attention that consists of a feed-forward network to calculate those alignment scores. we then compute the importance weight α ij using the softmax function. the resulting éi are concatenated and used as output o. this output is input to the following softmax layer for 3-label classification. this captures the relationships and importance of each embedding to utterance u 0 and enables robust detection of utterances with ambiguous intents while considering the task-specific and chat-specific nature of intelligent assistants. ## experiments in this section, we build several intent classifiers and investigate their performances. ## summary we focused on detecting ambiguous utterances in intelligent assistants. using crowdsourcing, we labeled real log data and analyzed trends in the dataset. to robustly detect ambiguous utterances, we proposed using sentence embeddings from external resources with a self-attention mechanism. experiments showed the effectiveness of our dataset and method. we plan to integrate our method along with a module of clarification questions into the actual system. this improves user experience and allows us to gather feedback from users. we will release the dataset to facilitate future studies.