ARTA : Collection and Classification of Ambiguous Requests and Thoughtful Actions (2021)

Abstract: Human-assisting systems such as dialogue systems must take thoughtful, appropriate actions not only for clear and unambiguous user requests, but also for ambiguous user requests, even if the users themselves are not aware of their potential requirements. To construct such a dialogue agent, we collected a corpus and developed a model that classifies ambiguous user requests into corresponding system actions. In order to collect a high-quality corpus, we asked workers to input antecedent user requests whose pre-defined actions could be regarded as thoughtful. Although multiple actions could be identified as thoughtful for a single user request, annotating all combinations of user requests and system actions is impractical. For this reason, we fully annotated only the test data and left the annotation of the training data incomplete. In order to train the classification model on such training data, we applied the positive/unlabeled (PU) learning method, which assumes that only a part of the data is labeled with positive examples. The experimental results show that the PU learning method achieved better performance than the general positive/negative (PN) learning method to classify thoughtful actions given an ambiguous user request.

## introduction task-oriented dialogue systems satisfy user requests by using pre-defined system functions (application programming interface (api) calls). natural language understanding, a module to bridge user requests and system api calls, is an important technology for spoken language applications such as smart speakers @xcite . although existing spoken dialogue systems assume that users give explicit requests to the system @xcite , users may not always be able to define and verbalize the content and conditions of their own requests clearly @xcite . on the other hand, human concierges or guides can respond thoughtfully even when the users' requests are ambiguous. for example, when a user says, "i love the view here," they can respond, "shall i take a picture?" if a dialogue agent can respond thoughtfully to a user who does not explicitly request a specific function, but has some potential request, the agent can provide effective user support in many cases. we aim to develop such a system by collecting a corpus of user requests and thoughtful actions (responses) of the dialogue agent. we also investigate whether the system responds thoughtfully to the user requests. the wizard of oz (woz) method, in which two subjects are assigned to play the roles of a user and a system, is a common method for collecting a user-system dialogue corpus @xcite . however, in the collection of thoughtful dialogues, the woz method faces the following two problems. first, even humans have difficulty responding thoughtfully to every ambiguous user request. second, since the system actions are constrained by its api calls, the collected actions sometimes are infeasible. to solve these problems, we pre-defined 70 system actions and asked crowd workers to provide the antecedent requests for which each action could be regarded as thoughtful. we built a classification model to recognize single thoughtful system actions given the ambiguous user requests. however, such ambiguous user requests can be regarded as antecedent requests of multiple system actions. for example, if the function "searching for fast food" and the function "searching for a cafe" are invoked in action to the antecedent request "i'm hungry," both are thoughtful actions. thus, we investigated whether the ambiguous user requests have other corresponding system actions in the 69 system actions other than the pre-defined system actions. we isolated a level definition q1 the request as presented to the dialogue agent since annotating completely multi-class labels is difficult in actual data collection @xcite , we left the training data as incomplete data prepared as one-to-one user requests and system actions. we defined a problem to train a model on the incompletely annotated data and tested on the completely annotated data foot_0 . in order to train the model on the incomplete training data, we applied the positive/unlabeled (pu) learning method @xcite , which assumes that some of the data are annotated as positive and the rest are not. the experimental results show that the proposed classifier based on pu learning has higher classification performances than the conventional classifier, which is based on general positive/negative (pn) learning. ## thoughtful system action to ambiguous user request existing task-oriented dialogue systems assume that user intentions are clarified and uttered in an explicit manner; however, users often do not know what they want to request. user requests in such cases are ambiguous. @xcite ) categorizes user states in information search into four levels according to their clarity, as shown in most of the existing task-oriented dialogue systems @xcite convert explicit user requests (q3) into machine readable expressions (q4). future dialogue systems need to take appropriate actions even in situations such as q1 and q2, where the users are not able to clearly verbalize their requests @xcite . we used crowdsourcing to collect ambiguous user requests and link them to appropriate system actions. this section describes the data collection. ## thoughtful action classification we collected pairs of ambiguous user requests and thoughtful system action categories in sec. 2. using this data, we developed a model that outputs thoughtful actions to given ambiguous user requests. the model classifies user requests into categories of corresponding actions. posi- # ratio (%) 0 70, 207 (72.68) 1 14, 425 (14.93) 2 6, 986 (7.23) 3 4, 982 (5.16) all 96, 600 figure 3 : user request classifier ## experiments we evaluate the models developed in sec. 3, which classify user requests into the corresponding action categories. ## related work we addressed the problem of building a natural language understanding system for ambiguous user requests, which is essential for task-oriented dialogue systems. in this section, we discuss how our study differs from existing studies in terms of corpora for task-oriented dialogue systems and dealing with ambiguous user requests. ## conclusion we collected a dialogue corpus that bridges ambiguous user requests to thoughtful system actions while focusing on system action functions (api calls). we asked crowd workers to input antecedent user requests for which pre-defined dialogue agent actions could be regarded as thoughtful. we also constructed test data as a multiclass classification problem, assuming cases in which multiple action candidates are qualified as thoughtful for the ambiguous user requests. furthermore, using the collected corpus, we developed classifiers that classify ambiguous user requests into corresponding categories of thoughtful system actions. the proposed pu learning method achieved high accuracy on the test data, even when the model was trained on incomplete training data as the multi-class classification task. as future work, we will study the model architecture to improve classification performance. it is particularly necessary to improve the performance of the label propagation. we will also investigate the features of user requests that are difficult to classify.