Database Search Results Disambiguation for Task-Oriented Dialog Systems (2022)

Abstract: As task-oriented dialog systems are becoming increasingly popular in our lives, more realistic tasks have been proposed and explored. However, new practical challenges arise. For instance, current dialog systems cannot effectively handle multiplesearch results when querying a database, due to the lack of such scenarios in existing public datasets. In this paper, we propose Database Search Result (DSR) Disambiguation, a novel task that focuses on disambiguating database search results, which enhances user experience by allowing them to choose from multiple options instead of just one. To study this task, we augment the popular task-oriented dialog datasets (MultiWOZ and SGD) with turns that resolve ambiguities by (a) synthetically generating turns through a pre-defined grammar, and (b) collecting human paraphrases for a subset. We find that training on our augmented dialog data improves the model’s ability to deal with ambiguous scenarios, without sacrificing performance on unmodified turns. Furthermore, pre-fine tuning and multi-task learning help our model to improve performance on DSR-disambiguation even in the absence of in-domain data, suggesting that it can be learned as a universal dialog skill. Our data and code will be made publicly available.

## introduction task-oriented dialog systems have been widely deployed for popular virtual assistants, like siri and google assistant. they help people with tasks such as booking restaurants and looking for a hotel by searching databases with constraints provided by users. after retrieving a result from the database, a system may continue by conducting actions like making a reservation or providing more information about receiving the result. however, there can be multiple results from the database that match the same constraints. for example, as shown in fig. 1 , the system finds two available hotels at different locations when the user is asking the system to help book a hotel. this kind of ambiguity stops system from proceeding until the system finds out which result the user looks for. therefore, we need to enhance the system with the ability to resolve such ambiguity brought out by multiple items returned from database search. we call this type of ambiguity as database search result ambiguity (dsr-ambiguity). different from semantic ambiguous words (e.g. "orange" can be referred as either color or fruit), the dsr-ambiguity focuses on results from multiple database search results. solving such disambiguation tasks consists of two steps: asking clarification questions and understanding user's corresponding answers. while there is a relatively larger body of literature focusing on when and how to give out the clarification question @xcite @xcite , the focus on understanding user's answers/intents has been relatively sparse. our work mainly focuses on improving model's ability of understanding the answers by augmenting two existing task-oriented dialog datasets: mul-tiwoz @xcite and schema-guided dataset (sgd) @xcite . multiwoz and sgd are the most popular largescale task-oriented dialog datasets, based on which most of the state-of-the-art dialog system models are commonly trained and evaluated. according to our analysis, there are around 66% dialogs of the dataset contains multiple dataset-searching results, which means the dsr-ambiguity exists. in this setting, ambiguities are skipped and the model trained based on these datasets can hardly handle the cases where users prefer to make their own choices among all the results satisfies the constraints. furthermore, users should be given more detailed information about search results. ideally, dialog models should provide the information and assist users to make choices, rather than picking one from the result list and recommending it to users. it is not necessary to list all the results, but enumerating 2 or 3 options would help increase user's engagement. to strengthen the model with the ability to handle the ambiguity, we propose to augment these two datasets with disambiguation turns, where the system provides all possible matched results and lets the user make their own decision based on the complete information. specifically, we first extract templates from the simmc 2.0 dataset @xcite , which is a multi-modal task-oriented dialog dataset containing disambiguation turns but only covering two domains. based on the extracted templates and database from multiwoz and sgd, we synthesize a one-turn dialog dataset, containing only the disambiguation turn, to check whether the model can learn the disambiguation from the data. to be applicable in reality, we expect the model to learn the skill of disambiguation without compromising the performance on other dialog skills. so, we propose to augment the multiwoz and sgd with disambiguation turns and train dialog models with the augmented dataset. to ensure naturalness and diversity of the automatically augmented dataset, we additionally recruit crowd-workers to paraphrase the modified turns. in conclusion, our contribution includes: figure 2 : for this disambiguation task, we assume the dialog context, system utterance including result list and user's answer are given. the goal is to extract the result that the user select and continue the dialog. ## task formulation in this paper, we propose a new task called disambiguation in dialog database search. as shown in fig. 2 , the task assumes that we are provided with the dialog context c, the system response s which includes all the optional results , and the user's utterance u that make a choice. to avoid redundant option lists, we limit the number of options to less than five. the target of the task is to extract the entity of the result selected by the user. ## dataset the most popular task-oriented dialog datasets (multiwoz, sgd) do not contain many cases for the disambiguation task. in order to enable the dialog model to handle this task, we propose to augment these two datasets in three steps described in the following subsections. ## experiment we use gpt2 @xcite as our backbone model and fine-tune it with the augmented sgd and multiwoz datasets separately. multiwoz. multiwoz @xcite is a multi-task task-oriented dialog dataset. it covers seven domains and contains 10k+ dialogs. our augmentation focuses mainly on three domains:"attraction", "hotel" and "restaurant", involving more than 3k dialogs. we choose to conduct our augmentation based on the multiwoz 2.2 @xcite , which is the most widelyaccepted version. schema-guided dataset. sgd @xcite is another popular multi-task dialog dataset. since the dsr-ambiguity problem requires the service containing a target entity and not every service satisfies that requirement, our augmentation involved totally 10 domains and 24 services. we directly compute the accuracy on whether the model can successfully predict the correct named entity as evaluation metric. since the generation is similar to the dialog state tracking task, we also compute the joint goal accuracy (details in appendix.c.2) to evaluate whether the augmentation maintain the model's performance of other tasks. we train gpt2 with both the original and augmented data, and test the fine-tuned models on original/augmented/human paraphrased test sets. the same experiment is conducted for both datasets. in addition to original and augmented training data, we also explore the impact of the synthesized single-turn dialog. learned from ## results and analysis in this section, we present our experimental results including key observations and ablation studies. in addition, we also analyze how to leverage our augmented dataset to deal with dsr-ambiguity in new datasets. ## conclusion & future work in this paper, we proposed a new task, dataset result disambiguation, which is ignored in most popular public task-oriented dialog datasets such as mul-tiwoz and sgd. we showed that models trained on these two datasets can not deal with entity ambiguities. we proposed to address this issue by augmenting existing datasets with relevant disambiguation turns. we extract templates of the disambiguation turns from the simmc2.0 dataset and jointly generate new turns with the databases from multiwoz and sgd for augmentation. to ensure the quality and correctness of the augmentation, we recruit crowd-workers to paraphrase the generated sentences. we benchmark our augmented dataset with the gpt2 model. we observe that the augmentations empower dialog models with a new skill to solve disambiguation tasks without performance drop on the original task. in the future, we plan to incorporate state-of-the-art and realistic entity referencing techniques cases to improve the datasets, which further enhances the dialog system. we hope that our work stimulates further research in identifying and incorporating such universal dialog skills in dialog systems avoiding exploding data-costs.