“What do others think?”: Task-Oriented Conversational Modeling with Subjective Knowledge (2023)

Abstract: Task-oriented Dialogue (TOD) Systems aim to build dialogue systems that assist users in accomplishing specific goals, such as booking a hotel or a restaurant. Traditional TODs rely on domain-specific APIs/DBs or external factual knowledge to generate responses, which cannot accommodate subjective user requests (e.g.,”Is the WIFI reliable?” or “Does the restaurant have a good atmosphere?”). To address this issue, we propose a novel task of subjective-knowledge-based TOD (SK-TOD). We also propose the first corresponding dataset, which contains subjective knowledge-seeking dialogue contexts and manually annotated responses grounded in subjective knowledge sources. When evaluated with existing TOD approaches, we find that this task poses new challenges such as aggregating diverse opinions from multiple knowledge snippets. We hope this task and dataset can promote further research on TOD and subjective content understanding. The code and the dataset are available at https://github.com/alexa/dstc11-track5.

## introduction task-oriented dialogue (tod) systems aim to build dialogue systems that assist users in accomplishing specific goals, such as booking a hotel or a restaurant. most solutions of tod are based on domain-apis @xcite and structured databases @xcite , which can only handle a limited range of scenarios within the scope of apis/dbs. to further enlarge the model's ability of task-oriented assistance, recent works @xcite @xcite @xcite incorporate unstructured textual information retrieved from the internet into dialogue modeling. most of these works focus on factual knowledge sources such as frequently asked questions (faqs) of online proddialogue 2 u: i'd like to find someplace to stay in downtown. s: both avalon and gonville are located near downtown. would you like to stay in one of those guesthouses? u: which one has free wifi? s: both facilities do. u: is it high quality wifi with a stable connection? s: overall, customers thought the wifi at gonville is fast, while that at avalon may be slow and unstable. ## problem formulation formally, we have a dialogue context @xmath0, s 1 , u 2 , s 2 , • • • , u t ] between a user and a system, where each user utterance u i is followed by a system response utterance s i , except for the last user utterance u t . the dialogue involves one or more entities, denoted as @xmath1, • • • , e m }. alongside the dialogue, we have a subjective knowledge source @xmath2, r 1 ), (e 2 , r 2 ), • • • } containing all the entities and their corresponding customer reviews. each entity e is associated with multiple reviews @xmath3, r 2 , • • • }. each review can be divided into segments [k 1 , k 2 , • • • ], such as paragraphs, sentences, or sub-sentential units. in this work, we regard each review sentence as a knowledge snippet. the sk-tod task aims to identify whether u t is a subjective knowledge-seeking request and, if it is, to select the relevant knowledge snippets k + from the knowledge source and finally generate a response s t grounded on k + . ## data collection and statistics we ground the data collection in multiwoz @xcite . we select dialogues from the domains of hotels and restaurants. the data collection is conducted by a group of crowd workers through amazon mechanical turk (amt). to control the data quality, we only choose workers that are pre-qualified. more details can be found in appendix a. ## subjective-knowledge-grounded tod in this section, we describe the method for sk-tod. as shown in figure 2 , we follow the pipeline introduced by @xcite which comprises four sequential sub-tasks: knowledge-seeking turn detection (ktd), entity tracking (et), knowledge selection (ks), and response generation (rg). we elaborate on each subtask below. model output subjective knowledge turn detection entity tracking knowledge selection response generation dialogue context tod based on domain api/db tod based on subjective knowledge ## experiments on sub-tasks we first conduct experiments on each individual subtask. to avoid any error accumulation from upstream tasks, we use the gold output of the previous task as the input to the current target task. the detailed experimental setup can be found in appendix c. ## experiments on end-2-end evaluation in the previous section, we use gold information as input for each module to avoid error accumulation. in this section, we evaluate the entire pipeline in an end-to-end manner, where the input of each subtask is predicted by the previous component. we gradually add ks, et, and ktd to the pipeline, and list the performance of ks and rg in the results show that errors introduced during ks can decrease the quality of response generation. however, et and ktd do not have a significant impact on the performance of downstream tasks. it is because et and ktd results include fewer noisy predictions compared to the ks results. ## comparison with fact-tod one difference between sk-tod and fact-tod is that responses in sk-tod are grounded on subjective knowledge instead of factual knowledge. in this section, we investigate whether a fact-tod model can ground on subjective knowledge to address subjective requests. to this end, we retrain our ktd (deberta), ks (deberta crossencoder), and rg (bart) models using the faqgrounded tod data provided by @xcite . we then apply it to the test set of sk-tod without further training. we compare the results of each sub-task using the fact-tod models with the results of models trained on sk-tod. as shown in reviews: review 1: the bed really needed a new mattress, too. review 2: that being said, the room was spacious and the bed was very comfortable. review 3: we both slept extremely well every night we were there. model outputs: fact-tod: yes, the hamilton lodge has a new mattress. would you like to make a reservation? sk-tod: the hamilton lodge has really comfortable beds according to most guests, but one guest did say that the bed needed a new mattress. reference: there are some mixed reviews on the beds. some say they're very comfortable while others were not impressed. would you like me to find another place with better reviews? ## conclusion in this paper, we have introduced sk-tod: a novel task focused on subjective-knowledge-based taskoriented dialogue response generation. we create and release a large-scale, manually-annotated dataset for this task. incorporating subjective knowledge requires models to accurately identify all relevant knowledge snippets and faithfully aggregate the information into concise and contextually appropriate responses, which brings unique challenges to this task. experiments with strong baselines show that there is a significant performance gap between human-generated and machinegenerated responses, particularly in faithfully capturing the diversity and proportion of opinions present in the subjective knowledge. we hope this task together with the provided dataset can promote future research on knowledge-grounded tod systems and subjective content understanding.