Task-Oriented Conversational Modeling with Subjective Knowledge Track in DSTC 11 (2023)

Abstract: Conventional Task-oriented Dialogue (TOD) Systems rely on domain-specific APIs/DBs or external factual knowledge to create responses. In DSTC11 track 5, we aims to provide a new challenging task to accommodate subjective user requests (e.g.,”Is the WIFI reliable?” or “Does the restaurant have a good atmosphere?” into TOD. We release a benchmark dataset, which contains subjective knowledge-seeking dialogue contexts and manually annotated responses that are grounded in subjective knowledge sources. The challenge track received a total of 48 entries from 14 participating teams.

## introduction task-oriented dialogue (tod) systems aim to build dialogue systems to help users to achieve specific goals (e.g., booking a hotel or a restaurant). most solutions of tod are either based on domain-apis @xcite and structured databases @xcite , which can only handle limited scenarios. recent works incorporate unstructured factual information @xcite @xcite @xcite into dialogue modeling, which further enlarges the model's ability of task-oriented assistance. however, in many tod tasks, users care about not only factual information but subjective information as well, such as the experiences, opinions, and preferences of other customers. for example, when booking a hotel or a restaurant, users may ask questions like "is the wifi reliable?" or "does the restaurant have a good atmosphere?". to respond to such user requests, an agent needs to seek information from subjective knowledge sources such as online customer reviews. in this track, we focus on enabling the tod model to leverage subjective knowledge during dialogue 3 u: i'm looking for a hotel called avalon. s: the avalon hotel is located in downtown. u: does it have wifi? s: yes, it has wifi. u: is the wifi reliable? s: according to previous reviews, some customers have complained that the wifi is slow and make work a bit difficult. but one said it is fast. dialogue 2 u: i'd like to find someplace to stay in downtown. s: both avalon and gonville are located near downtown. would you like to stay in one of those guesthouses? u: which one has free wifi? s: both facilities do. u: is it high quality wifi with a stable connection? s: overall, customers thought the wifi at gonville is fast, while that at avalon may be slow and unstable. dialogue 1 u: i need to book a hotel that has 4 stars. s: if you'd like something cheap, i recommend gonville. u: does the cheaper one include wifi? s: yes, gonville offers free wifi. u: is it fast enough to watch online videos? s: yes, reviewers said that their wifi is stable and fast. ## problem formulation the sk-tod task contains the following three sub-tasks. knowledge-seeking turn detection aims to identify the user request that requires to be addressed with subjective knowledge. we regard it as a binary-classification problem, where the input is the dialogue context c and the output is a binary indicator. knowledge selection is then used to select the knowledge snippets that are relevant to the user's request. the inputs are the dialogue context c and the knowledge snippets candidates k, which is a combination of all the knowledge snippets of the relevant entities in e. the output k + ⊆ k is a subset of relevant knowledge candidates. note that there might be multiple knowledge snippets in k + . response generation is to create an utterance s t that responds to the user's request based on the dialogue context c and the relevant knowledge snippets k + . ## data we ground the data collection in multiwoz @xcite and select dialogues from the domains of hotels and restaurants. the data collection is conducted by a group of crowd workers through amazon mechanical turk as described in @xcite . the dataset contains 19,696 instances with subjective user requests and subjective-knowledgegrounded responses in total. the average length of the subjective user request and the agent response is 8.75 and 24.07 tokens, respectively. while most of the instances contain a single entity, there are 1,047 instances where multiple entities are compared (like dialogue 2 in figure 1 ). each instance requires on average 3.88 subjective knowledge snip- model output subjective knowledge turn detection knowledge selection response generation dialogue context tod based on domain api/db tod based on subjective knowledge pets. to help identify the subjective knowledgeseeking user request, we randomly sample another 18,383 dialogues with non-subjective user requests from the original multiwoz dataset. we split the dataset into training (75%), validation (10.8%), and test (14.2%) sets. ## baseline in this section, we describe the baseline method for sk-tod. figure 3 shows the pipeline of the baseline method which consists of three sequential sub-tasks. the details of each subtask are described as follows. they are illustrated in figure 3 . ## evaluation criteria each participating team submitted up to five system outputs each of which contains the results for all three tasks on the unlabeled test instances. we first evaluated each submission using the task-specific objective metrics by comparing to the ground-truth labels and responses. for knowledge-seeking turn detection and knowledge selection, we report the precision, recall, f 1 score, and accuracy score. for response generation, following the evaluation of other generation tasks, we employ bleu @xcite , rouge @xcite , and meteor @xcite to evaluate results compared to the reference responses. considering the dependencies between the tasks in the pipelined architecture, the final scores for knowledge selection and knowledge-grounded response generation are computed by considering the first step knowledge-seeking turn detection recall and precision performance, as follows: based on the overall objective score, we selected the finalists to be manually evaluated by the following two crowd sourcing tasks: for appropriateness, we only show the dialogue context and the responses. for accuracy, we further show the oracle knowledge snippets. to increase the annotation quality, we first ask workers to annotate the sentiment label of each knowledge snippet, and then evaluate the accuracy of each response. both measures are evaluated using the 5-point likert scale. in both tasks, we assigned each instance to three crowd workers and took their average as the final human evaluation score for the instance. those scores were then aggregated over the entire test set following equation 3, i.e., weighted by the knowledge-seeking turn detection performance. finally, we used the average of the appropriateness and accuracy scores to determine the official ranking of the systems in the challenge track. ## results we received 48 entries in total submitted from 14 participating teams. to preserve anonymity, the teams were identified by numbers from 1 to 14. ## conclusion we presented the official evaluation results of the task-oriented conversational modeling with subjective knowledge track in dstc11 this challenge track addressed the new conversational modeling tasks to accommodate subjective user requests into task-oriented dialogue systems. a total of 14 teams participated with an overall number of 48 entries submitted. one notable thing is that some teams have used the large language models (llms) such as openai's chatgpt and gpt-4 for enhancing data augmentation and refining response ranking. while these methods were expected to help to improve our task performances, it was not a decisive factor to achieve elevated scores and win the benchmark in both automatic and human evaluations. this suggests a future research direction exploring more effective strategies for utilizing llms for the tasks.