Joint Reasoning on Hybrid-knowledge sources for Task-Oriented Dialog (2023)

Abstract: Traditional systems designed for task oriented dialog utilize knowledge present only in structured knowledge sources to generate responses. However, relevant information required to generate responses may also reside in unstructured sources, such as documents. Recent state of the art models such as HyKnow (Gao et al., 2021b) and SEKNOW (Gao et al., 2021a) aimed at overcoming these challenges make limiting assumptions about the knowledge sources. For instance, these systems assume that certain types of information, such as a phone number, is always present in a structured knowledge base (KB) while information about aspects such as entrance ticket prices, would always be available in documents. In this paper, we create a modified version of the MutliWOZ-based dataset prepared by (Gao et al., 2021a) to demonstrate how current methods have significant degradation in performance when strict assumptions about the source of information are removed. Then, in line with recent work exploiting pre-trained language models, we fine-tune a BART (Lewiset al., 2020) based model using prompts (Brown et al., 2020; Sun et al., 2021) for the tasks of querying knowledge sources, as well as, for response generation, without makingassumptions about the information present in each knowledge source. Through a series of experiments, we demonstrate that our model is robust to perturbations to knowledge modality (source of information), and that it can fuse information from structured as well as unstructured knowledge to generate responses.

## introduction most existing work on task-oriented dialog systems assumes that the knowledge required for completing a task (eg: booking a restaurant reservation), resides in structured knowledge sources. thus, typical task-oriented dialog systems require generating a belief state, that can be used to query a knowledge base to fetch entity results; these results are then used to generate responses. recognizing that information is not always present in structured resources, recently methods that can additionally use unstructured knowledge (eg: document collections), have also been developed @xcite . however, current state-of-the-art models designed for such tasks make limiting assumptions about the nature of knowledge sources, that make them unsuitable for use in real-world settings. limitations of existing methods: first, current task-oriented dialog systems designed to reason over hybrid knowledge sources assume that a knowledge base and the unstructured knowledge source encode separate pieces of information about entities (eg: the zip-code is always in structured knowledge, ticket prices are always available in unstructured text) @xcite . this is not reflective of real-world knowledge, where independent information systems are often fused to enable applications. second, existing systems are trained to learn the source of different pieces of information, thus, making them unsuitable for situations where any field that was previously in a structured knowledge source is now available in an unstructured knowledge source @xcite . in effect, a simple change in the modality of information can result in a failure of the model to utilize the information present in knowledge, as existing models memorize the source of every piece of information. third, such systems assume that each knowledge grounded response can contain information from only one source type @xcite @xcite -either structured or unstructured knowledge. this is an artificial constraint imposed to make modelling easier, but realworld conversations can routinely require systems to fuse information from more than one knowledge type (eg: see dialog turn 4 in figure 1 ). contributions: in this paper, we present our work aimed at removing each of these strict assumptions from task-oriented dialog systems. current methods for joint-reasoning in task oriented dialogs have been developed using an augmented version of multiwoz 2.1 which contains additional dialog turns based on new unstructured information @xcite . unfortunately, no attempt has been made to distribute information across knowledge sources. we therefore create a modified version of this dataset (called hybridtod) that optimally redistributes information across structured and unstructured knowledge so that most dialogs in the train dataset are affected by this change. a trivial method of redistributing information across structured and unstructured knowledge sources would be to arbitrarily move structured fields for some entities to the unstructured knowledge source. however, since the universe of entities in the dataset is very large and not all entities are directly referred to in the dialogs, such a method of redistributing information may not be as effective if the dialogs do not use the slot-values that have been redistributed. we therefore, develop an automated graph based approach which uses the max-cut of the graph to optimally redistribute infor-mation from structured to unstructured knowledge sources. lastly, in line with recent work exploiting pre-trained language models, we fine-tune bart @xcite using prompts for the tasks of querying knowledge as well as response generation without making assumptions about the information present in each knowledge source. specifically, we do prompt+lm finetuning @xcite in which both the prompt and model parameters are trainable @xcite @xcite . through a series of experiments, we demonstrate that our model is robust to perturbations to knowledge modality (source of information), and it can fuse information from structured as well as unstructured knowledge to generate responses. in summary we make the following contributions foot_0 : (1) we prepare a new version of the multiwoz-dstc9 combined dataset @xcite 2 related work modeling task oriented dialogs: multiple flavours of this problem have been defined to address different aspects of modeling -eg: belief state tracking to assess whether a model is able to correctly decode the query needed given a current conversational context @xcite @xcite , generating responses given belief states to assess whether a model is able to correctly predict the knowledge attributes to be used in a response @xcite @xcite , end-to-end modeling of dialog systems where models are assessed on the correctness of the response generated including the values used from the knowledge base @xcite , etc. recent work that assumes that belief state annotations are latent and not available for training have also been developed @xcite . knowledge grounded dialog: dialog systems that generate responses on information grounded in external knowledge have also been developed. unlike, work on task oriented dialogs, which primarily focuses on using structured knowledge to complete a 'goal' or accomplish a 'task' (eg: poi recommendation for in car navigation @xcite , restaurant, hotel or flight booking @xcite , etc), most existing knowledge grounded systems are designed to address informational needs of users (eg: answering queries based on collections of documents, making response recommendations to contact center agents). finally, contemporaneous to our work, knowledge grounded response generation tasks that combine information from hybrid knowledge sources have also been proposed @xcite . here, unlike task oriented dialog systems, which require the retrieval of an entity to make recommendations or accomplish a task, in such tasks, the goal is to answer an informational seeking query in a chit-chat conversation. models are required to use the dialog context to fetch related tables (often flattened and encoded as independent table cells), along with documents to generate a response. ## the hybridtod dataset the dataset prepared by @xcite (referred to as the seknow-multiwoz dataset in this paper) is the only publicly available taskoriented dialog dataset in which the dialogs are grounded on two types of knowledge sources: structured and unstructured (faqs). however, seknow-multiwoz is not indicative of a realworld setting due to two major limitations: (1) it has a strict, slot-type to knowledge-source type mapping. for example, the slot-type 'cuisine' is always in the structured source while 'timings' of operation would always be mentioned in unstructured documents, and (2) an agent response contains information from only one source (i.e., either from structured or unstructured). to alleviate these limitations, we systematically modify the knowledge sources in seknow-multiwoz to construct a new dataset that we refer to as hybridtod. ## jointlm the problem of utilizing information and responding to users in task-oriented dialogs can be broken down into parts: (i) querying knowledge source (structured and/or unstructured) to return entities (ii) generating responses (eg:sharing information about entities, requesting for more details from the user, etc). we represent the dialog context as @xmath0, r 1 , ..., u n ), where (u i , r i ) represent the user and the system response utterance at i th turn respectively. we represent the entity e required for generating the response as the concatenation of its slot-values (from structured kb), represented as e s , and faqs from the unstructured knowledge source, represented as e us : ## experiments our experiments are aimed at answering the following questions: (1) how does jointlm perform compared to the baseline when trained and tested on hybridtod? (2) how does the change in slot-value distribution across structured and unstructured sources affect the performance of the models? (3) is joint training of promtlm for the two tasks of entity retrieval and response generation helpful? (4) how does jointlm compare with natural baselines for entity retrieval? experimental setup: task oriented dialog systems have to identify relevant entities (e.g. restaurants) from associated knowledge sources needed to generate a response. in order to identify these relevant entities, existing datasets provide the belief state annotations during training. additionally, in our work for each dialog context, we associate a set of (positive) entities that exactly match the requirements present in the dialog context and a set of (negative) entities that do not match by an automated method. note that the text snippets in the unstructured corpus do not have any annotations. for all of our experiments, we use bart @xcite encoder-decoder based language model and finetune the pretrained model on the three datasets i.e, seknow-multiwoz @xcite , hybridtod and unstructured-tod datasets. baseline: we use the current state-of-the-art model for joint reasoning, seknow @xcite model as our baseline. seknow is designed to use belief state annotations -specifically, seknow is trained to generate the belief state given the dialog context. these belief states are then used to query the knowledge sources and generate a delexicalised response using the context and the generated belief state. the slot-values in the delexicalised response are then populated using an unordered set of entities returned by the belief state query on the structured knowledge source. ## conclusion in this paper we presented a new dataset, hybrid-tod that requires reasoning over both structured and unstructured knowledge sources to generate responses to dialogs. unlike existing task-oriented dialog datasets, it does not restrict slot-types to specific knowledge sources. through our experiments we demonstrated how existing methods do not adapt well to changing distributions of slottype sources and that our model jointlm (trained using entity annotations rather than belief state), not only generates better responses by reasoning over both knowledge sources, it also learns a better retriever for entities. in future work, we also plan to train our models without using any annotations i.e without any supervision on entity label information.