D ialog S titch: Synthetic Deeper and Multi-Context Task-Oriented Dialogs (2021)

Abstract: Real-world conversational agents must effectively handle long conversations that span multiple contexts. Such context can be interspersed with chitchat (dialog turns not directly related to the task at hand), and potentially grounded in a multimodal setting. While prior work focused on the above aspects in isolation, there is a lack of a unified framework that studies them together. To overcome this, we propose DialogStitch, a novel framework to seamlessly ‘stitch’ multiple conversations and highlight these desirable traits in a taskoriented dialog. After stitching, our dialogs are provably deeper, contain longer-term dependencies, and span multiple contexts, when compared with the source dialogs—all free of cost without any additional annotations! Though our framework generalizes to a variety of combinations, we demonstrate its benefits in two settings: (a) multimodal, imagegrounded conversations, and, (b) task-oriented dialogs fused with chit-chat conversations. We benchmark state-of-the-art dialog models on our datasets and find accuracy drops of (a) 12% and (b) 45% respectively, indicating the additional challenges in the stitched dialogs. Our code and data are publicly available.

## introduction task-oriented dialog agents have become increasingly popular in the recent years due to their ready deployment to several real-world applications. for such agents to be effective, they need to carryout long conversations spanning multiple contexts, interspersed with social chit-chat, and potentially grounded in multimodal settings. though prior works propose several datasets and task formulations to model these desired traits, we believe that they fall short on two counts. they either study these traits in isolation or in a simplified setting that does not cover the spectrum of requirements for real-world applications. the well-known task-oriented datasets multiwoz @xcite and google schema guided @xcite datasets contain only 13.4 and 20.4 turns respectively, on an average. while adequate for their intended purposes (e.g., find a restaurant or book a flight), these datasets do not support modeling task-oriented agents that need to go beyond and handle longer conversations (also argued by @xcite ). for instance, a real world customer service task might require conversations that last for hours, thus requiring more than 20 turns. as a step to bridge these gaps, we propose di-alogstitch, a novel framework that takes existing dialog dataset and creates dialogs that comparatively are longer, contain longer-term dependencies, and span multiple contexts. unlike existing works that either combine dialogs using human annotators @xcite , our framework imparts these desirable traits to task-oriented dialogs by using the available human annotations without collecting any additional ones and thus free of cost, due to its synthetic nature. as shown in fig. 1 , dialogstitch takes multiple dialogs and interleaves them carefully to ensure the resultant dialog is coherent, consistent, and more closely resembles the real-world scenarios. as the cherry on top, dialogstitch allows for the construction of dialog tasks analogous to the copying memory task @xcite , a synthetic task to benchmark model's capability to retain information over many time steps, i.e., modeling long-term dependencies. to summarize our contributions: • we propose dialogstitch, a novel framework to create task-oriented dialogs that are longer, contain longer-term dependencies, and handle multiple contexts by leveraging existing annotations. ## our approach consider a set of k dialogs {d i } 1:k where each dialog d i consists of n i turns with each turn t j @xmath0, s j i ) containing a user and a system utterance respectively. each dialog can also have a turn-independent foot_0 multimodal context m i , for example, an image in which the dialog is grounded. as shown in fig. 1 , dialogstitch interleaves di-alogs by inserting turns from one dialog into another. the exact strategy to interleave dialogs is domain-specific and uses the additional annotations accompanying the source datasets. however, care is taken to ensure that: (a) the user and system utterance in a turn are not separated, though the turns themselves are interleaved, (b) after stitching, the ordering among the turns in each dialog is preserved in the final dialog to avoid inconsistencies, and (c) no ambiguity (e.g., multiple referents for coreference, values for slots) results from this process of stitching. hence the resulting dialog is meaningful and coherent. ## stitching multimodal dialogs we showcase the ability of dialogstitch to handle and stitch dialogs with complex multi-round reasoning spanning across different multimodal contexts using the clevr-dialog dataset @xcite . clevr-dialog is a visuallysimple yet reasoning-wise complex visual dialog @xcite dataset, which contains a series of related question-answers pairs as dialog turns. these questions are grounded in an image, set in the abstract clevr world @xcite , and is made of spatially arranged objects (with shape, size, material, color attributes) against a plain background (see fig. 2a ). by design, dialogs in clevr-dialog have strong multi-turn dependencies. in addition, these dialogs also come with complete state annotations like type of question, objects/attributes of interest, and coreferences, for each turn. these two reasons make clevr-dialog a perfect testbed for dialogstitch. dialogstitch on clevr-dialog. each dialog d i in clevr-dialog starts off with a caption c i that partially describes the image, followed by 10 question-answer pairs (q j i , a j i ) 1:10 , as illustrated in fig. 1 . to align with our framework in sec. 2, we treat the caption as the first turn with an empty assistant utterance t 0 @xmath1, ∅), and the question-answer pairs as following turns, i.e., to stitch k different dialogs together, we: (a) identify the recaller questions that can help us recall their corresponding multimodal context (image) in the stitched dialog, using the question type annotations. these questions (with early tag) typically contain a reference to previously mentioned objects in the dialog, for example, 'what size is the earlier cylinder?'. refer @xcite for a full list of question types and tags in clevr-dialog. (b) breakdown each dialog into 2-3 chunks at randomly selected recaller question pivots. for each of these chunks, we note all the objects and attributes mentioned in the dialog so far. note that this is possible only due to the available annotations. (c) starting with the first chunk of a randomly selected dialog, we select a chunk from dialogs different from the one previous selected as a candidate. we then check for stitch compatibility by ensuring that there is no overlap of objects and attributes mentioned in both the stitched dialog and the candidate. if compatible, we append the candidate at the end and repeat the process, else discard and re-select a new one. note that when selecting chunks from a dialog, priority is given to the one that appear earlier. this ensures that the resultant stitched dialog respects the turn ordering from all model source ds (ours) vb-q 39.1 39.3 vb-qi 52.7 53.0 vb-qh 45.8 50.2 vb-qih 68.2 56.5 ## stitching open-domain dialogs being socially engaging is a desirable trait for task-orientated dialog agent as it facilitates a wider adoption in everyday applications. to achieve this, agents must additionally handle chit-chit about social topics. we emulate these scenarios to synthetically stitch task-oriented and open-domain dialogs. datasets. we adopt the parlai framework @xcite as a testbed for dialogstitch, since it grants a unified access to a vast repository of both open-domain and task-oriented dialog datasets. though dialogstitch is easily extendable to all these datasets within parlai, we consider the following datasets (see supp. for dataset statistics):