D ialog2 F low: Pre-training Soft-Contrastive Action-Driven Sentence Embeddings for Automatic Dialog Flow Extraction (2024)

Abstract: Efficiently deriving structured workflows from unannotated dialogs remains an underexplored and formidable challenge in computational linguistics. Automating this process could significantly accelerate the manual design of workflows in new domains and enable the grounding of large language models in domain-specific flowcharts, enhancing transparency and controllability.In this paper, we introduce Dialog2Flow (D2F) embeddings, which differ from conventional sentence embeddings by mapping utterances to a latent space where they are grouped according to their communicative and informative functions (i.e., the actions they represent). D2F allows for modeling dialogs as continuous trajectories in a latent space with distinct action-related regions. By clustering D2F embeddings, the latent space is quantized, and dialogs can be converted into sequences of region/action IDs, facilitating the extraction of the underlying workflow.To pre-train D2F, we build a comprehensive dataset by unifying twenty task-oriented dialog datasets with normalized per-turn action annotations. We also introduce a novel soft contrastive loss that leverages the semantic information of these actions to guide the representation learning process, showing superior performance compared to standard supervised contrastive loss.Evaluation against various sentence embeddings, including dialog-specific ones, demonstrates that D2F yields superior qualitative and quantitative results across diverse domains.

## introduction conversational ai has seen significant advancements, especially with the rise of large language models (llms) @xcite @xcite . dialog modeling can be divided into opendomain dialogs and task-oriented dialogs (tod), figure 1 : example segment of the dialog sng1533 from the hospital domain of the spokenwoz dataset. actions are defined by concatenating the dialog act label (in bold) with the slot label(s) associated to each utterance. with the latter focusing on helping users achieve specific tasks @xcite . in tod, structured workflows guide agents in assisting users effectively. this paper explores the underexplored terrain of automatically extracting such workflow from a collection of conversations. extracting workflows automatically is crucial for enhancing dialog system design, discourse analysis, data augmentation @xcite , and training human agents @xcite . additionally, it can ground llms in domain-specific workflows, improving transparency and control @xcite . recent works have attempted to induce structural representations from dialogs using either ground truth annotation or ad hoc methods @xcite @xcite . we believe that models specifically pre-trained for this purpose could significantly advance the field. in task-oriented dialog (tod), dialog acts and slots are key concepts @xcite . dialog acts represent the speaker's communicative intent, while slots capture task-specific information. a dialog action encapsulates both the dialog act and its corresponding slots, enabling us to view dialogs as sequences of canonical steps that convey both communicative and informative functions (figure 1 ). motivated by this perspective, we propose embedding sentences into a latent space grouped by representative actions rather than solely by sentence semantics. similar to how aggregating action sequences from multiple dialogs reveals a common underlying workflow (figure 2 ), clustering sentence embeddings in this latent space could uncover common conversational steps, potentially revealing the underlying workflow. the main contributions of this work are threefold: (a) we consolidate twenty task-oriented dialog datasets to create the largest publicly available dataset with standardized action annotations; (b) we introduce a novel soft contrastive loss that leverages the semantic information of dialog actions to guide the representation learning process, outperforming standard supervised contrastive loss; and (c) we introduce and release dialog2flow (d2f), to the best of our knowledge, the first sentence embedding model pre-trained specifically for dialog flow extraction. ## training corpus we identified and collected 20 tod datasets from which we could extract dialog act and/or slot anno-tations, as summarized in ## experimental setup for training d2f we mostly follow the experimental setup of dse @xcite and tod-bert @xcite , using bert base as the backbone model for the encoder to report results in the main text. additional configurations are reported in the ablation study (appendix c) while implementation details are given in appendix b. ## similarity-based evaluation before the dialog flow-based evaluation, we assess the quality of the representation space geometry through the similarity of the embeddings representing different actions. we use the following methods as quality proxies: ## dialog flow extraction evaluation dialog flow extraction is an underexplored hardto-quantify and challenging task with nuances in definition. however, to evaluate embedding quality, we formally define the problem as follows: let u and a denote sets of tod utterances and actions, respectively. let u and a be sets of tod utterances and actions, respectively. let α : @xmath0. ## conclusions this paper introduced dialog2flow (d2f), embeddings pre-trained for dialog flow extraction grouping utterances by their communicative and informative functions in a latent space. d2f embeddings were trained on a comprehensive dataset of twenty task-oriented dialog datasets with standardized action annotations, released along with this work. future work will enhance d2f embeddings by exploring larger backbone models and advanced methods for sentence embeddings @xcite . we will also investigate more sophisticated techniques for extracting and representing dialog flows, such as using subtask graphs @xcite or adapting dependency parsing for although not the exact same graph as the reference, this graph still allows us to understand the common flow of the conversations with a similar degree of detail: first, the user and system greet each other (u0 and s6), then the user inform the reason of the call requesting the phone number of a department (u4), the agent may confirm the department (s7) or request more information (s4) before providing the phone number ( s2 ). the user may then either confirm the number (u3) or thank the system (u5). finally, the system asks if anything else is required (s5), to which the user may either finish the conversation (u6) or, more likely, thank the system (u2) before the system says goodbye (s0). complex dialog structures @xcite . additionally, potential applications include using d2f embeddings to ground llms in domain-specific flows for improved transparency and controllability @xcite , and integrating d2f embeddings into various tod downstream tasks like dialog state tracking and policy learning.