Coherence-based Dialogue Discourse Structure Extraction using Open-Source Large Language Models (2024)

Abstract: Despite the challenges posed by data sparsity in discourse parsing for dialogues, unsupervised methods have been underexplored. Leveraging recent advances in Large Language Models (LLMs), in this paper we investigate an unsupervised coherence-based method to build discourse structures for multi-party dialogues using open-source LLMs fine-tuned on conversational data. Specifically, we propose two algorithms that extract dialogue structures by identifying their most coherent sub-dialogues: DS-DP employs a dynamic programming strategy, while DS-FLOW applies a greedy approach. Evaluation on the STAC corpus demonstrates a micro-F1 score of 58.1%, surpassing prior unsupervised methods. Furthermore, on a cleaned subset of the Molweni corpus, the proposed method achieves a micro-F1 score of 74.7%, highlighting its effectiveness across different corpora.

## introduction understanding multi-party dialogue structure is crucial for various natural language tasks like dialogue comprehension, summarization, and sentiment analysis @xcite @xcite . the goal is to extract a coherent discourse structure from a dialogue transcript, wherein pairs of clause-like texts are linked through rhetorical relations. to obtain a good understanding of the coherent discourse structures in dialogues, the segmented discourse representation theory (sdrt) @xcite framework proposes to annotate dialogues with dependency graphs, where edges link text spans labeled with semantic-pragmatic relations. an example dialogue derived from the strategic conversations corpus (stac) @xcite corpus is shown in figure 1 , with nodes denoting discourse units and edges relation types (i.e., question-answer pair (qap), question-elaboration (q-elab), contrast, and result). multi-party dialogues pose greater challenges compared to two-party dialogues, due to the involvement of numerous speakers, each contributing uniquely with more speech turn interactions and structural particularities @xcite . nevertheless, this complexity allows for the segmentation of dialogues into independent conversational flows that share a common overarching topic. these conversational flows exhibit distinct internal progression and structure, thereby permitting them to be regarded as sub-dialogues @xcite @xcite . as a result, the discourse structure of multi-party dialogues can be predicted by decomposing dialogues into coherent sub-dialogues, where each sub-dialogue reflects the flow of conversation, starting with an initial utterance and concluding when no further elaboration occurs. for instance, the dialogue in figure 1 comprises three sub-dialogues: (1, 2), (1, 3, 4, 5), and (1, 3, 4, 6). however, exploring all possible sub-dialogues to identify the coherent ones is unrealistic because it involves ana-lyzing all possible ordered sequences of utterances within a dialogue, leading to exponential growth that makes exhaustive analysis impractical. supervised evaluation of dialogues is challenging due to data sparsity @xcite . to address this issue, some studies have proposed unsupervised @xcite and semi-supervised @xcite @xcite methods. these methods typically predict the best discourse pair given a discourse unit, while overlooking the previous context. however, we advocate that identifying sub-dialogues can offer a broader context to better understand the thematic coherence within a dialogue, thus building a more accurate discourse structure. in this paper, we propose an unsupervised, subdialogue-oriented method for extracting "naked" discourse structures without discourse relations in multi-party dialogues. although without relations, discourse structures alone have been shown to be crucial features for tasks such as content selection @xcite and summarization @xcite . precisely, we introduce two algorithms: multi-party dialogue structure extraction based on dynamic programming (ds-dp) and multi-party dialogue structure extraction based on flow conversation analysis (ds-flow), designed to decompose dialogues into coherent sub-dialogues. ds-dp identifies the most coherent (partial) sub-dialogues ending in each discourse unit by applying a dynamic programming strategy. in contrast, ds-flow greedily predicts for each discourse unit the most likely coherent subsequent utterances, followed by a process that ensures the completeness of the resulting discourse structure. in both algorithms, we use perplexity as a metric to evaluate sub-dialogue coherence. to compute perplexity scores, we draw inspiration from work on pre-trained language models (plms) and large language models (llms) fine-tuned on conversational data implicitly capturing dialogue quality @xcite . we utilize open-source models, as proprietary models are limited to text-based prompts and do not permit analysis of output probabilities. in practice, we compare the performance of two open-source llms: a chatbot trained by fine-tuning llama on user-shared conversations vicuna-13b @xcite and a general-purpose model mistral-7b @xcite . we evaluate our method on the stac corpus @xcite and a revised subset of the molweni corpus @xcite . the results demonstrate the effectiveness of our solution, as it outperforms prior unsupervised methods. specifically, we achieve a micro-f 1 score of 58.1% on stac and 74.7% on molweni, demonstrating its robustness across different corpora. the contributions of this paper are twofold. first, we propose a fully unsupervised method for extracting graph structures of multi-party dialogues, which is the first of its kind to the best of our knowledge. second, we introduce and evaluate two novel algorithms that leverage open-source llms to decompose dialogues into coherent sub-dialogues, enabling a more fine-grained analysis of discourse structures. ## method in this section, we first formally describe the dialogue parsing task. we then describe two algorithms for sub-dialogue extraction. the first relies on a dynamic programming strategy and is formally denoted as multi-party dialogue structure extraction based on dynamic programming (ds-dp). conversely, the second algorithm, grounded in the analysis of conversation flows, is formally named multi-party dialogue structure extraction based on flow conversation analysis (ds-flow). ## experimental setup corpora we conduct experiments on two commonly used sdrt-annotated dialogue corpora: (i) stac @xcite . this corpus contains 1161 multi-party dialogues arising from interactions within an online version of the game "the settlers of catan". given the unsupervised nature of our method, we evaluate it on the test set, which consists of 109 documents, amounting to 1129 edu pairs. (ii) molweni @xcite . derived from the ubuntu chat corpus @xcite , this corpus centers around technical discussions concerning the ubuntu system. due to quality issues with the original annotations @xcite , we employ the "molweni-clean" version proposed by @xcite , which consists of 50 documents, encompassing 373 edu pairs. detailed corpus statistics are presented in evaluation metrics to assess the performance of the proposed approaches, we report the micro-f 1 , recall, and precision for the generated structures. ## discussion despite its significantly lower parameter count compared to the vicuna-13b model, we used the mistral-7b model for our algorithms' assessment owing to its superior performance relative to larger models like llama 2-13b @xcite and llama 1-34b @xcite ) across multiple benchmarks. although our algorithms exhibit polynomial complexity, employing exceedingly large models increases the computational time required for calculating perplexity in extended dialogues 5 . presently, the state-of-the-art lacks alternative unsupervised metrics with efficient time complexity for evaluating dialogue quality. metrics such as fed @xcite and full @xcite entail computing multiple log-likelihood values for 5 detailed insights into the algorithm execution times are provided in appendix f. dialogue assessment, contrasting with perplexity, which mandates the computation of a singular loglikelihood value foot_3 . we leave the comparison among these metrics for future investigations. ## conclusion and future work in this paper, we introduce an innovative, fully unsupervised method for extracting discourse structures in multi-party dialogues. to this end, we leverage open-source llms and introduce two algorithms, ds-dp and ds-flow, to detect coherent sub-dialogues within a dialogue. on the stac and molweni-clean corpora, we achieve micro-f 1 scores of 58.1% and 74.7%, respectively, demonstrating the efficacy of our solution in constructing dialogue structures without the need for labeled data. in the future, we intend to enhance the coherence evaluation metric, particularly addressing cognitive aspects as in @xcite , and explore applying llms for unsupervised prediction of rhetorical relation types to deduce full discourse structures. furthermore, we aim to improve algorithm link selection by incorporating linguistically motivated constraints as in @xcite . lastly, we plan to evaluate our architectural choices across diverse corpora and discourse parsing tasks to further validate their efficacy in assessing dialogues in real-world scenarios.