Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialogues (2023)

Abstract: Discourse processing suffers from data sparsity, especially for dialogues. As a result, we explore approaches to infer latent discourse structures for dialogues, based on attention matrices from Pre-trained Language Models (PLMs). We investigate multiple auxiliary tasks for fine-tuning and show that the dialogue-tailored Sentence Ordering task performs best. To locate and exploit discourse information in PLMs, we propose an unsupervised and a semi-supervised method. Our proposals thereby achieve encouraging results on the STAC corpus, with F1 scores of 57.2 and 59.3 for the unsupervised and semi-supervised methods, respectively. When restricted to projective trees, our scores improved to 63.3 and 68.1.

## introduction in recent years, the availability of accurate transcription methods and the increase in online communication have led to a vast rise in dialogue data, necessitating the development of automatic analysis systems. for example, summarization of meetings or exchanges with customer service agents could be used to enhance collaborations or analyze customers issues @xcite ; machine reading comprehension in the form of question-answering could improve dialogue agents' performance and help knowledge graph construction @xcite . however, simple surface-level features are oftentimes not sufficient to extract valuable information from conversations @xcite . rather, we need to understand the semantic and pragmatic relationships organizing the dialogue, for example through the use of discourse information. along this line, several discourse frameworks have been proposed, underlying a variety of annotation projects. for dialogues, data has been primarily annotated within the segmented discourse representation theory (sdrt) @xcite . discourse structures are thereby represented as dependency graphs with arcs linking spans of text and labeled with semantico-pragmatic relations (e.g. acknowledgment (ack) or question-answer pair (qap)). figure 1 shows an example from the strategic conversations corpus (stac) @xcite . discourse processing refers to the retrieval of the inherit structure of coherent text, and is often separated into three tasks: edu segmentation, structure building (or attachment), and relation prediction. in this work, we focus on the automatic extraction of (naked) structures without discourse relations. this serves as a first critical step in creating a full discourse parser. it is important to note that naked structures have already been shown to be valuable features for specific tasks. @xcite mentioned that they are the most reliable indicator of importance in content selection. @xcite ; @xcite on summarization, and @xcite on thread extraction, also demonstrated the advantages of naked structures. data sparsity has always been an issue for discourse parsing both in monologues and dialogues: the largest and most commonly used corpus annotated under the rhetorical structure theory, the rst-dt @xcite ) contains 21, 789 discourse units. in comparison, the largest dialogue discourse dataset (stac) only contains 10, 678 units. restricted to domain and size, the performance of supervised discourse parsers is still low, especially for dialogues, with at best 73.8% f 1 for the naked structure on stac @xcite . as a result, several transfer learning approaches have been proposed, mainly focused on monologues. previous work demonstrate that discourse information can be extracted from auxiliary tasks like sentiment analysis @xcite and summarization @xcite , or represented in language models @xcite and further enhanced by fine-tuning tasks @xcite . inspired by the latter approaches, we are pioneering in addressing this issue for dialogues and introducing effective semi-supervised and unsupervised strategies to uncover discourse information in large pre-trained language models (plms). we find, however, that the monologueinspired fine-tuning tasks are not performing well when applied to dialogues. dialogues are generally less structured, interspersed with more informal linguistic usage @xcite , and have structural particularities @xcite . thus, we propose a new sentence ordering (so) fine-tuning task tailored to dialogues. building on the proposal in @xcite , we propose crucial, dialogue-specific extensions with several novel shuffling strategies to enhance the pair-wise, inter-speech block, and inter-speaker discourse information in plms, and demonstrate its effectiveness over other fine-tuning tasks. in addition, a key issue in using plms to extract document-level discourse information is how to choose the best attention head. we hypothesize that the location of discourse information in the network may vary, possibly influenced by the length and complexity of the dialogues. therefore, we investigate methods that enables us to evaluate each attention head individually, in both unsupervised and semi-supervised settings. we introduce a new metric called "dependency attention support" (das), which measures the level of support for the dependency trees generated by a specific self-attention head, allowing us to select the optimal head without any need for supervision. we also propose a semi-supervised approach where a small validation set is used to choose the best head. experimental results on the stac dataset reveal that our unsupervised and semi-supervised methods outperform the strong last baseline (f 1 56.8%, sec. 4), delivering substantial gains on the com-plete stac dataset (f 1 59.3%, sec. 5.2) and show further improvements on the tree-structured subset (f 1 68.1%, sec. 6.3). to summarize, our contributions in this work are: (1) discourse information detection in pre-trained and sentence ordering fine-tuned lms; (2) unsupervised and semi-supervised methods for discourse structure extraction from the attention matrices in plms; (3) detailed quantitative and qualitative analysis of the extracted discourse structures. ## related work discourse structures for complete documents have been mainly annotated within the segmented discourse representation theory (sdrt) @xcite or the rhetorical structure theory (rst) @xcite , with the latter leading to the largest corpora and many discourse parsers for monologues, while sdrt is the main theory for dialogue corpora, i.e., stac @xcite and molweni @xcite . in sdrt, discourse structures are dependency graphs with possibly non-projective links (see figure 1 ) compared to constituent trees structures in rst. early approaches to discourse parsing on stac used varied decoding strategies, such as maximum spanning tree algorithm @xcite @xcite or integer linear programming @xcite . @xcite first proposed a neural architecture based on hierarchical gated recurrent unit (gru) and reported 73.2% f 1 on stac for naked structures. recently, @xcite adopted graph neural networks (gnns) and reported marginal improvements on the same test set (73.8% f 1 ). data sparsity being the issue, a new trend towards semi-supervised and unsupervised discourse parsing has emerged, almost exclusively for monologues. @xcite ) leveraged sentiment information and showed promising results in cross-domain setting with the annotation of a silver-standard labeled corpus. @xcite extracted discourse trees from neural summarizers and confirmed the existence of discourse information in self-attention matrices. another line of work proposed to enlarge training data with a combination of several parsing models, as done in @xcite ; @xcite ; @xcite . in a fully unsupervised setting, @xcite used similarity and dissimilarity scores for discourse tree creation, a method that can not be directly used for discourse graphs though. as for dialogues, transfer learning approaches are rare. @xcite investigated a weak supervision paradigm where expert-composed heuristics, combined to a generative model, are applied to unseen data. their method, however, requires domain-dependent annotation and a relatively large validation set for rule verification. another study by @xcite focused on cross-domain transfer using stac (conversation during online game) and molweni (ubuntu forum chat logs). they applied simple adaptation strategies (mainly lexical information) on a sota discourse parser and showed improvement compared to bare transfer: trained on molweni and tested on stac f 1 increased from 42.5% to 50.5%. yet, their model failed to surpass simple baselines. very recently, @xcite investigated bootstrapping methods to adapt bert-based parsers to out-of-domain data with some success. in comparison to all this previous work, to the best of our knowledge, we are the first to propose a fully unsupervised method and its extension to a semi-supervised setting. as pre-trained language models such as bert @xcite , bart @xcite or gpt-2 @xcite are becoming dominant in the field, bertology research has gained much attention as an attempt to understand what kind of information these models capture. probing tasks, for instance, can provide fine-grained analysis, but most of them only focus on sentencelevel syntactic tasks @xcite @xcite @xcite . as for discourse, @xcite and @xcite applied probing tasks and showed that bert and bart encoders capture more discourse information than other models, like gpt-2. very recently, @xcite introduced a novel way to encode long documents and explored the effect of different fine-tuning tasks on plms, confirming that pre-trained and fine-tuned plms both can capture discourse information. inspired by these studies on monologues, we explore the use of plms to extract discourse structures in dialogues. ## experimental setup datasets: we evaluate our approach on predicting discourse dependency structures using the stac corpus @xcite , a multi-party dialogue dataset annotated in the sdrt framework. for the summarization and question-answering fine-tuning tasks, we use publicly available hug-gingface models @xcite ) (see appendix f). for the novel sentence ordering task, we train bart model on the stac corpus and the dailydialog corpus @xcite . the key statistics for stac and dailydialog can be found in baselines: we compare against the simple yet strong unsupervised last baseline @xcite , attaching every edu to the previous one. furthermore, to assess the gap between our approach and supervised dialogue discourse parsers, we compare with the deep sequential model by @xcite and the structure self-aware (ssa) model by @xcite . metrics: we report the micro-f 1 and the unlabeled attachment score (uas) for the generated naked dependency structures. ## analysis 6.1 effectiveness of das we now take a closer look at the performance degradation of our unsupervised approach based on das in comparison to the upper-bound defined by the performance of the oracle-picked head. to this end, figure 4 shows the das score matrices (left) for three models with the oracle heads and das selected heads highlighted in green and yellow, respectively. these scores correspond to the global support strategy (i.e., h g ). it becomes clear that the oracle heads do not align with the das selected heads. making a comparison between models, we find that discourse information is consistently located in deeper layers, with the oracle heads (light green) consistently situated in the same head for all three models. it is important to note that this information cannot be determined beforehand and can only be uncovered through a thorough examination of all attention heads. while not aligning with the oracle, the top performing das heads (in yellow) are among the top 10% best heads in all three models, as shown in the box-plot on the right. hence, we confirm that the das method is a reasonable approximation to find discourse intense self-attention heads among the 12 × 16 attention matrices. ## conclusion in this study, we explore approaches to build naked discourse structures from plms attention matrices to tackle the extreme data sparsity issue in dialogues. we show sentence ordering to be the best fine-tuning task and our unsupervised and semisupervised methods for selecting the best attention head outperform a strong baseline, delivering substantial gains especially on tree structures. interestingly, discourse is consistently captured in deeper plms layers, and more accurately for shorter links. in the near future, we intend to explore graphlike structures from attention matrices, for instance, by extending treelike structures with additional arcs of high das score and applying linguistically motivated constraints, as in @xcite . we would also like to expand shuffling strategies for sentence ordering and to explore other auxiliary tasks. in the long term, our goal is to infer full discourse structures by incorporating the prediction of rhetorical relation types, all while remaining within unsupervised or semi-supervised settings.