Discourse Relation Prediction and Discourse Parsing in Dialogues with Minimal Supervision (2024)

Abstract: Discourse analysis plays a crucial role in Natural Language Processing, with discourse relation prediction arguably being the most difficult task in discourse parsing. Previous studies have generally focused on explicit or implicit discourse relation classification in monologues, leaving dialogue an under-explored domain. Facing the data scarcity issue, we propose to leverage self-training strategies based on a Transformer backbone. Moreover, we design the first semi-supervised pipeline that sequentially predicts discourse structures and relations. Using 50 examples, our relation prediction module achieves 58.4 in accuracy on the STAC corpus, close to supervised state-of-the-art. Full parsing results show notable improvements compared to the supervised models both in-domain (gaming) and cross-domain (technical chat), with better stability.

## introduction discourse analysis aims at uncovering the inherent structure of documents, where spans of textknown as elementary discourse units (edus)are linked by semantic-pragmatic relations such as explanation, acknowledgment, contrast, etc. discursive information is useful in various downstream applications, from sentiment analysis or fake news detection @xcite , to summarization or machine translation @xcite . current data-driven methods for discourse parsing have predominantly been applied to monologues, leading to limited availability and generalizability of discourse parsers for dialogues. as dialogue data soared in all kinds of forms, the need for automatic analysis systems has rapidly increased. here, we propose to tackle the crucial problem of discourse relation identification in dialogues, and show performance of a full discourse parser that could enhance these applications. discourse relation classification labels the arcs in a discourse graph and is considered the most difficult part in discourse parsing: it is a multi-way classification task involving class imbalance and information at varied levels, from morpho-syntactics, to semantics, pragmatics and world knowledge. discourse relations are often split into explicittriggered by connectives (e.g. because, while...) thus allegedly easier to classify -, and implicit, without such markers. however, this distinction is not annotated in dialogue corpora. we thus cast the task as identifying all relations, which also makes for a more practical scenario as in disrpt shared task @xcite . one of the main hurdles in developing highfunctioning parsing models is the scarcity of annotated data, along with limitations of supervised approaches in cross-domain scenarios @xcite . strategic conversations corpus (stac) @xcite -the most commonly used dialogue dataset annotated using the segmented discourse representation theory (sdrt) @xcite -contains merely 1000 short documents. the labeling effort being expensive in terms of time and labor costs, it appears unlikely to create new large-scale expert-annotated datasets. semisupervised strategies are thus appealing. a few studies proposed weak or distant supervision for naked structure building @xcite while missing the important relation information. remarkably, despite recent powerful large language models (llms) such as chat-gpt excel in many nlp tasks, discourse parsing remains a significant challenge, given their poor performance @xcite . in this paper, we extend the bootstrapping approach to dialogues with even less annotated data, by relying on self-training @xcite where a model is used to produce pseudo labels and increase training data, a simple method shown as effective @xcite . using the bert model @xcite as a base classifier and applying self-training, we achieve competitive results on a 16-way classification on stac using only 50 dialogues for initial training. we also build a pipeline upon @xcite 's work to perform full parsing, where we assign discourse relations on established structures, giving important extensions on semi-supervised approaches for dialogues until now limited to naked structures. our pipeline yields 38.6 micro-f 1 score with gold edus and 32.8 with predicted edus: representing strong baselines for discourse parsing in dialogues with minimal supervision. this pipeline, or structurethen-relation approach, allows for a flexible architecture and greater generalizability. we further conduct cross-domain experiments by testing on a re-annotated subset of molweni @xcite )a ubuntu dataset. despite the domain difference, our pipeline shows remarkable performances (link 75.6, link and relation 31.2), outperforming supervised sota models by a large margin foot_0 . to summarize: we propose (1) a simple and effective method that requires minimal supervision for discourse relation prediction; ## related work discourse relation prediction as an individual task receives rich attention, mostly conducted on the penn discourse treebank (pdtb) @xcite . semi-supervised models have been mostly limited to implicit relation identification relying on synthetic data @xcite or translations @xcite . these methods create pseudo-labeled data by using expert-composed rules or convenient linguistic resources: both in short for dialogues. the more recent effort utilizes pre-trained language models (plms) @xcite as backbones as they show superior performance for many classification tasks. plms have also been used as reliable classifiers to produce pseudo labels in self-training scenarios @xcite . through prompt adaptation, @xcite reveal that implicit relation prediction is still a tricky task, even for chatgpt. in recent years, there has been an increasing interest in discourse parsing in dialogues. a range of discourse parsers has emerged, including classic statistical models @xcite and neural architecture models @xcite @xcite , some are trained within multi-task learning framework @xcite . although these supervised models achieve good performance on stac corpus, they face limitations when applied to cross-domain scenarios @xcite . to address the challenge of data scarcity, researchers turn to weakly and semi-supervised methods @xcite @xcite . @xcite show that co-training can considerably increase cross-domain performance on monologues, but they benefit from a larger amount of annotated data than we do for dialogues. despite the revolutionary achievements offered by llms @xcite , they remain notably behind fully and semi-supervised benchmarks in discourse parsing. @xcite illustrate that chatgpt struggles on stac with 50% f 1 gap from supervised models. @xcite find that chatgpt tends to establish discourse structures in a linear fashion. while in-context learning methods are helpful, their enhancement is limited. ## discourse parsing pipeline a standard full discourse parsing involves three tasks: edu segmentation, link attachment, and relation prediction (figure 1 ). most previous work applies a structure-then-relation approach @xcite @xcite . we follow the pipeline by providing relations on the established discourse structures. ## molweni re-annotation to evaluate the cross-domain adaptability of our parsing pipeline, we release a newly annotated dataset, "molweni-clean", sourced from the molweni corpus @xcite . molweni contains 10, 000 sdrt-annotated documents from the ubuntu chat corpus @xcite . however, it presents heavily redundant documents and inconsistent annotations @xcite , making the results less reliable. therefore, we revised the annotation of a subset of molweni to ensure a more robust evaluation (test only). ## relation prediction module 6.1 self-training results results for relation prediction are presented in with a significant amount of unlabelled data, the self-training process can be repeated multiple times. however, limited by the data size in stac, we can only test iterative learning with few values, @xmath2, 1800, 2800]. we define a stopping cri-5 https://openai.com/blog/chatgpt . ## conclusion in this study, we introduce a substantial extension to semi-supervised discourse parsing in dialogues by incorporating relation predictions into the established naked structures. we define simple yet effective sample selection strategies in self-training, achieving sota results with a minimal training set. importantly, the efficacy of our discourse parsing pipeline is fully demonstrated across indomain and cross-domain settings. we also contribute a small expert-annotated discourse dataset, along with semi-supervised benchmarks for subsequent comparisons. future work should explore the use of more out-of-domain raw data and investigate bootstrapping methods for relation prediction, while also improving on structure prediction, possibly with the same strategies.