Distributed Marker Representation for Ambiguous Discourse Markers and Entangled Relations (2023)

Abstract: Discourse analysis is an important task because it models intrinsic semantic structures between sentences in a document. Discourse markers are natural representations of discourse in our daily language. One challenge is that the markers as well as pre-defined and human-labeled discourse relations can be ambiguous when describing the semantics between sentences. We believe that a better approach is to use a contextual-dependent distribution over the markers to express discourse information. In this work, we propose to learn a Distributed Marker Representation (DMR) by utilizing the (potentially) unlimited discourse marker data with a latent discourse sense, thereby bridging markers with sentence pairs. Such representations can be learned automatically from data without supervision, and in turn provide insights into the data itself. Experiments show the SOTA performance of our DMR on the implicit discourse relation recognition task and strong interpretability. Our method also offers a valuable tool to understand complex ambiguity and entanglement among discourse markers and manually defined discourse relations.

## introduction discourse analysis is a fundamental problem in natural language processing. it studies the linguistic structures beyond the sentence boundary and is a component of chains of thinking. such structural information has been widely applied in many downstream applications, including information extraction @xcite , long documents summarization @xcite , document-level machine translation @xcite , conversational machine reading @xcite . discourse relation recognition (drr) focuses on semantic relations, namely, discourse senses between sentences or clauses. such inter-sentence structures are sometimes explicitly expressed in natural language by discourse connectives, or markers (e.g., and, but, or). the availability of these markers makes it easier to identify corresponding relations @xcite , as is in the task of explicit discourse relation recognition (edrr), since there is strong correlation between discourse markers and relations. on the contrary, implicit discourse relation recognition (idrr), where markers are missing, remains a more challenging problem. prior work aims to address such challenges by making use of discourse marker information over explicit data in learning implicit discourse relations, either by injecting marker prediction knowledge into a representation model @xcite , or transferring the marker prediction task into implicit discourse relation prediction by manually defining a marker-relation mapping @xcite . it has been shown that discourse marker information can effectively improve relation prediction results. nevertheless, relatively little work has investigated various subtleties concerning the correlation between discourse markers and discourse relations, and their effect to idrr in further detail. to properly model discourse relations and markers, we need to consider that manually-defined discourse relations can be semantically entangled and markers are ambiguous. as shown in fig. 1 , for a pair of clauses, based on different emphasis on semantics, we have different choices on discourse relations and their corresponding markers. the existence of multiple plausible discourse relations indicates the entanglement between their semantic meaning. besides, discourse markers and relations do not exclusively map to each other. as an example, "ann went to the movies, and bill went home" (temporal.synchrony) and "ann went to the movies, and bill got upset" (contingency.cause) both use the marker and but express different meanings. identifying relations based on single markers are difficult in certain scenarios because of such ambiguity. thus, a discrete and deterministic mapping between discourse relations and markers can not precisely express the correlations between them. based on the study of above issues, we propose to use distributed marker representation to enhance the informativeness of discourse expression. specifically, we use a probabilistic distribution on markers or corresponding latent senses instead of a single marker or relation to express discourse semantics. we introduce a bottleneck in the latent space, namely a discrete latent variable indicating discourse senses, to capture semantics between clauses. the latent sense then produces a distribution of plausible markers to reflect its surface form. this probabilistic model, which we call dmr, naturally deals with ambiguities between markers and entanglement among the relations. we show that the latent space reveals a hierarchical marker-sense clustering, and that entanglement among relations are currently under-reported. empirical results on the idrr benchmark penn discourse tree bank 2 (pdtb2) @xcite shows the effectiveness of our framework. we summarize our contributions as follows: ## related work discourse analysis @xcite @xcite , targets the discourse relation between adjacent sentences. it has attracted attention beyond intra-sentence semantics. it is formulated into two main tasks: explicit discourse relation recognition and implicit discourse relation recognition, referring to the relation identification between a pair of sentences with markers explicitly included or not. while edrr has achieved satisfactory performance @xcite with wide applications, idrr remains to be challenging @xcite @xcite . our work builds upon the correlation between the two critical elements in discourse analysis: discourse relations and markers. discourse markers have been used for not only marker prediction training @xcite , but also for improving the performance of idrr @xcite and representation learning @xcite . prior efforts on exploring markers have found that training with discourse markers can alleviate the difficulty on idrc @xcite @xcite . compared to their work, we focus on a unified representation using distributed markers instead of relying on transferring from explicit markers to implicit relations. @xcite first extended the usage of markers to sentence representation learning, followed by @xcite ; @xcite which introduced principled pretraining frameworks and large-scale marker data. @xcite ; @xcite explored the possibility of connecting markers and relations with prompts. in this work, we continue the line of improving the expression of discourse information as distributed markers. ## conclusion in this paper, we propose the distributed marker representation for modeling discourse based on the strong correlation between discourse markers and relations. we design the probabilistic model by introducing a latent variable for discourse senses. we use the em algorithm to effectively optimize the framework. the study on our well-trained dmr model shows that the latent-included model can offer a meaningful semantic view of markers. such semantic view significantly improves the performance of implicit discourse relation recognition. further analysis of our model provides a better understanding of discourse relations and markers, especially the ambiguity and entanglement issues.