Discourse Relation Embeddings: Representing the Relations between Discourse Segments in Social Media (2022)

Abstract: Discourse relations are typically modeled as a discrete class that characterizes the relation between segments of text (e.g. causal explanations, expansions). However, such predefined discrete classes limit the universe of potential relationships and their nuanced differences. Adding higher-level semantic structure to contextual word embeddings, we propose representing discourse relations as points in high dimensional continuous space. However, unlike words, discourse relations often have no surface form (relations are in between two segments, often with no word or phrase in that gap) which presents a challenge for existing embedding techniques. We present a novel method for automatically creating discourse relation embeddings (DiscRE), addressing the embedding challenge through a weakly supervised, multitask approach to learn diverse and nuanced relations in social media. Results show DiscRE representations obtain the best performance on Twitter discourse relation classification (macro F1=0.76), social media causality prediction (from F1=0.79 to 0.81), and perform beyond modern sentence and word transformers at traditional discourse relation classification, capturing novel nuanced relations (e.g. relations at the intersection of causal explanations and counterfactuals).

## introduction relations between discourse segments (i.e., phrases rooted by a main verb phrases or clauses) have mostly been studied as discrete classes; most notably penn discourse treebank (pdtb) @xcite and rhetorical structure theory discourse treebank (rst dt) @xcite contain 43 and 72 types of discourse relations respectively. at the same time, such work has taken place over newswire, the domain of both the pdtb and rst. with many different relation classes over sophisticated schema, annotation is non-trivial prohibiting extensive development in new domains (e.g., social media). thus, progress in developing, training and evaluating discourse relation identifiers has happened over discrete-class models with labeled newswire corpora @xcite @xcite @xcite . to address this challenge and enable expansion of discourse work to social media, we propose a weakly supervised learning method which does not require any explicit labels. instead, it adds a semantic structure that can effectively capture various types of discourse relations, even in other domains leveraging a multitask learning method called "discourse relation embeddings (discre)". our discre model represents discourse relations as continuous vectors rather than single discrete classes. as the first study of embedding discourse relations into high dimensional continuous spaces, we mainly focus on social media. social media is a challenging domain because it contains many acronyms, emojis, unicode, and informal variations of grammatical structure, but its personal nature provides diverse and psychologically-relevant dis-course patterns which are not often found from newswire text. according to our best knowledge, there are only relatively small datasets for specific types of discourse relations for causal relation @xcite and counterfactual relations @xcite , but they are not diverse and large enough to learn general discourse relations. in this paper, we propose a novel weakly supervised learning method for deriving discourse relation embeddings on social media. we created a social media discourse relation dataset and validated our new approach. furthermore, we conducted visual investigations on continuous discourse relation spaces and thorough qualitative analysis on the behaviors of discre in both pdtb and social media. then, we also validated how well our learning method can generalize across different domains by applying discre as transfer learning features for discourse relation downstream tasks. our contributions include: (1) a novel model structure which, when weakly supervised, creates embeddings capturing discourse relations (dis-cre), (2) the creation of new twitter discourse relation dataset and the validation of our approach for the discourse relation classification on the dataset, (3) quantitative and qualitative evaluation of dis-cre on ptdb and downstream social media discourse relation tasks in which discre outperformed strong modern contextual word and sentence embeddings, obtaining a new state-of-the-art performance for causality and counterfactuals, and (4) the release of all of our datasets and models. ## related work our work builds on previous studies in discourse relations with two key distinctions: (1) the predominant set of work on discourse relations has focused on annotated newswire datasets (pdtb and rst dt) rather than social media; (2) work to improve discourse parsing has focused either on feature engineering or models for better predicting predefined discourse relations rather than embeddings (or latent relations). such work takes pre-segmented clauses as input @xcite or builds full end-to-end discourse parsers @xcite . @xcite looked into adapting bert for relation classification by pretraining with domain text and connective prediction. other methods have zeroedin on implicit discourse relations (those without a connective token) and also used a hierarchical model but for discourse classification rather than embedding @xcite . some work from @xcite ; @xcite ; @xcite leverage cnns and graph networks to capture relationships between adjacent discourse units for implicit discourse relation classification. some have studied single discourse relations over social media. @xcite used a hybrid rule-based and feature based supervised classifier to capture counterfactual statements from tweets. @xcite and @xcite applied rst discourse parsing to social media movie review sentiment analysis, showing a pretrained model which was optimized for rst dt, suffered from domain differences when it was run on different domains (e.g., legislative bill). @xcite developed a causal relation extraction model using hierarchical rnns to parse social media. in general, hierarchical rnn-based models have worked well in general for capturing specific relations in social media and other discourse relations outside social media @xcite @xcite . our work is related to modern multi-purpose contextual word embeddings @xcite in the motivation to utilize latent representations in order to capture context-specific meaning. however, our model generates contextual discourse relation embeddings by learning probabilities rather than discrete labels and, it can learn all possible relations even from the same text leveraging posterior probabilities from well-established study @xcite . we also build on research that has assembled custom discourse relation datasets or created training instances from existing datasets using discourse connectives @xcite @xcite . @xcite designed an objective function to learn discourse relation categories (conjunction) based on discourse connectives along with other discourse coherence measurements while @xcite al. (2019) and sileo et al. (2019) used objectives to predict discourse connectives. here, we devised an objective function for learning posterior probabilities of discourse relations of the given discourse connectives, so the model can capture more fine-grained senses and discourse relation properties of the connectives foot_0 . also, all of them used sentence encoders to learn sentence representations and compared their learned representations with other state-of-the-art sentence embeddings such as infersent @xcite . however, our discre model learns a "discourse relation" representation (i.e. embedding) between discourse arguments rather than the representation of a respective text span of the pair (figure 1 ). finally, some have studied an rnn-attentionbased approach to multitask learning for discourse relation predictions in pdtb @xcite and a sentence encoder with multi-purpose learning for discourse-based objectives @xcite . also, liu et al. (2016) leveraged a multi-task neural network for discourse parsing across existing discourse trees and discourse connectives. @xcite used next sentence prediction to get better at implicit discourse relation classification. a particular challenge of these prior works has been to improve performance when no connective is explicitly mentioned in the text. all of these works utilized predefined discrete classes of possible discourse relations. while we were inspired and build on some of their techniques, our task is more broadly defined as producing vector representations of the relationship between discourse segments not limited to predefined discourse relations (whether defined with explicit connectives or conventional discourse signals exist or not) and is evaluated over a broad diversity of discourse relation tasks as well as downstream applications. ## methods the base for our model is a hierarchical birnn, following work on capturing causal relations in social media @xcite ), but we have added word-level attention, reflecting the necessity to keep word-level markers while parsing higherorder discourse relations (e.g., word pairs, modality, or n-grams) @xcite . ## results discre was validated on both newswire and social media discourse relation tasks. additionally, qualitative analysis on the discre representations were explore for both the domains. ## conclusion this paper suggests a difference in how semantics is modeled in nlp, moving beyond word-level embeddings to embeddings that capture the semantics of discourse relations. we explored a new task of creating latent discourse relation embeddings, designing a novel weakly supervised multitask learning method and evaluating it both quantitatively and qualitatively over social media and newswire domains. while we built on previous work over discourse relation classes, our results suggest the continuous discourse relation embeddings (discre) has certain benefits over manual categorizations. continuous representations of relations between segments of text have been relatively unexplored yet they can yield subtle attributes of discourse relations, yielding strong performance in applications and perhaps new organizations of functional discourse relations. our model obtained the best performance on the discourse relation classification tasks in both pdtb and our new twitter discourse dataset. our model also obtained a new state-of-the-art performance using discre in the social media causal relation prediction task. further, for predicting discourse relations over pdtb, we found discre achieved the higher performance than other embeddings, suggesting a focus on embedding relations can capture information not available in other types of modern embeddings which focus on representing particular word or phrase instances rather than their relationships. we release our dataset, code and pretrained models, for others to explore this new task, better develop continuous representations of discourse relations, as well as to extend discourse relation parsing beyond newswire to other domains.