Where Are We in Discourse Relation Recognition? (2021)

Abstract: Discourse parsers recognize the intentional and inferential relationships that organize extended texts. They have had a great influence on a variety of NLP tasks as well as theoretical studies in linguistics and cognitive science. However it is often difficult to achieve good results from current discourse models, largely due to the difficulty of the task, particularly recognizing implicit discourse relations. Recent developments in transformer-based models have shown great promise on these analyses, but challenges still remain. We present a position paper which provides a systematic analysis of the state of the art discourse parsers. We aim to examine the performance of current discourse parsing models via gradual domain shift: within the corpus, on in-domain texts, and on out-of-domain texts, and discuss the differences between the transformer-based models and the previous models in predicting different types of implicit relations both inter- and intra-sentential. We conclude by describing several shortcomings of the existing models and a discussion of how future work should approach this problem.

## introduction discourse analysis is a crucial analytic level in nlp. in natural language discourse, speakers and writers often rely on implicit inference to signal the kind of contribution they are making to the conversation, as well as key relationships that justify their point of view. while early ai literature is full of case studies suggesting that this inference is complex, open-ended and knowledge-heavy (e.g., @xcite ; @xcite ), recent work on computational discourse coherence offers a different approach. take the following example from @xcite : she found it boring." this discourse shows the classic pattern of implicit information. the overall point is that alice had a negative opinion of the story: the underlying explanation is that the story was not interesting because it had no surprises. but given available lexical resources and sentiment detection methods, we can capture such inferences systematically by recognizing that they follow common general patterns, known as "discourse relations", and are guided by shallow cues. an example of an instance in which discourse analysis can produce insights that may be missed by employing other nlp methods is this example from @xcite , where without discourse relations it may be difficult to capture sentiment: this represents a concession relation according to both rhetorical structure theory and the penn discourse treebank (where it is notated as comparison.concession), resolving the incongruity of the first clause being negative and the second clause being positive by illustrating how the negative statement in the subordinate clause is reversed by the positive one in the main clause. the importance of discourse has led to active research based on predicting what coherence relations are present in text based on shallow information. the predicted relations are then used to draw inferences from the text. the value of predicting the semantic classes of coherence relations has been demonstrated in several applications, including sentiment analysis @xcite , machine comprehension @xcite , summarization @xcite @xcite , and predicting instructor intervention in an online course discussion forum @xcite . however, it is still the case that few works have so far found discourse relations as key features @xcite . we argue that one reason for this gap between theory and empirical evidence is the quality of the parsers exacerbated by the distributional shifts in the texts they need to apply to. the necessity of discourse research has resulted in several shared tasks @xcite and corpora development in multiple languages @xcite @xcite @xcite ). yet shallow discourse parsing is a very difficult task; more than 10 years after the introduction of the penn discourse treebank @xcite , performance for english implicit discourse relation recognition has gone from 40.2 f-1 @xcite to 47.8 @xcite , less than 8 percentage points; a similar story could be said about the relation prediction performance of rst parsers. such performance hinders the wider application of parsers. if downstream tasks are to use predicted relation senses, the data to which the systems are applied is typically different from their training data-the wall street journal (wsj) in a 3-year window-to varying degrees. this tends to further aggravate the low performance observed. as a result, often we find that adding parsed discourse relations into models are unhelpful. although domain difference is a recognized issue in shallow discourse parsing by existing work @xcite , we still have little understanding of the types of distributional shift that matter and by how much, even within one language. this position paper seeks to shed some light on our current state in discourse parsing in english. surprisingly, we found that parsers have some issues even within the same news source as the training set (wsj); the differences in accuracy were not significant between indomain and out-of-domain data for the qualitative examples that we looked at, although the distribution of errors tend to be different. this differs from other nlp tasks such as entity recognition, where training on data in the target domain increased the f1 score by over 20 points @xcite . we further found that parsers perform differently on implicit discourse relations held within vs. across sentences. we believe these findings are strong evidence for the sensitivity of existing models to distributional shift in terms of both linguistic structure and vocabulary. additionally, as part of our evaluation, we asked linguists to perform manual annotation, which allowed us to evaluate the accuracy of these parsers on plain, unlabeled text, and gain some insight about the mistakes made by the parsers. during the annotation process, we uncovered information that can guide future research, including but not limited to the critical role of context for implicit discourse sense classification. we discuss this need for context, hypothesize what scenarios may cause two arguments to need additional context, and provide some examples for which this is the case. we urge future researchers to consider developing contextaware models for shallow discourse parsing moving forward. we release our dataset to facilitate further discourse analysis under domain shift. foot_0 2 related work there are various frameworks for studying inferential links between discourse segments, from local shallow relations between discourse segments in pdtb @xcite to hierarchical constituent structures in rst @xcite or discourse graphs in segmented discourse representation theory (sdrt) @xcite and the discourse graphbank @xcite . rhetorical structure theory (rst) @xcite provides a hierarchical structure for analyzing text that describes relations between text spans known as elementary discourse units (edus). the rst discourse treebank @xcite ) contains 385 wall street journal articles from the penn treebank @xcite which have been split into elementary discourse units and annotated according to rhetorical structure theory, where discourse relations are annotated in a tree structure across the whole document. a full list of these relations can be found in @xcite . the penn discourse treebank (pdtb) @xcite @xcite , which also uses penn treebank wall street journal articles, contains discourse relations annotated in a shallow, non-hierarchical manner. for each relation between two arguments, each argument and the discourse connective (word or phrase that indicates the discourse relation) are labeled. the pdtb also annotates whether a relation is explicit or non-explicit, the latter type of which has three subtypes: implicit, altlex, and entrel. in this paper, we focus on implicit relations, where a connective can be inserted between the two arguments that indicates a discourse relation. these relations are considered extremely challenging for discourse parsers to automatically identify. there is a need to examine the performance of the proposed discourse parsers, their representational choices, their generalizability, and interpretability both across domains, distributions, and frameworks. one recently developed framework is the pdtb-3. since its release in 2019, several papers have evaluated the performance of implicit sense classifiers on this new corpus, which includes newly annotated intra-sentential implicit discourse relations. in addition to proposing a new evaluation framework for pdtb, @xcite evaluate the performance of pretrained encoders for implicit sense classification on the pdtb-2 and the pdtb-3. @xcite identify locating the position of relations as a new challenge in the pdtb-3, due to the significantly increased number of intra-sentential implicit relations annotated. techniques of discourse parsing range from supervised @xcite @xcite @xcite and weakly supervised and unsupervised approaches @xcite @xcite ; recent developments such as word/contextual embeddings have improved parser performance, although not as significantly as other tasks @xcite yet most works have made simplifying assumptions concerning the linguistic annotations for practical purposes that affect their evaluation and generality. for instance, most shallow discourse parsers use only the argument pairs to determine the discourse sense without considering further context. additionally, in rst parsing, standard practice involves classifying only the 18 top-level rst classes @xcite @xcite . thus, all elaboration relations are lumped together, making it a huge class. we reveal findings about these assumptions in section 4. other works evaluating discourse parsers include discoeval @xcite , a test suite of evaluation tasks that test the effectiveness of different sentence encoders for discourse parsers, and an im-proved evaluation protocol for the pdtb-2 @xcite . in contrast, our work aims to analyze and evaluate existing discourse parsers via gradual domain shift. we provide a comparative genrebased analysis on distributionally shifted text data and present a qualitative analysis of the impact of the practical choices that these models make while doing discourse parsing across frameworks. we then evaluate the performance of three widely used or state-of-the-art models under gradual shift of the domain of texts, noting that users who would want to use a parser will be applying it on data that varies linguistically to different degrees from the parser's training data (a fixed 3-year window of wsj articles). the data we examine is: wsj texts outside of the penn treebank , other news texts, and the gum corpus @xcite . note that none of these texts contain gold pdtb annotations, and only the gum corpus contains gold rst annotations. ## insights for model development while inspecting the results of the annotations, we found several helpful phenomena for developing future models, including observations regarding the role of context in shallow discourse parsing and errors that current rst parsers are making. ## conclusion and future work discourse parsing for text has seen a recent surge in experimental approaches. in this work we presented a detailed analysis of the performance of the state of the art discourse parsers and analysed their weaknesses and strength. the conclusions drawn above from these experiments make it clear that discourse parsing, though it has come a long way in the past decade or so, still has a long way to go, particularly with respect to parsing on out-ofdomain texts and addressing issues of class imbalances, although the bert-based model has made some improvements in this area. additionally, we investigated how and when pdtb-3 can help in improving the prediction of intra-sentential implicit relations. there are several promising future directions for the area of discourse parsing. a model that detects intra-sentential implicit relations is necessary in order to be able to parse on the pdtb-3. exploring new neural parsing strategies is also a must. we observed that neural parsers are ignorant about what they do not know and overconfident when they make uninformed predictions. quantifying prediction uncertainty directly by training the model to output high uncertainty for the data samples close to class boundaries can results in parsers that can make better decisions. one takeaway of our empirical analysis was the importance of the role of context in identifying the correct discourse relations. this observation suggests the need for new computational experiments that can identify the right context window that is required for the model to accurately predict relations. another useful direction is designing models that can learn discourse relations on their own without the help of annotated corpora. there are several unsupervised models @xcite ) that are used for determining the structure of discourse parse trees but few that infer the relations themselves.