Signals as Features: Predicting Error/Success in Rhetorical Structure Parsing (2024)

Abstract: This study introduces an approach for evaluating the importance of signals proposed by Das and Taboada in discourse parsing. Previous studies using other signals indicate that discourse markers (DMs) are not consistently reliable cues and can act as distractors, complicating relations recognition. The study explores the effectiveness of alternative signal types, such as syntactic and genre-related signals, revealing their efficacy even when not predominant for specific relations. An experiment incorporating RST signals as features for a parser error / success prediction model demonstrates their relevance and provides insights into signal combinations that prevents (or facilitates) accurate relation recognition. The observations also identify challenges and potential confusion posed by specific signals. This study resulted in producing publicly available code and data, contributing to an accessible resources for research on RST signals in discourse parsing.

## introduction discourse parsing has sparked significant interest in recent nlp applications. this task goes beyond the conventional scope of sentences and may extend to encompass the identification of coherence relations (relations between segments of text) at the discourse level. one of the most popular formalisms for representing coherence relations is rhetorical structure theory (rst; @xcite , which has spurred the construction of various datasets that are now used for hierarchical discourse parsing. this last task is challenging and discourse parsers have not achieved the same level of success as other tasks at the sentence level. moreover, analyzing failure cases, especially in deep learning-oriented parsers, proves difficult. concurrently, research on coherence relations has also been struggling with identifying the exact @xcite . they are encoded in a binary feature vector. linguistic elements that signal them. at the discourse level, a diverse array of signals may occur, making it challenging to discern typical signals for specific relations and their underlying motivation. this has been addressed in the work of @xcite , where the author provides a comprehensive overview of signals present in the rst-dt dataset @xcite and subsequently annotate the signals at play for every relation in this corpus. this process ultimately has resulted in the development of the rst signaling corpus @xcite . see the appendix for a comprehensive list of individual signals and signal types for all relations in this corpus. in the present paper, our aim is to assess the relevance of das and taboada's signals in rst discourse parsing and understand how they contribute to the errors or success of a state-of-the-art parser. we first describe an experimental set up where we replicate @xcite s dmrst discourse parser @xcite which achieves state-of-the-art results for coherence relation recognition and then align the results with the rst signaling corpus. we then show that although discourse markers (dms) are prevalent in various sorts of relations, they are not necessarily effective signals (see, for example, the dm when for the temporal relation in example (1)) , unlike other types of signal such as syntactic signals (for example, the nominal modifier in example ( 2 )) or genre signals. foot_0 (1) "[representing investor clubs from around the u.s were attending] pred:background ← --------gold:temporal [when the market started to slide friday.]" wsj_2686 (2) "[negotiable, bank-backed business credit instruments] pred:elaboration ←-------gold:elaboration [typically financing an import order.]" wsj_0602 next, we validate the initial analysis by incorporating these signals into a model, using them as features for a predictive model that distinguishes between errors and successes of the dmrst parser (see the flow diagram in figure 1 ). we first observe that these signals could serve as relevant features in the context of discourse parsing before delving into a more detailed analysis of the signals influencing the parser's predictions of errors or success. ## predictive model for success/error analysis in this section we aim to provide a deeper insight of the previous analysis by building an error / success prediction model. our goal here is to utilize signals from das and taboada's signaling corpus to predict whether the dmrst parser will encounter an error or not. this approach enables us to assess the utility of signals in discourse parsing and determine if the presence or absence of these signals is linked to errors or successful parsing outcomes. the implementation of our predictive model for error/success analysis is based on the xgboost algorithm @xcite . this ensemble gradient boosting approach is renowned for its high accuracy. it has the capability to capture arbitrary interactions among features and is well-regularized to avoid overfitting. the present experiment consists in training an xgboost model to predict the dmrst parsing errors, the predicted label set being {1,0} where 1 is a correctly predicted error or successful parse and where 0 is where our model fails. the signals from the signaling corpus are encoded in a binary feature vector. with this configuration we train xgboost on the 2306 relations outputs by the dmrst parser and get an 0.78 accuracy for a randomly selected 761 relations test set. figure 2 presents an analysis of feature importance using classification gain which is often used to estimate feature importance @xcite . ## conclusion we have presented an approach for assessing the importance of das and taboada's signals within the context of discourse parsing. our initial obser subsequently, we conduct an experiment incorporating the modeling of rst signals as features for an parser error or parser success prediction model. the results demonstrate the relevance of