Neural RST -based Evaluation of Discourse Coherence (2020)

Abstract: This paper evaluates the utility of Rhetorical Structure Theory (RST) trees and relations in discourse coherence evaluation. We show that incorporating silver-standard RST features can increase accuracy when classifying coherence. We demonstrate this through our tree-recursive neural model, namely RST-Recursive, which takes advantage of the text’s RST features produced by a state of the art RST parser. We evaluate our approach on the Grammarly Corpus for Discourse Coherence (GCDC) and show that when ensembled with the current state of the art, we can achieve the new state of the art accuracy on this benchmark. Furthermore, when deployed alone, RST-Recursive achieves competitive accuracy while having 62% fewer parameters.

## introduction discourse coherence has been the subject of much research in computational linguistics thanks to its widespread applications @xcite . most current methods can be described as either stemming from explicit representations based on the centering theory @xcite , or deep learning approaches that learn without the use of hand-crafted linguistic features. our work explores a third research avenue based on the rhetorical structure theory (rst) @xcite . we hypothesize that texts of low/high coherence tend to adhere to different discourse structures. thus, we pose that using even silver-standard rst features should help in separating coherent texts from incoherent ones. this stems from the definition of the coherence itselfas the writer of a document needs to follow specific rules for building a clear narrative or argument structure in which the role of each constituent of the document should be appropriate with respect to its local and global context, and even existing discourse parsers should be able to predict a plausible structure that is consistent across all coherent documents. however, if a parser has difficulty interpreting a given document, it will be more likely to produce unrealistic trees with improbable patterns of discourse relations between constituents. this idea was first explored by @xcite , who followed an approach similar to @xcite by estimating entity transition likelihoods, but instead using discourse relations (predicted by a state of the art discourse parser (feng and hirst, 2014)) that entities participate in as opposed to their grammatical roles. their method achieved significant improvements in performance even when using silver-standard discourse trees, showing potential in the use of parsed rst features for classifying textual coherence. our work, however, is the first to develop and test a neural approach to leveraging rst discourse representations in coherence evaluation. furthermore, sentence permutation task, which involves ranking a sentence-permuted text against the original. as noted by @xcite , this is not an accurate proxy for realistic coherence evaluation. we evaluate our method on their more realistic grammarly corpus of discourse coherence (gcdc), where the model needs to classify a naturally produced text into one of three levels of coherence. our contributions involve: (1) rst-recursive, an rst-based neural tree-recursive method for coherence evaluation that achieves 2% below the state of the art performance on the gcdc while having 62% fewer parameters. (2) when ensembled with the current state of the art, namely parseq @xcite , we achieve a notable improvement over the plain parseq model. (3) we demonstrate the usefulness of silver-standard rst features in coherence classification, and establish our results as a lower-bound for performance improvements to be gained using rst features. ## conclusions and future work in this paper, we explore the usefulness of silverstandard parsed rst features in neural coherence classification. we propose two new methods, rst-recursive and ensemble. the former achieves reasonably good performance, only 2% short of state of the art, while more robust with 62% fewer parameters. the latter demonstrates the added advantage of rst features in improving classification accuracy of the existing state of the art methods by setting new state of the art performance with a modest but promising margin. this signifies that the document's rhetorical structure is an important aspect of its perceived clarity. naturally, this improvement in performance is bounded by the quality of parsed rst features and could increase as better discourse parsers are developed. in the future, exploring other rst-based architectures for coherence classification, as well as better rst ensemble schemes and improving rst parsing can be avenues of potentially fruitful research. additional research on multipronged approaches that draw from centering theory, rst and deep learning all together can also be of value.