Toward Automatic Discourse Parsing of Student Writing Motivated by Neural Interpretation (2022)

Abstract: Providing effective automatic essay feedback is necessary for offering writing instruction at a massive scale. In particular, feedback for promoting coherent flow of ideas in essays is critical. In this paper we propose a state-of-the-art method for automated analysis of structure and flow of writing, referred to as Rhetorical Structure Theory (RST) parsing. In so doing, we lay a foundation for a generalizable approach to automated writing feedback related to structure and flow. We address challenges in automated rhetorical analysis when applied to student writing and evaluate our novel RST parser model on both a recent student writing dataset and a standard benchmark RST parsing dataset.

## introduction automatic writing feedback technologies (e.g., mi write @xcite , criterion @xcite , @xcite , writing pal @xcite , and revision assistant @xcite ) show promises in helping students to develop writing skills at scale. one challenging area where these technologies meet is in providing feedback for improving coherence of student essays @xcite . efforts have been made to address the challenge of providing structural level feedback via automatically extracting discourse structure from essays @xcite . extracting hierarchical discourse structure and organization from documents has been shown to be valuable for numerous applications including text categorization, authorship attribution, and automatic essay feedback @xcite . a popular approach to analysis of the structure of writing that leverages principles of the dependency-based hierarchical nature of text and is common across genres is the discourse analytic framework known as rhetorical structure theory (rst, described in section 3.1) @xcite . rst holds the promise of providing specific structural writing feedback for free-form essays @xcite . however, rst parsing has remained a challenging task due to the dearth of annotated data and the challenges of decision making for discourse relations based on local context @xcite . this paper builds on the same theoretical foundation using a neural network based rst parser as a means for automation. specifically, we propose a novel neural approach to automated rst analysis that improves over the best previously published approach from the field of language technologies. in particular, of existing neural architectures for rst parsing, neural transition based parsers have been making headway @xcite , however, at their core, transition parsers make parsing decisions locally. while they use recurrent models to construct their stacks and buffers, in practice, recurrent models have been shown to primarily to use very near context @xcite . this is a limitation for discourse parsing where knowledge about the document as a whole may provide essential context for judging relations. we therefore propose and evaluate two improvements to the neural transition parser paradigm that provide better performance, both on standard rst parsing and on student writing by utilizing the limited data more efficiently: the first improvement builds on the general concept of multitask learning in nlp @xcite and the intuition that a topic-like sentence, as a common key component in many writing assessments and rubrics @xcite , may provide important contextual information to aid local parser decision-making. the second improvement suggests a potential for a reflective form of neural network learning related neural component reuse that grows out of state-of-the-art work in neural network interpretation. in the following sections, we evaluate our parsing model on both the standard english rst discourse treebank (rst-dt) @xcite and a more recent rst dataset of student writing @xcite . ## rhetorical structure theory rhetorical structure theory decomposes a document into basic units of analysis called elementary discourse units (edu) that can be combined through rhetorical relations between units into larger composite units @xcite . thus, the rhetorical relations combine to build a hierarchical tree structure that represents the overall structure of the document (figure 1a ). each relation has one (mononuclear) or more (multinu-clear) nuclei where a nucleus is an essential span which, if deleted, would leave the remaining text incoherent. mononuclear relations have satellites that are related to the nucleus by means of a rhetorical relation. they play a supporting role, and are therefore not necessary for coherence of the document. each node of the tree represents a relation tuple ⟨s, n, r⟩ where s is the span, n is the direction of nuclearity, and r is the relation label. this is more readily seen in figure 1b which depicts an alternate representation of the rst tree structure. rst has a long history @xcite , and its original formulation continues to be treated as authoritative. however, for some types of writing, especially student writing, additional and combined relations have been proposed in order to bring the set of used relations in line with the writing practices that are applicable to the corpus @xcite . ## related work this paper makes its fundamental contribution to work on automated feedback for student writing by expanding analysis capabilities that lay a foundation for a new form of support. our technical contribution is grounded within the field of neural network modeling, contributing to work on neural approaches to rhetorical structure analysis and leveraging approaches originating in the area of neural model interpretation. an effective method for performing discourse parsing has been to utilize techniques from syntactic parsing and applying them at the document level. while rst parsing research has more frequently seen parsers influenced by another approach referred to as constituency parsing, it was shown that using techniques pioneered for dependency parsing could be as or more effective @xcite . as methods for rst parsing moved from those that rely on discourse markers and hand-coded rules @xcite to those that rely on deep learning @xcite @xcite , many of the improvements have been through techniques from syntactic parsing @xcite . in a similar way, our work builds on past rst parsers using neural transition parsing @xcite . we extend this work by leveraging another area of neural network research, namely neural network interpretation, in order to yield a reflective form of learning that improves performance by leveraging lessons learned in an earlier stage of the training, as in a stage-based regression. neural pathways @xcite refer to a method for pinpointing sets of a model's neurons that function together in groups. these groups of neurons are referred to as pathways because they cut across architectural layers and allow representation of the flow of activation through a network, potentially from input all the way to output. for our application, we follow the original authors and use pca @xcite for this step as the resulting factor loadings (decoster, 1998) can then be used to determine which neurons belong to each pathway, and that forms the basis for our pruning approach. the remaining stages of this approach are not used in the work reported here but offer opportunities for promising follow up work. in offering an abstraction over the details of a neural model, this approach offers the possibility of identifying portions of learned networks that can be dissected from the network as a whole and then reused as pre-packaged basic functionality within a more complex model learned at a later stage. thus, we seek to harvest components pretrained on a simpler dataset to aid in learning a more robust model later on a more challenging dataset. while a deep dive into the differences between the learned functions of an rst parser trained on a relatively clean standard dataset and one trained solely on a noisier student writing dataset is beyond the scope of this paper, we will demonstrate that this work provides inspiration for development of what we will refer to as a neural pruning method that protects important simple generalizations while enabling accounting for complex special cases as well, and to represent an awareness of the difference between these in the final decision making. ## a corpus of student writing in this section, we first offer more understanding about rst and then describe a corpus of student writing that has been annotated with rst. ## improving and validating rst parsing for student writing in this section, we begin with and then improve on the best previously published approach in automating rst analysis for writing. transition parsers are common among state-of-the-art models for discourse parsing with rst in the past several years. their power lies in their ability to make strong local decisions about the next action the parser must take given an embedding that, because of recurrent neural models, has the capacity to contain features from the whole document. however, recurrent neural networks often do not in practice retain sufficient context for long range dependencies @xcite . we address this by providing an additional embedding for the predicted most nuclear sentence of the document to provide a reference point for the parsing decisions. furthermore, inspired by neural interpretation techniques, we further augment the model with a two stage parsing approach that allows the second stage of the model to learn from mistakes made by the first. ## experiments we provide three quantitative evaluations of our method: first, in order to compare our parser to previous rst parsers, we train and evaluate our parser on the english rst-dt corpus. second, we provide an ablation study of the added components of our model along with the model we used as a base. the ablation study uses the same test set as the first experiment, so results are directly comparable. lastly, we train another version of our model on the turnitin dataset, which has a very different set of properties when compared to the rst-dt corpus. this last set of experiments is designed to test the ability of the model to handle unpolished, less structured text. the model is compared to the strongest baseline from the rst-dt corpus retrained on the turnitin dataset. ## conclusion we presented two principal augmentations to neural transition parsers for rst that resulted in a 7.9% increase in span prediction and a 4.5% increase in nuclearity prediction. these improvements were made while remaining competitive on relation prediction, though no improvement was observed for that metric. furthermore, we evaluated our model on an alternate, noisier dataset. we found that on this dataset our model had more accurate relation predictions than past approaches from the inclusion of a neuron selection step between the training of parent and child models in a boosting-like neural ensemble enhancement. for future work, we want to empirically verify that the prediction of structural breaks (joint relations) in student writing align with teacheridentified organization feedback. this can enable automated essay feedback on the absence of structure, providing support where it's needed most. furthermore, conveying the necessary information contained within rst trees to students and teachers provides an additional rich area of inquiry. it is worthwhile to further explore how prospective users respond to the technological instruction support to facilitate students' ability to locate places for revision and teachers' ability to integrate the automated feedback into their instruction.