Using and comparing R hetorical S tructure T heory parsers with rst-workbench (2021)

Abstract: I present rst-workbench, a software package that simplifies the installation and usage of numerous end-to-end Rhetorical Structure Theory (RST) parsers. The tool offers a web-based interface that allows users to enter text and let multiple RST parsers generate analyses concurrently. The resulting RST trees can be compared visually, manually post-edited (in the browser) and stored for later usage.

## introduction rhetorical structure theory (rst) provides a formalism for hierarchical text organization that can be applied to a wide range of natural language processing tasks, ranging from text generation @xcite to the assessment of conversational patterns of alzheimer's patients @xcite . most research on rst parsing is focused on parser engineering, i.e. the evaluation of parsers against a "gold standard" hand-annotated dataset. although rst corpora exist for a variety of other languages (e.g. german, dutch and spanish), endto-end discourse parsers are usually only trained and evaluated on english data, with the notable exception of @xcite . there are a number of rst parsers that were developed for other languages, but either are they not publicly available (e.g. @xcite for german and english as well as @xcite for portuguese) or they do not produce complete rst analyses, e.g. @xcite for japanese (no intra-sentence relations) and da @xcite for spanish (no inter-sentence relations). compared to other nlp tasks like syntax parsing, the amount of available training data is limited, with corpora being in the range from dozens to a few hundred hand-annotated texts. there is also little work evaluating rst parsers beyond the parseval-based procedure proposed by @xcite . 2 for example, machine learning models are usually not compared with respect to their ability to detect rare rhetorical relations. @xcite found that rhetorical relations in rst-dt at different levels-i.e. between clauses within sentences, between sentences within paragraphs and between paragraphs-all follow the same zipf's law-related distribution. individual relations show different patterns, e.g. attribution is more common in intrasentential relations than on higher levels. in addition, no systematic review exists of the impact of the preprocessing steps-sentence splitting, syntax parsing and segmentation into elementary discourse units (edus)-on the quality of the resulting rst parses. there is some work in this direction, though. for example, @xcite implemented two rst parsers that only differ in the syntax parser used-the constituent-based rst parser produced slightly better results, but the dependency-based equivalent was 2.5 times faster. @xcite evaluated the influence of syntactic information (using either constituency parses, dependency parses or only pos tags) on discourse segmentation. @xcite reviewed the impact of different neural network architectures on implicit discourse relation detection. while @xcite showed that rst parser performance can be improved by train-ing them on large rst treebanks automatically generated using distant supervision, i hypothesize that rst parsers can profit even more from larger human-annotated training corpora. in turn, the annotation of rst corpora can likely be sped up by leveraging rst parsers. in the same vein that translators can produce high-quality translations by post-editing machine-translated texts more quickly than by manual translation alone @xcite , i assume that linguists can produce rst analyses faster with machine support (i.e. by selecting the best automatic analysis from a number of rst parsers and then post-editing it) than by relying on handannotation alone. if the goal is to make annotators use rst parsers productively, the parsers need to be adapted to meet their needs. while the primary focus of rst parser development is improving upon state-of-theart benchmark results, this work focuses on usability and compatibility, i.e. the parsers need to be easy to install and run while supporting the same format(s) that common rst annotation and visualization tools use. to achieve this, i implemented rst-workbench, which: ## related work to the best of my knowledge, rst-workbench is the first tool that offers a graphical user interface for and integrates several rst parsers. besides rst-web @xcite , which is integrated in this software, there are two other annotation tools specifically made for rhetorical structures: rsttool (o'donnell, 2000) and treeannotator @xcite . for visualizing rst trees and querying rst corpora, there is annis3 @xcite . while i implemented very minimal rest apis around the individual rst parsers in python, clam (van gompel and reynaert, 2014) could be used to create rest api wrappers around command-line nlp tools by writing a configuration file. for simple cases, it is slightly more complicated to setup than my approach (cf. section 3.2), but it comes with many additional features (e.g. user authentication, batch processing and a generic web interface for each api) and can be extended with additional format converters and visualization components. ## software architecture and usage the rst-workbench provides a simple way to install multiple rst parsers on a computer, run them as well as visually compare and edit their analyses in a web browser. its architecture and usage is summarized in figure 1 . screenshots of the workflow are provided in figures 2 and 3 . at the core, the rst-workbench consists of six existing open-source rst parsers-hilda @xcite , feng and hirst (2014), dplp @xcite , @xcite , codra @xcite and stagedp @xcite -packaged as docker containers to make them easily installable without any user intervention (cf. section 3.1). users do not have to learn the different command-line interfaces of the parsers, but can simply interact with them via a web browser. to make this possible, i added a rest api to each of the parsers, which the browser can talk to (cf. section 3.2). in the browser, annotators can enter text or upload a plain text document. after clicking the "run parsers" button, all rst parsers are started concurrently to analyze the given text. the results appear asynchronously in the browser, i.e. the user sees the result of the fastest parser immediately when it is available and does not have to wait for the remaining parsers to finish processing. users can now select the parse tree that most resembles their linguistic intuition, and click "edit in rstweb" to load the analysis into the rstweb annotation tool. here, all aspects of the rst tree can be modified, e.g. rhetorical relations between edus and/or larger subtrees can be replaced (figure 3 ). afterwards, the result can be saved locally for further inspection or corpus creation. the technical setup needed to integrate all these stand-alone tools into one software package with a unified interface is described in the following subsections. ## conclusions in this paper, i presented a software package that simplifies the installation, usage and visual comparison of rst parsers. i showed how it can help linguists to produce manual rst analyses with less effort. i plan to integrate the rst-workbench directly into rstweb to facilitate corpus annotation projects. in rstweb, an "administrator" can create an annotation project, upload documents to be annotated and assign them to annotators. with an integrated rstworkbench, the administrator could precompute and store the automatic rst analyses once for all annotators, which would reduce wait time for the annotators and allow them to work without switching browser tabs and tools. in the current setup, analyzing a text with all parsers may take up to two minutes. most of this time is spent on loading the models of the underlying syntax parsers. this can be drastically reduced by "outsourcing" the syntax parsers into their own web services, as i have already done for dplp. 7