Can we obtain significant success in RST discourse parsing by using Large Language Models? (2024)

Abstract: Recently, decoder-only pre-trained large language models (LLMs), with several tens of billion parameters, have significantly impacted a wide range of natural language processing (NLP) tasks. While encoder-only or encoder-decoder pre-trained language models have already proved to be effective in discourse parsing, the extent to which LLMs can perform this task remains an open research question. Therefore, this paper explores how beneficial such LLMs are for Rhetorical Structure Theory (RST) discourse parsing. Here, the parsing process for both fundamental top-down and bottom-up strategies is converted into prompts, which LLMs can work with. We employ Llama 2 and fine-tune it with QLoRA, which has fewer parameters that can be tuned. Experimental results on three benchmark datasets, RST-DT, Instr-DT, and the GUM corpus, demonstrate that Llama 2 with 70 billion parameters in the bottom-up strategy obtained state-of-the-art (SOTA) results with significant differences. Furthermore, our parsers demonstrated generalizability when evaluated on RST-DT, showing that, in spite of being trained with the GUM corpus, it obtained similar performances to those of existing parsers trained with RST-DT.

## introduction rhetorical structure theory (rst) @xcite is one of the influential discourse theories used to explain the coherence of texts. it plays an important role in various natural language processing (nlp) tasks at the document level, including sentiment analysis @xcite , automatic summarization @xcite @xcite , question answering @xcite , machine translation @xcite , and mt evaluation @xcite . according to rst, a text is represented as a binarized constituent tree (rst-tree), whose terminal nodes correspond to elementary discourse units (edus), clause-like units, and non-terminal nodes indicate the nuclearity status, i.e., either nucleus or satellite, of text spans consisting of single or contiguous edus. the edges represent the rhetorical relation between two adjacent text spans dominated by non-terminal nodes. figure 1 shows an example of the rst tree obtained from rst discourse treebank (rst-dt) @xcite . in the figure, the nucleurity status of the text span consisting of e 1 and e 2 is the nucleus, and it is modified by the satellite, e 3 . a mono-nuclear relation, attribution, is given to the two spans. since the late 2010s, neural rst discourse parsing methods that use encoder-only pre-trained language models @xcite to encode text spans into vectors have been proposed due to advances in neural models. while earlier models, e.g., @xcite ; @xcite ; @xcite , obtained vector representations for text spans from a static plm, such as glove @xcite , recent models, e.g., @xcite ; @xcite ; @xcite ; @xcite , obtained them from a transformer-based plm, such as xlnet @xcite . to form rst trees, they obtained vectors via plms and exploited them to determine the parsing actions in a top-down or bottom-up strategy. more recently, there is also a parser @xcite that utilizes an encoder-decoder plm to transform input text into a linearized rst tree. there was a shift in focus from encoder-only to massive-scale decoder-only plms. some large language models (llms), such as gpt-3 @xcite and llama 2 @xcite , have several tens of billions of parameters and are pre-trained with only a decoder. these have significantly impacted nlp, similar to encoder-only and encoder-decoder plms. llms have demonstrated remarkable success in various nlp tasks due to their large numbers of parameters and ease of availability. their impact extends beyond generation tasks and includes classification tasks @xcite @xcite . therefore, they could also be advantageous in rst discourse parsing. furthermore, we are strongly motivated to adopt llms because previous discourse parsing methods have been greatly improved by using encoder-only or encoder-decoder pre-trained language models. in this paper, we explore the potential of using llms for rst discourse parsing. as a first step in exploiting llms, our approach is to translate the parsing steps of both fundamental top-down and bottom-up strategies into prompts. then, we finetune llama 2 using qlora @xcite , an extension of lora @xcite , which is an adapter that injects trainable low-rank matrices into each layer of the transformer, while it freezes the weights of the pre-trained model for efficient computing. the experimental results from rst-dt, instructional discourse treebank (instr-dt) @xcite , and the gum corpus @xcite demonstrate that our parser with the bottom-up parsing strategy surpassed the current state-of-the-art (sota) results. it outperformed the current sota models by around 2-3 points on rst-dt, by 0.4-3.7 points on instr-dt, and by 1.5-6 points on the gum corpus. furthermore, out-of-domain evaluations using rst-dt and the gum corpus demonstrate the potential generalizability of our parsers. our parsers, trained with the gum corpus, achieved smaller degradation when evaluated on rst-dt. the performances are close to those of the existing parsers trained with rst-dt itself. these findings provide valuable insights into the future direction of rst discourse parsing. we will release our code at https://github.com/ nttcslab-nlp/rstparser_eacl24 . ## proposed approach three approaches can be possible when using decoder-only llms for rst discourse parsing. the first is to create linearized s-expressions for rst trees using llms, which is similar to how encoder-decoder plms have been used. the second is to replace encoder-only plms with an encoder of llms in conventional top-down or bottom-up parsing. the third method involves using llms to imitate the parsing process, which u s1 u q u s0 stack[1] encoder-only plm [t b(i-2) t e(i-1) ] [t b(i) t e(i) ] w b(i-2) w e(i-1) w b(i) w e(i) [t b(i+1) t e(i+1) ] w b(i+1) w e(i+1) tokens: ) ave. ave. ave. e i-2 e i-1 e i e i+1 stack[0] queue[0] involves combining feature extraction and classification layers, similar to how encoder-only plms have been used. the first approach can be challenging, especially when dealing with lengthy documents. this is because the number of output tokens increases disproportionately with the number of input tokens. furthermore, additional techniques, such as constrained beam search, are required to obtain valid linearization forming a tree. the second approach is not promising because it does not perform as well as encoder-only plms @xcite . thus, we adopt the third approach in this work. before describing our own approach, we first illustrate fundamental top-down and bottom-up parsing methodologies using encoder-only plms. ## results and discussion overall performance: when focusing on the number of the parameters in llama 2, the largest number naturally yields the best results. in particular, 70b parameters obtained these results indicate the effectiveness of llms for rst discourse parsing. the findings are interesting in that simple pre-trained language models consisting of only a transformer decoder can be easily tailored for determining parsing actions by fine-tuning with prompts. our parsers perform well on all datasets, which are from different domains. the results demonstrate the advantage of llmbased rst discourse parsing in domain portability, particularly in achieving the best scores on instr-dt with less training data. cross-corpus generalization: to examine the generalizability of our parsers in detail, we evaluated them with out-of-domain evaluations using rst-dt and the gum corpus. tables 2 and 3 show the results of training parsers on one dataset and evaluating them on the other. comparing the results with the successful improvements were achieved by using a large number of llm parameters and training them with a massive amount of text, compared to the encoder-only models. although we could potentially improve the performance of the encoderonly plms by increasing their parameters to the level of the current llms, this task poses a significant challenge. furthermore, considering the focus of the research has shifted from encoder-only plms to llms, our findings are highly valuable for future research in rst discourse parsing. ## conclusion this paper explored the potential of using llama 2, the largest publicly available decoder-only language model, pre-trained with a trillion tokens of text data, for rst discourse parsing. to exploit llama 2, we translated fundamental bottom-up and top-down parsing processes into prompts. then, we fine-tuned llama 2 with them using qlora for efficient computing. the experimental results obtained from three datasets, rst-dt, instr-dt, and the gum corpus, which come from different domains, demonstrated the effectiveness of our parsers with llama 2, including their domain portability. specifically, our approach with llama 2 (70b) obtained better results than the current sota parser on all datasets. furthermore, findings from the experimental results for cross-corpus generalization showed the significant promise of our approach, that is, that our parsers, in spite of being trained with the gum corpus, obtained comparable performance to parsers trained with rst-dt itself in span f 1 scores, keeping small degradation in nuc., rel., and full f 1 scores. since this work is just a first step in exploiting llms for rst discourse parsing, we can work in this direction with various topics to further improve the llm-based model; e.g., incorporating richer information to improve the current top-down parsing model.