RST - L o RA : A Discourse-Aware Low-Rank Adaptation for Long Document Abstractive Summarization (2024)

Abstract: For long document summarization, discourse structure is important to discern the key content of the text and the differences in importance level between sentences. Unfortunately, the integration of rhetorical structure theory (RST) into parameter-efficient fine-tuning strategies for long document summarization remains unexplored. Therefore, this paper introduces RST-LoRA and proposes four RST-aware variants to explicitly incorporate RST into the LoRA model. Our empirical evaluation demonstrates that incorporating the type and uncertainty of rhetorical relations can complementarily enhance the performance of LoRA in summarization tasks. Furthermore, the best-performing variant we introduced outperforms the vanilla LoRA and full-parameter fine-tuning models, as confirmed by multiple automatic and human evaluations, and even surpasses previous state-of-the-art methods.

## introduction the advent of pre-trained large language models (llms), such as llama-2 @xcite , vicuna @xcite , and gpt-related models from openai @xcite , has greatly accelerated research progress of natural language processing (nlp). with the continual growth in the scale of llms, the requirements for both software and hardware in order to fully fine-tune llms to adapt to downstream tasks, especially in processing long sequence data, will become increasingly demanding @xcite . parameter-efficient fine-tuning (peft) strategies are noteworthy in mitigating the aforementioned problem by reducing the number of parameters that need to be adjusted @xcite @xcite @xcite @xcite lighted that by updating only 0.01-1% of the (additional) parameters and freezing all other parameters of llms, peft methods can match or even exceed the performance of vanilla full-parameter finetuning @xcite @xcite @xcite @xcite @xcite @xcite . among these methods, lora algorithm @xcite has achieved state-of-the-art (sota) performance due to its ability to circumvent the latency associated with adapter tuning @xcite as well as the input length constraints of prefix/prompt tuning @xcite during model training and inference @xcite @xcite @xcite . recent investigations @xcite @xcite @xcite @xcite have revealed that peft strategies face challenges in distinguishing latent text relations and determining the importance level of different sentences during fine-tuning. this issue arises because such distinctions are not a primary focus in peft's learning process and are not explicitly represented in the input data. however, this is essential for the task of long document summarization since generating a good summary often requires natural language generation (nlg) models to have the ability to discern salient information within the text and comprehend the intricate interrelations among different text components. our approach proposed here takes inspiration from @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite , who have advised that explicitly integrating document structure and/or discourse knowledge can enhance the performance of neural summarization models when fully finetuning the nlg models. this motivates us to investigate the following research questions: can the rhetorical structure theory @xcite improve the performance of lora strategy in summarizing lengthy documents? specifically, we want to explore and verify whether infusing rst knowledge into lora can improve the performance of long document summarization. to answer this question, this paper will propose, introduce, and integrate four rst structure variants to guide the training of lora. these variants include (i) binary (ii) probability rst distribution, both with (w) and without(w/o) relation labels. in summary, our contributions are as follows: our approach is compatible with both seq2seq and gpt transformer-based architectures, allowing for its easy adoption across different llms. ## rst prerequisite knowledge rhetorical structure theory (rst) is a discourse framework that is helpful for determining which sentences in a document should or should not be included in a summary @xcite @xcite @xcite . to be specific, rst delineates a set of coherence relations between text segments, known as elementary discourse units (edus), at the document level (e.g., one segment might provide clarification for another, or conversely, two segments could present contrasting viewpoints). moreover, rst categorizes edus based on their discourse importance, labeling central edus as 'nuclei', and less central edus as figure 1: an example of rst tree: [utilizing discourse structure to enhance text summarization is beneficial.] edu1 [this technique can be used to identify key ideas and capture often overlooked nuances.] edu2 [accurate capture of these complex structures facilitates the generation of good summaries.] edu3 ## proposed approach a limitation observed with lora and other peft methods during the fine-tuning phase is that, while their primary function is to act as a low-rank approximation of the weight matrices in llms, they do not adequately capture textual context relations @xcite @xcite . one of the reasons is that lora is not driven or guided by discourse knowledge during the training phase, because this part of knowledge is not explicitly present in the input data @xcite . in addition, the matrices obtained by low-rank approximation strategies may have more difficulty in capturing complex textual discourse relations due to the smaller semantic space that can be expressed when compared to llms' weight matrices @xcite @xcite . hence, we propose a method that directly and explicitly incorporates discourse architecture into lora. this approach allows lora's adaptation mechanism for downstream tasks to discern intricate text relations through soft guidance, which can leverage contextual discourse connections and steer the learning trajectory toward a discourse-informed summarization process. ## conclusion we present rst-lora, a novel discourse-aware lora model tailored for long document summarization. our approach primarily incorporates rhetorical knowledge into the lora training process by transforming rst structures into rst distributions. we develop four rst-lora variants, examining the impact of uncertainty in rst relational connections and discourse labels on overall performance. empirical evidence from our studies demonstrates a consistent improvement in the performance of the standard lora model. by finetuning less than 0.5% of the llms parameters, our best rst-lora variant not only surpasses the performance of lora and fft but also exceeds previous state-of-the-art methods. furthermore, our analysis underscores the efficacy of our approach in leveraging discourse knowledge, which strengthens lora's capabilities in producing more factually consistent and better-quality summaries.