Leveraging Plug-and-Play Models for Rhetorical Structure Control in Text Generation (2024)

Abstract: We propose a method that extends a BART-based language generator using a plug-and-play model to control the rhetorical structure of generated text. Our approach considers rhetorical relations between clauses and generates sentences that reflect this structure using plug-and-play language models. We evaluated our method using the Newsela corpus, which consists of texts at various levels of English proficiency. Our experiments demonstrated that our method outperforms the vanilla BART in terms of the correctness of output discourse and rhetorical structures. In existing methods, the rhetorical structure tends to deteriorate when compared to the baseline, the vanilla BART, as measured by n-gram overlap metrics such as BLEU. However, our proposed method does not exhibit this significant deterioration, demonstrating its advantage.

## introduction language generation technology has been significantly improved due to the advance of pre-trained language models. however, although we would often like to have a text with a certain discourse or logical structure, the current technology has difficulty in following such global constraints. in this paper, we address the task of controlling natural language generation in terms of the discourse structure of the generated text. as a discourse structure, we employ a tree structure based on rhetorical structure theory (rst) @xcite ). an existing work rstgen @xcite incorporates rhetorical structures into text generation by transforming trees into embeddings prior to generation. in contrast to rstgen, our method dynamically controls the rhetorical structure during text generation using the plug-and-play language model (pplm) @xcite , as shown in figure 1 . one significant advantage of our method language model classifier pplm @xcite proposed method figure 1: formulating a task for rst-based text generation. our model is based on the plug-and-play language model (pplm) that controls language models to generate texts with a specific attribute a. we consider the relation label in rst @xcite as the desired attribute. we evaluate our method on the newsela corpus @xcite , a dataset, which consists of texts at various levels of english proficiency. our experiments demonstrated that our method outperforms the vanilla bart baseline in terms of the correctness of output rhetorical structures. in existing methods, the rhetorical structure tends to deteriorate when compared to the baseline, as measured by n-gram overlap metrics such as bleu @xcite . however, our proposed method does not exhibit this significant deterioration, demonstrating an advantage. ## related work rhetorical structure theory (rst) @xcite represents the semantic relationships within a text as a constituency binary tree, while a dependency tree-based framework (prasad et al., 2008) also exists. a recent study proposed to incorporate discourse structures into a language model using variational auto encoder @xcite . we use rst by following recent works in generation @xcite . early approaches treated the incorporation of rst into text generation as a planning problem @xcite . integrating tree structure into neural network-based language generators has been actively studied. @xcite incorporated rst trees into an autoregressive language model by converting them into embeddings. chernyavskiy (2022) created the entire text plan as an rst tree, followed by autoregressive generation of the text span using a language model. in contrast to the aforementioned works, our method dynamically controls rhetorical structure during text generation using pplm. ## rst-based controllable generation for our experiments, we utilize a binary form of rst tree following rstgen @xcite . to construct a binary form of the rst tree from a text, the text is divided into smaller units, called discourse units (dus). we assign an index to each node in the tree, starting from zero. when the index of a parent node is i, the left child node is indexed as 2i + 1, and the right child node as 2i + 2. the text at a node with no children represents elementary discourse unit (edu), and the text at a parent node corresponds to a pair of dus. a parent node has a relationship label and a nuclearity label indicating the semantic relationship of sibling dus. ## experimental setup the newsela corpus @xcite consists of news articles for readers with various english proficiency levels. paragraphs extracted from these articles are utilized as the dataset in this paper. we employ a trained rst parser @xcite to parse each of the dataset. we extract keyphrases using the trained topicrank keyphrase we used pytorch library @xcite ) for the implementation. the baseline model was trained by fine-tuning bart @xcite . adamw @xcite was used as the optimization method, and the parameters are included in the appendix. we introduced early stopping when the validation loss did not decrease for three epochs. we use bart, trained to generate text conditioned on the information of keyphrases and their positions, as the baseline model. we compare our model with two models; 1) discodvt @xcite is a discourse structure-based text generation model. discodvt uses a discrete variational auto encoder, reflecting discourse structures into bart @xcite . 2) rstgen (adewoyin et al., 2022) introduces additional embedding layers for representing rst trees. embeddings of an rst tree are added to token embeddings, which serve as inputs to language models. we use the rst embeddings from rstgen as prefix embeddings for the baseline model. to assess whether the generated texts have specified rhetorical structures, we use the standard parse-eval @xcite metric. this metric measures how well a labeled tree matches the reference tree in terms of span units. first, we parsed the generated texts using the same rst parser used for annotating the dataset to obtain rst trees. next, we converted the rst trees into a rightheavy binary structure following @xcite . span, nuclearity, relation, and full refer to evaluations of unlabeled, nuclearity-labeled, relation-labeled, and fully labeled tree structures, respectively. we also use bleu @xcite , rouge @xcite , and me-teor (mtr) @xcite as evaluation metrics. these metrics evaluate the quality of the generated texts by comparing n-gram overlaps with reference texts. bleu measures precision of n-gram, whereas rouge measures recall. meteor considers both precision and recall. we report bleu-4 (b-4), which evaluates the overlap of 4-grams, and rouge-l (r-l), which measures the longest common subsequence between the generated texts and reference texts. bertscore (b-s) @xcite is used for evaluating semantic similarities. fluency is evaluated through perplexity (ppl) computed using the medium model of gpt-2 @xcite . coherence of generated texts is evaluated using two sets of metrics. firstly, we measure the recall of discourse markers (dm). discourse markers are words which semantically connect sentences. the recall represents the percentage of correctly generated markers present in the references. additionally, gruen @xcite is used. this metric assesses generated texts from following for perspectives: grammaticality, non-redundancy, focus, and coherence. ## results ## limitations in our experiments, we used rst trees with depths of two or less. thus, our method primarily considers shallow relationships. in contrast, rstgen imposes a limit of twelve or less levels of tree depth, allowing our proposed method to handle a smaller range of depths. we aim to explore the application of our method to deeper trees. ## conclusion we proposed a method for controllable text generation by language models based on rhetorical structures, inspired by pplm. while our model did not improve accuracy compared to the baseline, it showed improvement over prior models based on discourse and rhetorical structures. additionally, we evaluated text coherence in terms of discourse markers and generally observed improved accuracy. however, the depth of the rst tree considered in this paper is limited. thus, we will extend the proposed model to deeper trees.