RSTG en: Imbuing Fine-Grained Interpretable Control into Long- F orm T ext Generators (2022)

Abstract: In this paper, we study the task of improving the cohesion and coherence of long-form text generated by language models. To this end, we propose RSTGen, a framework that utilises Rhetorical Structure Theory (RST), a classical language theory, to control the discourse structure, semantics and topics of generated text. Firstly, we demonstrate our model’s ability to control structural discourse and semantic features of generated text in open generation evaluation. Then we experiment on the two challenging long-form text tasks of argument generation and story generation. Evaluation using automated metrics and a metric with high correlation to human evaluation, shows that our model performs competitively against existing models, while offering significantly more controls over generated text than alternative methods.

## introduction controllable text generation has attracted much attention in recent years. generating coherent and cohesive long-form text with controllable attributes is particularly challenging due to the relatively complex syntactic and semantic structures involved compared to short-text generation. long-form text generation can find applications in automatic argumentation, motivational speech, and opinionated writing, to name a few. a popular approach to controllable text generation is to design prompts, which control high-level linguistic features (i.e., text style and sentiment) by prefixing simple context to the input of a language model. these non-granular methods are often too coarse to allow different parts of the text to have diverse or contrasting features. furthermore, they tend to focus on a single linguistic feature of text, meaning that extra frameworks such as plugand-play @xcite are required to control multiple linguistic features. to achieve improved content control, researchers have coupled a content planning module with a text generation module @xcite , in which the content planning module firstly generates a set of keyphrases and their corresponding positions in sentences and the generation module fills in the gaps. to control syntactic structures of text, recent works have proposed a neuro-symbolic approach leveraging dependency structure theory (dst) @xcite . enforcing specific structures over the generated text has been proven to be useful in increasing coherence and cohesion of short-form text. however, a relatively large amount of dst information is required to encode comparatively short texts. thus, they are unsuitable for use in long-form text generation tasks. similar to dst, rhetorical structure theory (rst) @xcite is a classical tree-based interpretation of natural language. whereas dst is most useful for intra-sentence interpretation, rst is most useful for inter-sentence interpretation of language. we propose a neurosymbolic framework that can imbue an rst understanding of text to existing language models. more specifically, our framework (1) allows more finegrained control of syntax, semantics and text structure; (2) utilises a well-defined rhetorical structure of language, thus offering better interpretability; (3) can be directly integrated into existing pre-trained language models such as gpt-2 @xcite and bart @xcite . we evaluate our proposed framework on two tasks: argument generation and story generation. we show that our proposed framework improves upon existing content control approaches on automatic evaluation metrics including bleu @xcite , meteor @xcite and generates better text in terms of grammar and coherence measure. we further demonstrate our model's ability to generate text with control over syntactic, semantic and discourse features. our main contributions can be summarised below: we discuss several lines of research in controllable text generation. prompts for prompt-based control, a context is prepended to a language model input. the prepended item is related to the type of desired output. this method has been used to manipulate the syntax @xcite and semantics @xcite of the output. alternatively, the prepended item can provide semantic control in order to cover a given topic @xcite , mention specified entities @xcite , display a certain attribute @xcite , or even exhibit a style of text @xcite . these methods suffer from the inability to exert fine-control, that is, a change in any one of the input prompts will change the whole generated text. furthermore, all these works utilise non-expansive features for their prompts, which prevents them from making iterative improvements to existing generated text. content planning despite the impressive progress made in many generation tasks, earlier neural models are known to produce low-quality content @xcite , often with low relevance, and poor discourse structure @xcite . content-based planning approaches were added into neural systems to enhance content relevance @xcite @xcite . @xcite extended previous content planning approaches by dividing the content plan into different types of information; 'entities', 'claim' and 'concepts'. an alternative planning approach focuses on where the key phrases should be placed in the generated text in order to improve coherence @xcite . the text generator is then conditioned on the provided key phrases and their respective positions in text. similar to our approach, these methods allow users varying levels of custom control by manual augmentation of the planning module output. ## rhetorical structure theory rhetorical structure theory (rst) provides a formal structure for interpreting language based on relations between parts of text. each structure is defined by three sets of features: the binary tree structuring of discourse units (edu)s, the nuclearity between sibling dus, and the relationship between sibling dus. an example rst tree using the schema provided by @xcite as shown in figure 1 . ## rstgen: rst-dependent long-form text generator we propose a long-form text generation approach, call rstgen, which provides control over semantics and structure of text, with the aim of improving coherence and cohesion of the generated text. coherence is concerned with how well constituent ## open generation evaluation we first train a model using our rst annotated dataset described below. then we analyse the model's ability to control semantic, syntactic and text structure. ## evaluation on argument generation and story generation we evaluate rstgen for the tasks of argument generation and story generation. these tasks require our model to output coherent and cohesive text. ## conclusion we present a novel controlled text generation framework, rstgen, which uses fine-control over a rhetorical structure theory based context as a means to improve the coherence and cohesion of generated text. we also leverage the structural information presented by rst to propose an rst aware attention scheme, ensuring that the model attends to the correct information during long form generation. through investigation of rstgen's open generation text, we showed that our approach can exhibit a high level of intepretable fine-control over syntactic, semantic and structural features of text.