Using Structured Content Plans for Fine-grained Syntactic Control in Pretrained Language Model Generation (2022)

Abstract: Large pretrained language models offer powerful generation capabilities, but cannot be reliably controlled at a sub-sentential level. We propose to make such fine-grained control possible in pretrained LMs by generating text directly from a semantic representation, Abstract Meaning Representation (AMR), which is augmented at the node level with syntactic control tags. We experiment with English-language generation of three modes of syntax relevant to the framing of a sentence - verb voice, verb tense, and realization of human entities - and demonstrate that they can be reliably controlled, even in settings that diverge drastically from the training distribution. These syntactic aspects contribute to how information is framed in text, something that is important for applications such as summarization which aim to highlight salient information.

## introduction language models pretrained on enormous corpora have become a staple in natural language processing because of their power and adaptability @xcite @xcite . these models exhibit strong performance across a range of applications, but are not inherently controllable beyond the choice of input. to enforce specific constraints on their output, we must introduce an additional mechanism for control, whether by inserting control codes during pretraining @xcite or by adding components to the model @xcite . these prior methods allow specification of high-level attributes (e.g., topic or sentiment) but leave the specifics of sub-sentential realization to the model. in contrast, we present a method for controlling generation of specific verbs and entities within a sentence. we investigate the setting in which a pretrained language model is used not as an endto-end solution, but rather to directly generate text from a predefined content plan. such a content plan may be created at an intermediate stage in a given task (e.g., as the product of content selection over a document graph in amr-based summarization) but is not inherently task-specific. specifically, we focus on the controllability of bart when it is fine-tuned to generate text from an intermediate abstract meaning representation (amr) @xcite , a form of graphical semantic representation. we choose amr because it is relatively widely used, including as an intermediate representation in summarization @xcite and machine translation @xcite , and the problem of amr-to-text generation is well-studied @xcite @xcite . we focus on bart because of its competitive performance in summarization, a task where we feel our work is particularly applicable, as document-level content graphs have been shown to be useful intermediate representations in long-context summarization settings @xcite . our setting makes use of the powerful generation abilities of a pretrained language model while also exposing a direct graphical representation of the content, which allows us to target control tags to specific nodes in order to impose fine-grained control (i.e., constrain the text-level realization of specific verbs or entities). this is not possible in an end-to-end approach to some tasks, such as in summarization: although high-level attributes can be controlled during generation, there is no preexisting syntactic or content plan, and so verbs and entities in the summary cannot be directly controlled as they have not been determined yet. in cases where we desire such control -e.g., in queryfocused summarization, or when highlighting important entities @xcite -it is thus useful to have a representation that specifies syntactic aspects at the level of individual verbs or entities. in our experiments, we augment the amr input to our generator with tags which we automatically infer for three modes of fine-grained syntax, which we choose for their relevance to summarization: verb voice (active or passive), verb tense, and syntactic realization of human entities (i.e., names and pronouns). controlling voice and entities contributes to the model's ability to use syntax to highlight a specific topic or focus, following centering theory @xcite , which is important in cases where the focus of the summary may differ from the focus of sentences in the document. as plain amr does not contain information about tense, which is important for maintaining faithful summaries, we also consider controlling verb tense to avoid generating hallucinations about when an event took place. we find that finetuning bart to generate from amr augmented with these syntax tags makes it largely controllable across all three types of syntax. importantly, this holds true even when the finetuned models are evaluated on a test set designed to have a radically different class distribution than the training set, without any tradeoff as to the fluency of the generated output. we further find that training the same model with control on multiple syntactic modes improves performance on voice controllability, though not on tense. our experiments show that our tagged models are far better at controlling voice, tense and entity realization than a model without tags. in summary, our contributions are: our code will be made available at https://github.com/feitzin/amr-controlledgeneration. work on controllable summary generation also shares this focus on global attributes. @xcite trains a convolutional model to generate summaries controlled by attribute markers for length, entities to focus upon, domain, and subset of the text. @xcite fine-tunes a bart model to generate output summaries that are controlled using keywords or prompts, allowing the model to focus on specific entities or desired information. this approach addresses a similar problem to our work, but focuses on global rather than fine-grained control, does not necessarily frame a summary around selected relevant content, and is applicable to classic single-document summarization, whereas our approach is generalizable to multi-document and long-document settings due to its use of a content selection model pipelines in surface realization. elder et al. (2019) demonstrate that a symbolic intermediate representation (based on a dependency graph) used as input to a neural generator can yield improvements on the surface realization task. castro ferreira et al. (2019) find that a pipeline of discrete modules can yield improvements over end-to-end neural models for data-to-text generation. however, @xcite find that a pretrained language model (bart) can outperform previous state-of-the-art modular pipelines on surface realization. we aim to take the best of both by using bart as our generator, but leaving the choice of content selector free. ## data and methodology the amr bank @xcite is the largest gold standard corpus for amr, but it contains only around 60,000 annotated sentences in total, which is a relatively small amount of data for finetuning a large model such as bart. thus, for finetuning, we instead use a much larger text corpus which we parse automatically using the published code for the amr parser of @xcite . as the application we are interested in is summarization, we use the multidocument summarization corpus of gholipour ghalandari et al. (2020), which consists of 10,200 clusters containing on average 235 documents each. we refer to this as the reservoir corpus. although we do not use the amr bank for finetuning, we do use its proxy subset, which contains news documents and summaries, as a second evaluation set. due to constraints on time and processing power, we do not use the entire reservoir corpus for finetuning, but rather finetune our models primarily on the reservoir validation set, which contains 580,787 sentences in total. we split the reservoir corpus' validation set into a training set of 500,000 sentences and validation set of 80,000 sentences, which we use for finetuning. for evaluation, we sample 10,000 sentences from the reservoir training set to use for model analysis, and sample 10,000 sentences from the reservoir test set for final performance numbers. we reserve the remainder of the reservoir training set for experiments that require filtering out a portion of the data. we give an overview of the reservoir corpus' split sizes and partitioning in ## experiments in our experiments, we compare three types of models in total: first, a baseline "untagged" bart model finetuned on pure amr-to-text generation with no control tags; second, bart models finetuned to produce text from linearized amr with syntax tags for each mode of syntax individually; and finally, a set of models finetuned with control tags for multiple modes of syntax. for both voice and tense, we report results both on our own test set (10k sentences) as well as the test set of the proxy subset of the amr bank (approximately 800 sentences) for comparison on gold amr parses. for entity, we report results only on our own test set, as after filtering the proxy test set to remove sentences with :desc tags and sentences with greater or fewer than one person node, only 16 sentences remained. ## analysis 6.1 syntax interactions in order to investigate the interaction between the different modes of syntax, we additionally train a set of models that incorporate multiple types of tags. these are reported in the results tables as the "voice+tense", "voice+entity", and "voice+tense+entity" models. interestingly, adding tense seems to improve performance on voice, whereas the converse does not hold, while entity realization seems to be an orthogonal task: the "voice+tense" model achieves better performance on voice but not on tense in the more difficult flipped setting, whereas combining entity realization with other types of syntax leads to a drop in performance. ## conclusion and future directions in this paper, we investigate the controllability of three modes of syntax -verb voice, verb tense, and syntactic entity realization -when using bart to generate text from amr input augmented with automatically extracted syntactic labels. we find that all three modes of syntax are more reliably reproduced when these augmentations are added, yielding more accurate and more faithful outputs. further, even when we artificially engineer the distribution of tags to be as far from training as possible, the models with tags still far outperform the model without, without sacrificing fluency. ultimately, our labeling strategy allows us to automatically create data at good enough quality and scale that we can successfully finetune a pretrained lm generator to reliably follow a content plan augmented with fine-grained syntactic tags. this setting is particularly useful in tasks such as summarization, where there is no one-to-one mapping between input and output content, and thus no appropriate place to insert fine-grained tags in the original input. there are many avenues of possible future study with our method of controllable generation; one natural future direction would be an expansion of the setting from individual sentence amrs to a collection of amrs forming a contiguous passage or document, as is the ultimate goal of this work.