SAPPHIRE : Approaches for Enhanced Concept-to-Text Generation (2021)

Abstract: We motivate and propose a suite of simple but effective improvements for concept-to-text generation called SAPPHIRE: Set Augmentation and Post-hoc PHrase Infilling and REcombination. We demonstrate their effectiveness on generative commonsense reasoning, a.k.a. the CommonGen task, through experiments using both BART and T5 models. Through extensive automatic and human evaluation, we show that SAPPHIRE noticeably improves model performance. An in-depth qualitative analysis illustrates that SAPPHIRE effectively addresses many issues of the baseline model generations, including lack of commonsense, insufficient specificity, and poor fluency.

## introduction there has been increasing interest in constrained text generation tasks which involve constructing natural language outputs under certain preconditions, such as particular words that must appear in the output sentences. a related area of work is data-to-text natural language generation (nlg), which requires generating natural language descriptions of structured or semi-structured data inputs. many constrained text generation and nlg tasks share commonalities, one of which is their task formulation: a set of inputs must be converted into natural language sentences. this set of inputs can be, in many cases, thought of as concepts, e.g. higher-level words or structures that play an important role in the generated text. with the increased popularity of transformerbased models and their application to many nlp tasks, performance on many text generation tasks has improved considerably. much progress in recent years has been from the investigation of model improvements, such as larger and more effectively pretrained language generation models. however, are there simple and effective approaches to improving performance on these tasks that can come from the data itself? further, can we potentially use the outputs of these models themselves to further improve their task performance -a "selfintrospection" of sorts? in this paper, we show that the answer is yes. we propose a suite of simple but effective improvements for concept-to-text generation called sap-phire: set augmentation and post-hoc phrase infilling and recombination. specifically, sap-phire is composed of two major approaches: 1) the augmentation of input concept sets ( §4.1), 2) the recombination of phrases extracted from baseline generations into more fluent and logical text ( §4.2). these are mainly model-agnostic improvements that rely on the data itself and the model's own initial generations, respectively. 1 we focus on generative commonsense reasoning, or @xcite , which involves generating logical sentences describing an everyday scenario from a set of concepts, which in this case are individual words that must be represented in the output text in some form. commongen is a challenging instance of constrained text generation that assesses 1) relational reasoning abilities using commonsense knowledge, and 2) compositional generalization capabilities to piece together concept combinations. further, commongen's task formulation and evaluation methodology are quite broadly applicable and encompassing, making it a good benchmark for general constrained text generation capability. further, this is an opportune moment to investigate this task as commonsense ability of nlp models, particularly for generation, has received increasing community attention through works like comet @xcite . we perform experiments on varying sizes of two state-of-the-art transformer-based language generation models: bart @xcite and t5 @xcite . we first conduct an extensive correlation study ( §3.1) and qualitative analysis ( §3.2) of these models' generations after simply training on commongen. we find that performance is positively correlated with concept set size, motivating concept set augmentation. we also find that generations contain issues related to commonsense and fluency which can possibly be addressed through piecing the texts back together in different ways, motivating phrase recombination. fleshing out our first intuition -we devise two methods to augment concepts from references during training through extracted keywords ( §4.1.1) and attention matrices ( §4.1.2). for the phrase recombination intuition, we propose two realizations based on a new training stage ( §4.2.1) and masked infilling ( §4.2.2). finally, through comprehensive evaluation ( §6), we show how the sapphire suite drives up model performance across metrics, besides addressing aforementioned baseline deficiencies on commonsense, specificity, and fluency. ## results and analysis automatic evaluation results on test cg can be found in tables 7 , 8 , 9, 10, and results on test o in ## conclusion and future work in conclusion, we motivated and proposed several improvements for concept-to-text generation which we call sapphire: set augmentation and post-hoc phrase infilling and recombination. we demonstrated their effectiveness on commongen through experiments on bart and t5. extensive evaluation showed that sapphire improves model performance, addresses many issues of the baselines, and has potential for further exploration. potential future work includes improving mask infilling performance, and trying combinations of sapphire techniques as they could be complementary. better exploiting regularities of commongen-like tasks, e.g. invariance to input order, presents another avenue. sapphire methods can also be investigated for other data-to-text nlg tasks, e.g. webnlg, and explored for applications such as improving the commonsense reasoning of personalized dialog agents @xcite , data augmentation for nlg @xcite , and constructing pseudo-references for long-context nlg @xcite .