Stylistic MR -to-Text Generation Using Pre-trained Language Models (2021)

Abstract: We explore the ability of pre-trained language models BART, an encoder-decoder model, GPT2 and GPT-Neo, both decoder-only models for generating sentences from structured MR tags as input. We observe best results on several metrics for the YelpNLG and E2E datasets. Style based implicit tags such as emotion, sentiment, length etc., allows for controlled generation but it is typically not present in MR. We present an analysis on YelpNLG showing BART can express the content with stylistic variations in the structure of the sentence. Motivated with the results, we define a new task of emotional situation generation from various POS tags and emotion label values as MR using EmpatheticDialogues dataset and report a baseline. Encoder-Decoder attention analysis shows that BART learns different aspects in MR at various layers and heads.

## introduction recent advances in nlg focus on generating text from structured data encoded as meaningful representations (mr). mr typically comprises of semantic content to be realized for generation. this can be used for automating writing reports from tabular data, descriptions and reviews for products or restaurants from catalog, etc. however, style based implicit tags can add dynamic, engaging and immersive effect in real world nlg applications such as social and empathetic chatbots. the style aspects along with content information allows generating varied and customized text with same content. in this work, we explore capabilities of an encoderdecoder model, bart @xcite , and two decoder-only models, gpt2 @xcite and @xcite for mrto-text generation task. we evaluate bart, gpt2 and gpt-neo on three datasets, one for content and other for both content and style. these datasets include e2e original and clean version @xcite ) @xcite which are restaurant description datasets comprising of content based mr and yelp nlg @xcite , a restaurant reviews corpus having both semantic and stylistic tags. we define a new task of emotional situation generation on empathetic dialogues (ed) dataset @xcite . we construct mrs using set of pos tag @xcite values from situation along with emotion label. our main contributions are defined as: a) the ability of encoder-decoder based and decoder-only pretrained transformer models to generate fluent sentences from content and style based mr. b) a new task on emotional situation generation using pos tag and emotion label values as mr and report its baseline. c) encoder-decoder attention map analysis of bart to further understand which layer and head learns which concept. ## related work existing structured data to text datasets -e2e @xcite , webnlg @xcite , @xcite , agenda (koncel-kedziorski et al., 2019) etc consider input in various formats such as slot value pair, triplets, or graph. they consist of content based semantic input in mr. recently introduced yelpnlg dataset by @xcite considers style aspect in addition to content slot value in mr and provides lstm encoder decoder baseline. our work focuses on exploring recent language model capability for content and style based mr. researchers have attempted to improve content slot value mr to text in attention based encoder decoder architectures by incorporating various techniques. @xcite performed joint training of nlu and nlg. @xcite in-dataset size content style yelp 300k restaurant[],cuisine[], food[], staff[], service[], ambiance[], price[] sentiment(positive, negative, neutral), length(short,medium,long), perspective( first,person, not first person),exclamation (has exclamation, no exclamation) e2e 50k name[], eattype[],food[],near[], pricerange[], customerrating[], area[], kidsfriendly[] na ed 25k pos values subset from [noun], [adjective], [verb], [pronoun] 32 emotion labels ## dataset description ## experiments we fine-tune pre-trained language models like bart-large, gpt2-medium and gpt-neo 125m for mr-to-text on train split of respective datasets. we use early stopping and choose the best model for evaluation on test set. other parameters used for fine tuning are adamw optimizer with a learning rate of 3e-5 and a linear learning rate scheduler. while generating the output text from mr we use beam search decoding with beam size of 4. we evaluate the generated output text using the standard automatic evaluation metrics foot_0 bleu @xcite , meteor @xcite , nist @xcite , cider (vedantam et al., 2015) and rougue @xcite , and semantic error rate (ser) @xcite . yelpnlg: we report the results for yelp nlg in ## conclusion we describe the benefits and importance of mr2text generation. we fine-tune recently introduced transformer-based language models like bart, gpt2 and gptneo, and produce results on two versions of e2e, yelpnlg and ed datasets. we have defined a new task on emphatetic dataset to emphasize the usefulness of implicit tags in nlg. quantitative and qualitative analyses show how well bart captures the specifications and brings stylistic variations in generated outputs.