Inference Time Style Control for Summarization (2021)

Abstract: How to generate summaries of different styles without requiring corpora in the target styles, or training separate models? We present two novel methods that can be deployed during summary decoding on any pre-trained Transformer-based summarization model. (1) Decoder state adjustment instantly modifies decoder final states with externally trained style scorers, to iteratively refine the output against a target style. (2) Word unit prediction constrains the word usage to impose strong lexical control during generation. In experiments of summarizing with simplicity control, automatic evaluation and human judges both find our models producing outputs in simpler languages while still informative. We also generate news headlines with various ideological leanings, which can be distinguished by humans with a reasonable probability.

## introduction generating summaries with different language styles can benefit readers of varying literacy levels @xcite or interests @xcite . significant progress has been made in abstractive summarization with large pre-trained transformers @xcite @xcite @xcite . however, style-controlled summarization is much less studied @xcite , and two key challenges have been identified: (1) lack of parallel data, and (2) expensive (re)training, e.g., separate summarizers must be trained or finetuned for a pre-defined set of styles @xcite . both challenges call for inference time methods built upon trained summarization models, to adjust styles flexibly and efficiently. to address these challenges, we investigate justin-time style control techniques that can be directly applied to any pre-trained sequence-to-sequence (seq2seq) summarization model. we study two methods that leverage external classifiers to favor daily mail article: . . . a 16-year-old who was born a girl but identifies as a boy has been granted the opportunity to go through male puberty thanks to hormone treatment. . . . the transgender boy, who has felt as though he is living in the wrong body since he was a child, has been given permission by a brisbane-based judge to receive testosterone injections . . . the generation of words for a given style. first, decoder state adjustment is proposed to alter the decoder final states with feedback signaled by style scorers, which are trained to capture global property. second, to offer stronger lexical control, we introduce word unit prediction that directly constrains the output vocabulary. example system outputs are displayed in fig. 1 . notably, our techniques are deployed at inference time so that the summary style can be adaptively adjusted during decoding. we experiment with two tasks: (1) simplicity control for document summarization with cnn/daily mail, and (2) headline generation with various ideological stances on news articles from the semeval task @xcite and a newly curated corpus consisting of multi-perspective stories from allsides foot_0 . in this work, the algorithms are experimented with the bart model @xcite , though they also work with other transformer models. both automatic and human evaluations show that our models produce summaries in simpler languages than competitive baselines, and the informativeness is on par with a vanilla bart. moreover, headlines generated by our models embody stronger ideological leaning than nontrivial comparisons. foot_1 2 related work summarizing documents into different styles are mainly studied on news articles, where one appends style codes as extra embeddings to the encoder @xcite , or connects separate decoders with a shared encoder @xcite . similar to our work, @xcite leverage large pre-trained seq2seq models, but they modify model architecture by adding extra style-specific parameters. nonetheless, existing work requires training new summarizers for different target styles or modifying the model structure. in contrast, our methods only affect decoder states or lexical choices during inference, allowing on-demand style adjustment for summary generation. style-controlled text generation has received significant research attentions, especially where parallel data is scant @xcite @xcite . typical solutions involve disentangling style representation from content representation, and are often built upon autoencoders @xcite with adversarial training objectives @xcite . the target style is then plugged in during generation. recently, @xcite propose plug and play language models (pplms) to alter the generation style by modifying all key-value pairs in the transformer, which requires heavy computation during inference. @xcite then employ a generative discriminator (gedi) to improve efficiency. our methods are more efficient since we only modify the decoder final states or curtail the vocabulary. concretely, to generate the t-th token, a style score of q(z|y 1:t+2 ) is first computed. in addition to what have been generated up to step t -1, we also sample y t and two future tokens for style estimation. the decoder state is updated as follows: below, we define one discriminative and one generative style classifier, to illustrate the method. ## simplicity-controlled document summarization for experiments, we use bart fine-tuned on the cnn/dailymail (cnn/dm) @xcite , by following @xcite for data preprocessing and splitting. the numbers of data in train, validation and test splits are 287,188, 13,367 and 11,490, respectively. we use paragraph pairs from normal and simple english wikipedia articles in @xcite for simplicity style scorer and class-conditional language model training. we split the pairs into 86,467, 10,778, and 10,788 for training, validation and testing, respectively. on the test set, our simplicity style scorer achieves an f1 score of 89.7 and our class-conditional language model achieves a perplexity of 30.35. to learn the word unit predictor, for each paragraph pair, the predictor reads in the normal version and is trained to predict the word units used in the simple version. for the dynamic version, it predicts which word units are used to generate the rest of the text, after every 5 steps. recalls for the two predictors on the test set are 81.5 and 80.0. for comparison, we consider reranking beams based on our style score at the last step. we also use a label-controlled (lblctrl) baseline as described in @xcite , where summaries in the training data are labeled as simple or normal by our scorer. we further compare with gedi and two pipeline models: a style transfer model @xcite applied on the output of bart (ctrlgen) and a normal-to-simple translation model fine-tuned from bart (trans), both trained on wikipedia. finally, we consider lightls @xcite , a rule-based lexical simplification model. ## ideology-controlled headline generation to generate news headlines of various ideological leanings, we use the semeval hyperpartisan news detection dataset @xcite , where each article is labeled with a stance: left, leaning left, neutral, leaning right, or right. here, we combine left and leaning-left articles into one bucket, and similarly for right and leaning-right articles. we use the lead paragraph as the input, and the headline as the target generation. the data is processed following @xcite , and split into 346,985 for training, 30,000 each for validation and testing. details of the ideology distribution for semeval are in appendix b. we fine-tune bart and train ideology classifiers on the semeval training set. first, two binary style scorers are trained on headlines of left and right stances, with f1 scores of 76.1 and 78.0, respectively. one class-conditional language model is trained on headlines with a stance token (left or right) prepended, achieving a perplexity of 54.7. to learn the word unit predictor for the left (and similarly for the right), we use samples that are labeled as left-leaning, treat the lead paragraph as the input, and then predict the word units used in the headline. recalls for our predictors range from 77.8 to 83.5. ## conclusion we present two just-in-time style control methods, which can be used in any transformer-based summarization models. the decoder state adjustment technique modifies decoder final states based on externally trained style scorers. to gain stronger lexical control, word unit prediction directly narrows the vocabulary for generation. human judges rate our system summaries to be simpler with better readability. we are also able to generate headlines with different ideological leanings. in our human evaluation of ideology-controlled headline generation, we use data collected from allsides. the allsides news clusters are curated by editors. the stance labels for different publishers are provided by allsides, which are synthesized from blind surveys, editorial reviews, third-party analyses, independent reviews, and community feedback. we collect all the allsides news clusters by april 26, 2020. after removing empty clusters, the total number of news clusters is 4,422. among them, 2,985 clusters contain articles written in all three stances. for each article in the cluster, we keep the first paragraph and pair it with the headline. we remove the bylines in the first paragraphs.