NEWTS : A Corpus for News Topic-Focused Summarization (2022)

Abstract: Text summarization models are approaching human levels of fidelity. Existing benchmarking corpora provide concordant pairs of full and abridged versions of Web, news or professional content. To date, all summarization datasets operate under a one-size-fits-all paradigm that may not reflect the full range of organic summarization needs. Several recently proposed models (e.g., plug and play language models) have the capacity to condition the generated summaries on a desired range of themes. These capacities remain largely unused and unevaluated as there is no dedicated dataset that would support the task of topic-focused summarization. This paper introduces the first topical summarization corpus NEWTS, based on the well-known CNN/Dailymail dataset, and annotated via online crowd-sourcing. Each source article is paired with two reference summaries, each focusing on a different theme of the source document. We evaluate a representative range of existing techniques and analyze the effectiveness of different prompting methods.

## introduction with the recent advances in neural sequence-tosequence models, the automatic generation of text has reached unparalleled levels of fidelity. abstractive summarization models that aim at generating condensed versions of a source article have outperformed lead-3 baselines on most benchmark datasets @xcite . however, all existing summarization benchmarks assume a one-size-fits-all paradigm under which model output is evaluated based on similarity to general-purpose reference summaries reflecting the full content of the original document. while certainly a necessary step, such evaluation approaches might not reflect the full range of summarization * the first author and the second author have an equal contribution. needs anymore. there are manifold settings in which tailored summaries matching the interests of the reader may be required. some examples include the summarization of complex event streams with a focus on regions, entities or topics of interest for journalists or analysts, understanding reviews or opinions from different perspectives @xcite , the summarization of electronic health records with a focus on the medical sub-specialty of the physician reader, or any other form of personalized summarization targeting explicitly defined or implicitly mined preference parameters. several recently proposed text generation models already offer the potential of steering the generation process to conform to specific topic distributions @xcite , or sentiment polarity @xcite . plug and play language models (pplm) @xcite let us condition the generation process on themes of interest and text style transfer controls selected attributes, such as politeness, emotions, or humor of the generated text @xcite . despite increased efforts and interest in con-trolled summarization, no dataset exists on which these models can be evaluated. this paper closes this gap by introducing newts, a news topicfocused summarization corpus for the controlled generation of text. it is based on documents from the well-known cnn/dailymail dataset, to which it adds new topic-focused summaries. figure 1 illustrates an article summarized with respect to two different topics. we believe that newts will significantly enrich the existing range of benchmarking collections, allowing the research community to better study and evaluate controlled text generation for summarization. the main contributions of this paper are: the remainder of this paper is organized as follows: section 2 presents previous work on datasets for text generation. next, section 3 explains the dataset collection methodology and describes the resulting corpus. section 4 discusses several existing models that we fine-tune and evaluate on the dataset. section 5 presents an evaluation of these models and the various prompting strategies. finally, section 6 concludes with an outlook on future work. ## related work in this section, we review existing work focusing on (1) controlled text generation and (2) existing datasets in this domain. we note that this paper presents the first dataset on topic-focused abstractive summarization. ## a novel dataset for controlled summarization in this section, we present newts, a new dataset for controlled topic-focused text generation. we first elaborate on the steps to building the dataset. subsequently, we present detailed statistics about the dataset. our dataset is built based on the well-known cnn/dailymail dataset @xcite , introducing an all-new facet of topical human-written summaries. for this purpose, we annotate a sample of the news articles from the cnn/dailymail dataset via online crowd-sourcing such that each article is paired with two topicfocused human-written summaries corresponding to the top two topics present in the source article. figure 2 presents the steps to creating the dataset explained in detail bellow. computing topics for the dataset. we begin by computing a 250-topic latent dirichlet allocation (lda) @xcite model on the training portion of the cnn/dailymail dataset. lda was selected due to convenience of use, and @xmath0, 300] range. from this model, we manually discard noisy or uninformative topics, keeping only the top 20% (50 topics) with the highest normalized point-wise mutual information (npmi) coherence score @xcite . we perform this aggressive pruning of topics out of feasibility considerations regarding the number of documents per topic provided for fine-tuning neural summarization models. a list of all 50 topics is presented in the appendix. selecting articles for annotation. after computing the 50 target topics of the dataset, we search the cnn/dailymail dataset for source articles containing at least two topics from the pool of 50 topics with a topic prevalence above an empirically determined threshold. by identifying documents that contain at least two topics with a topic prevalence above the empirical threshold 0.1, and a cumulative probability of both topics above 0.30, we ensure that the main content of the source article can be captured by focusing on the two main topics. consequently, figure 4: comparison of per topic normalized counts of train documents of our dataset versus cnn/dailymail the automatic filtering script is developed to identify and reject summaries that are too short (i.e., shorter than three sentences required from the workers) or do not form a grammatical sentence, summaries that are not related to the topics discussed in the source article, summaries that do not mention the same entities discussed in the source article (using named entity recognition) and summaries that contain exact copy-pasting of full sentences from the source article. to check the topics of the summary and compare them with that of the source article, the script uses the same lda topic model described earlier in this section. subsequently, the script is validated by conducting three pilot studies, each annotating 100 documents, bringing the total number of documents tested to 300. we manually assess each annotation in order to evaluate the script. in the third pilot study, our script reached 100% agreement with two independent human experts in terms of accepting/rejecting the annotations. we still conduct manual spot checks of the script output throughout the crowd-sourcing process to ensure a high-quality dataset. one of the two human experts read each sampled annotation and determine whether the quality satisfies the task description and the criteria explained earlier and rejects those annotations that do not meet the requirements. we use a z-test with a 95% confidence level and an error margin of +/ -9.24% (i.e., from 85.76% to 100% of our population) as our sampling technique. therefore, with a confidence of 95%, high quality for the annotations is ensured. designing prompts for conditional text generation. in order to be able to condition a generation process sequence-to-sequence models on certain topics for producing summaries, we design four different prompt types paired with each summary to allow advanced prompt engineering techniques. in the following, we explain each method: ## topical summarization models text-to-text transfer transformer: the t5 (text-to-text transfer transformer) model is an important example of the transformer family (raffel et al., 2019) that uses transfer-learning on the original transformer architecture @xcite . the authors study several variants of the transformer architecture and finally fine-tune them on different natural language processing tasks. the main difference from the original model is the use of relative positional embeddings as an explicit position signal of the tokens. bart: the next model that is noteworthy in this domain is bart @xcite . bart is a denoising autoencoder for pretraining sequenceto-sequence natural language processing models. it is trained by "corrupting text with an arbitrary noising function and learning a model to reconstruct the original text" @xcite . analogous to the t5 model, bart is based on the transformer architecture @xcite . it uses a number of noising approaches, such as token masking, token deletion, randomly shuffling the order of the original sentences, and a novel infilling scheme, where spans of text are replaced with a single mask token. the only major difference to the transformer architecture is that, following gpt, the authors replace relu activation functions with gelus @xcite . they also state that their proposed architecture "is closely related to that used in bert, with the following differences: (1) each layer of the decoder additionally performs cross-attention over the final hidden layer of the encoder (as in the transformer sequence-to-sequence model); and (2) bert uses an additional feed-forward network before word prediction, which bart does not" @xcite . bart is then fine-tuned on in-domain data for text generation tasks such as abstractive summarization. prophetnet: the final model in this category is prophetnet @xcite , which currently represents the state-of-the-art in abstractive summarization. this model also utilizes the transformer architecture @xcite . the main feature of prophetnet is changing the original sequence-to-sequence optimization problem of predicting the next single token into predicting the n next tokens simultaneously. the authors show that this approach outperforms all other baselines in abstractive summarization in terms of rouge scores. plug and play language models: the plug and play language model (pplm) @xcite is based on gpt-2 using the same original transformer architecture @xcite as the models above. pplm uses gpt-2 for text generation. however, it comes with an attribute model that conditions the generation process on given or previously generated text. the attribute model is fed with a bag of words signaling the target topical focus to the model. customizable abstractive topic-based summarization: finally, we include the customizable abstractive topic-based summarization (cats) @xcite model as an example of pre-transformer seq-to-seq models based on lstms. the encoder-decoder architecture has bidirectional lstms as the encoder and an lstm network as the decoder. the model utilizes attention weights governed by an lda topic model to modify the attention weights of the input tokens as represented by the encoder based on their topic assignment. this process utilizes a set of pre-defined topics derived from target summaries to learn the topics the output text should cover. as we observe in the table, in the case of 'bartb,' 't5-b', 't5-l' as well as 'prophetnet,' the model variations with topical prompts outperform both the 'no prompt' version as well as the 'cnn-dm' version in terms of the rouge scores. we do not observe a conclusive pattern when comparing the different prompting methods in terms of the rouge scores. that is, there is no one prompt that leads to a higher rouge performance for all models. as a result, we conclude that while the topical prompts do lead to performance improvement on the topic-focused summarization task, we do not observe a conclusive superiority pattern among the prompts in terms of the rouge performance. evaluating the topicality of output summaries. in the second experiment, we evaluate the topical focus of the generated summaries by each model in terms of the topic probability score computed by the lda topic model, indicating the strength of a target topic presence. therefore, we design an experiment to assess the performance of the different models with different prompt types in how topic-focused their output summaries are. for this purpose, we utilize the lda topic model to compute a per target-topic score in each generated summary. then we compute the average of this score across all generated summaries for their corresponding pre-defined target topic. we expect the models using topical information to have a higher topic_focus score. we present the results of this experiment in the right-most column of from the results of this experiment, we observe that in all cases, the topical prompt variations of each model outperform the 'cnn-dm' variation indicating that the models trained for topic-focused summarization produce summaries that are more target-topic-oriented. subsequently, we observe that topic sentence prompts outperform all other prompting techniques in achieving a high lda target-topic score, suggesting that topic sentence prompting provides models with superior topic context information. ## conclusions and future work this paper designs and releases the first publicly available dataset for controlled topic-focused abstractive summarization, newts. our dataset encompasses four prompt types to allow various conditional text generation techniques. we showed through extensive experimentation that the new dataset is of high quality. we believe that this dataset will serve the community to advance research in controlled text generation and topical summarization as a foundation for future research. our findings indicate that the sequence-tosequence transformer baselines fine-tuned with topical prompting outperform the non-topical variation model counterparts showing that the models do learn topical representations for a topic-focused source of summary summary text ground truth sum-mary1 (pop music) after experiencing some terrible customer service on an airline, a band wrote a sarcastic song about the experience. it became a hit, notably among other passengers on that airline. however, not everyone is impressed with their musical talent and lyrics.