Post-Editing Extractive Summaries by Definiteness Prediction (2021)

Abstract: Extractive summarization has been the mainstay of automatic summarization for decades. Despite all the progress, extractive summarizers still suffer from shortcomings including coreference issues arising from extracting sentences away from their original context in the source document. This affects the coherence and readability of extractive summaries. In this work, we propose a lightweight post-editing step for extractive summaries that centers around a single linguistic decision: the definiteness of noun phrases. We conduct human evaluation studies that show that human expert judges substantially prefer the output of our proposed system over the original summaries. Moreover, based on an automatic evaluation study, we provide evidence for our system’s ability to generate linguistic decisions that lead to improved extractive summaries. We also draw insights about how the automatic system is exploiting some local cues related to the writing style of the main article texts or summary texts to make the decisions, rather than reasoning about the contexts pragmatically.

## introduction more than half a century after hans peter luhn's seminal work @xcite , automatic summarization remains a challenge, one that is increasingly pressing with the explosion of information online and elsewhere. one of the proposed approaches is extractive summarization: the task of selecting spans, typically sentences, from a source text such that they best convey the overall meaning. it is the most popular approach given its simplicity and scalability compared to more sophisticated abstractive approaches. the simplicity of this method, however, is not without its costs, as extractive summaries are known to suffer from a variety of issues. in addition to problems pertaining to verbosity @xcite , a system that centers around sentence extraction is inherently exposed to the risk of selecting a sentence that depends on a non-selected source text: the school had to deal with a suspicious package received early in the morning. a student thought to be from another district addressed a mail that had a very strong smell. police was called in. the student was eventually questioned by the police for 5 hours. ## study 2: automatic evaluation of the post-editor method in this study, we carry out a human annotation study to obtain gold standard annotations on the definiteness of noun phrases in sampled subsets of extractive summaries that are generated by different summarizers. by comparing our model's decisions to the collected annotations, we can evaluate its performance using standard classification accuracy. also, the collected dataset of annotated extractive summaries can be useful for further development and evaluation of similar post-editing systems. ## analyzing the hyperparameters effect on model performance in this section, we attempt to understand the effect of certain hyperparameters and modeling choices on the models' performance on this task. we focus on the cnndm/banditsum scenario and vary the size of the neural models in terms of depth (number of recurrent layers) and width (size of the recurrent layers) and whether the recurrent layer is bidirectional or not. we also investigate the effect of having a local context (i.e. within the current np) versus an extended context. figure 5 presents the results showing the performance of the models across these different dimensions. as a note, given the very large size of the stories dataset (∼48m samples), we decided to train on it only the biggest models (8 or 5 layers). effect of network size. the trends in the results point to the fact that, as expected, bigger networks lead to a higher performance. while the trend in performance upwards is not 100% perfect, it shows that when we start with 2 layers in the lstm case, the performance is lower in all three relevant datasets and goes up as we increase the number of layers. similarly, if we look at increasing the width of layers, a similar trend holds. ## conclusion in this work, we proposed a method to modify the output of an extractive summarizer via a postediting step involving definiteness prediction. the goal is to generate decisions on modifying articles (or not) such that the quality of the extractive summary is improved in terms of coherence and readability. we presented evidence that human expert judges show substantial preference for the output of such a system. we collected annotations of generated extractive summaries on np definiteness which we believe would be useful for further development and evaluation of similar post-editing systems. based on automatic evaluation, we validated our system's ability to generate linguistic decisions that highly overlap with the golden annotations, thus pointing at the system's efficacy and potential for generating improved extractive summaries. finally, we presented insights about how the system could be exploiting some local cues related to the writing style of the main article texts or summary texts to make the decisions, rather than pragmatically reasoning about the contexts. our work points to the importance of future research that centers around understanding the discourse context to make predictions that lead to pseudoextractive summaries of even higher quality.