Exploring Content Selection in Summarization of Novel Chapters (2020)

Abstract: We present a new summarization task, generating summaries of novel chapters using summary/chapter pairs from online study guides. This is a harder task than the news summarization task, given the chapter length as well as the extreme paraphrasing and generalization found in the summaries. We focus on extractive summarization, which requires the creation of a gold-standard set of extractive summaries. We present a new metric for aligning reference summary sentences with chapter sentences to create gold extracts and also experiment with different alignment methods. Our experiments demonstrate significant improvement over prior alignment approaches for our task as shown through automatic metrics and a crowd-sourced pyramid analysis.

## introduction when picking up a novel one is reading, it would be helpful to be reminded of what happened last. to address this need, we develop an approach to generate extractive summaries of novel chapters. this is much harder than the news summarization tasks on which most of the summarization field (e.g., @xcite @xcite ) focuses; chapters are on average seven times longer than news articles. there is no one-to-one correspondence between summary and chapter sentences, and the summaries in our dataset use extensive paraphrasing, while news summaries copy most of their information from the words used in the article. we focus on the task of content selection, taking an initial, extractive summarization approach given the task difficulty. 1 as the reference sum-maries are abstractive, training our model requires creating a gold-standard set of extractive summaries. we present a new approach for aligning chapter sentences with the abstractive summary sentences, incorporating weighting to rouge @xcite and meteor @xcite metrics to enable the alignment of salient words between them. we also experiment with bert @xcite alignment. we use a stable matching algorithm to select the best alignments, and show that enforcing one-toone alignments between reference summary sentences and chapter sentences is the best alignment method of those used in earlier work. we obtain a dataset of summaries from five study guide websites paired with chapter text from project gutenberg. our dataset consists of 4,383 unique chapters, each of which is paired with two to five human-written summaries. we experiment with generating summaries using our new alignment method within three models that have been developed for single document news summarization @xcite @xcite . our evaluation using automated metrics as well as a crowd-sourced pyramid evaluation shows that using the new alignment method produces significantly better results than prior work. we also experiment with extraction at different levels of granularity, hypothesizing that extracting constituents will work better than extracting sentences, since summary sentences often combine information from several different chapter sentences. here, our results are mixed and we offer an explanation for why this might be the case. our contributions include a new, challenging summarization task, experimentation that reveals potential problems with previous methods for creating extracts, and an improved method for creating gold standard extracts. ## related work relatively little work has been done in summarization of novels, but early work @xcite provided a dataset of novel/summary pairs drawn from cliffsnotes and gradesaver and developed an unsupervised system based on meade @xcite and textrank @xcite ) that showed promise. more recently, @xcite developed an approach for summarizing characters within a novel. we hypothesize that our proposed task is more feasible than summarizing the full novel. previous work has summarized documents using rhetorical structure theory (rst) @xcite to extract elementary discourse units (edus) for compression and more contentpacked summaries @xcite @xcite . some abstractive neural methods propose attention to focus on phrases within a sentence to extract @xcite . fully abstractive methods are not yet appropriate for our task due to extensive paraphrasing and generalization. while previous work on semantic textual similarity is relevant to the problem of finding alignments between chapter and summary text, the data available @xcite is not suitable for our domain, and the alignments we generated from this data were of a poorer quality than the other methods in our paper. ## data we collect summary-chapter pairs from five online study guides: barronsbooknotes (bb), bookwolf (bw), cliffsnotes (cn), gradesaver (gs) and novelguide (ng). 2 we select summaries from these sources for which the complete novel text can be found on project gutenberg. our initial dataset, for summaries with two or more sources, includes 9,560 chapter/summary pairs for 4,383 chapters drawn from 79 unique books. as our analysis shows a very long tail, two rounds of filtering were applied. first, we remove reference texts @xmath0, as these are too large to fit into mini-batches (∼10% of data). second, we remove summaries with a compres2 we do not have the rights to redistribute the data. to allow others to replicate the dataset, we provide a list of novel chapters we used at https://github.com/ manestay/novel-chapter-dataset train split statistics are given in in figure 1 , we show the first three sentences of a reference summary for chapter 11, the awakening which is paraphrased from several, nonconsecutive chapter sentences shown near the bottom of the figure. we also show a portion of the summaries from two other sources which convey the same content and illustrate the extreme level of paraphrasing as well as differences in detail. we show the full chapter and three full reference summaries in appendix a.2. ## alignment experiments to train models for content selection, we need saliency labels for each chapter segment that serve as proxy extract labels, since there are no gold extracts. in news summarization, these are typically produced by aligning reference summaries to the best matching sentences from the news article. here, we align the reference summary sentences with sentences from the chapter. gs: in this chapter mr. and mrs. pontellier participate in a battle of wills. when mr. pontellier gets back from the beach, he asks his wife to come inside. she tells him not to wait for her, at which point he becomes irritable and more forcefully tells her to come inside. ng: mr. pontellier is surprised to find edna still outside when he returns from escorting madame lebrun home. ... although he asks her to come in to the house with him, she refuses, and remains outside, exercising her own will. bw: leonce urges edna to go to bed, but she is still exhilarated and decides to stay outside in the hammock... chapter sentences: he had walked up with madame lebrun and left her at the house. "do you know it is past one o'clock? come on," and he mounted the steps and went into their room. "don't wait for me," she answered. "you will take cold out there," he said, irritably. "what folly is this? why don't you come in?" figure 1: portions of three reference summaries for the awakening, chapter 11 by kate chopin, along with chapter sentences they summarize. and summary sentences to generate gold standard extracts: 1) which similarity metric works best for alignment (section 4.1)? and 2) which alignment method works best (section 4.2)? ## summarization experiments in order to assess how alignments impact summarization, we train three extractive systems -hierarchical cnn-lstm extractor (chen and bansal, 2018) (cb), seq2seq with attention @xcite (k), and rnn @xcite . the target word length of generated summaries is based on the average summary length of similarly long chapters from the training set. 5 we also experiment with aligning and extracting at the constituent level, 6 given our observation during data analysis that summary sentences are often drawn from two different chapter sentences. we create syntactic constituents by taking sub-trees from constituent parse trees for each sentence @xcite rooted with s-tags. to ensure that constituents are long enough to be meaningful, we take the longest s-tag when one stag is embedded within others (see appendix a.5). summary quality is evaluated on f1 scores for r-{1,2,l}, and meteor. each chapter has 2-5 reference summaries and we evaluate the generated summary against all the reference summaries. part of a generated summary of extracted constituents for chapter 11, the awakening, is shown in figure 3 . the full generated summaries for this chapter (both extracted constituents and extracted sentences) are shown in appendix a.2. ## discussion and conclusion we present a new challenging task for summarization of novel chapters. we show that sentence-7 see the screen shot in appendix a.4 8 we use the best baseline alignment, @xcite with the cb model to keep model choice consistent.