Two-step Text Summarization for Long-form Biographical Narrative Genre (2023)

Abstract: Transforming narrative structure to implicit discourse relations in long-form text has recently seen a mindset shift toward assessing generation consistency. To this extent, summarization of lengthy biographical discourse is of practical benefit to readers, as it helps them decide whether immersing for days or weeks in a bulky book turns a rewarding experience. Machine-generated summaries can reduce the cognitive load and the time spent by authors to write the summary. Nevertheless, summarization faces significant challenges of factual inconsistencies with respect to the inputs. In this paper, we explored a two-step summary generation aimed to retain source-summary faithfulness. Our method uses a graph representation to rank sentence saliency in each of the novel chapters, leading to distributing summary segments in distinct regions of the chapter. Basing on the previously extracted sentences we produced an abstractive summary in a manner more computationally tractable for detecting inconsistent information. We conducted a series of quantitative analyses on a test set of four long biographical novels and showed to improve summarization quality in automatic evaluation over both single-tier settings and external baselines.

## introduction text summarization is a principal tool for reasoning about narrative structure and foretell the content of a literary novel in a succinct form. dated four decades back, the earlier seminal work by @xcite pursued analytical summarization of narratives, and offered a graphical representation of human-generated plot units. in this graph, plot units are defined as conceptual elements referring to propositions or states that are linked by character relations. to produce a distilled version of the original discourse, a vast amount of information are selectively ignored by the reader. similarly, traversing the graph identifies complex elements that are central to the story, and thus points of high relevance for summaries, and ones considered peripheral details. recently, the domain of narrative understanding has gained interest of the research community @xcite . a wide array of computational models developed by language technology professionals provided for expressive generative textualsummaries. presently, the prevailing approach to natural language generation (nlg) tasks, including summarization, is data driven and uses a sequenceto-sequence neural model pretrained on large text corpora. our work centers on evaluating the quality of producing summaries from chapters of longform biographical novels. unlike fictional narratives that require concatenating chapter summaries due to an inherit progressive plot nature, biographical chapters are relatively context independent and thus more readily manageable individually. automatic generation of fluent summaries in the literary domain can be useful to complement the short description of a book provided by the author and to a certain extent assist in constructing expert critiques. the work by @xcite demonstrated that a functional unit approach to summarizing computational storytelling can perform at around human level and contribute to better framing. we note that the narrative summarization task-while a rich source of innovation-is by and large untapped. pretrained language models based on the transformer network @xcite have achieved state-of-the-art performance generating fluent summaries from short input text. however, for long documents, model efficiency and summary quality characterized by remaining faithful to the respectful source present a challenge to natural language generation practitioners @xcite . to mitigate the severity, nlg research applied both topical and generic approaches to the task of summary generation, distinguishing extractive summarization that produces high lexical overlap between a summary and the source document, and hence tends to be factually consistent. while abstractive summaries are prone to unaligned content that is not obviously inferable from the original text. one of the more constraining facet of current neural models tasked with producing abstractive summarizations is that the generated text can contain factually incorrect information with respect to the grounding text they are conditioned on. summary inconsistencies are diverse and may include inversion, also known as negation, incorrect use of an entity that transpires as object swapping, or the introduction of an entity not in the original document, recognized as hallucination. @xcite conducted a large-scale study and concluded that hallucination is the most critical to the coherence of abstractive summaries, while @xcite developed a detection approach that separates factual from non-factual hallucinations. the complexity of the summarization task made automatic evaluation particularly challenging. in their recent line of work, @xcite proposed the intuition of information alignment between input and output text, and developed unified and interpretable metrics across a multitude of diverse nlg tasks. distinctly for generative summaries, they offered effective definitions of relevance and consistency, widely identified as key aspects to characterize generation quality. supported by robust theoretical grounds, their prevailing definitions strongly correlate with human judgment on how to concisely describe the most salient content in the input document. we adopted their interpretations in our empirical analysis and extended the consistency measure to a chapter-level rather than book-level over our test set of literary novels. in ## related work we briefly survey existing methods that propose multi-stage text summarization systems evaluated on datasets from a broad range of domains. ling and rush (2017) introduced a coarse-tofine attention model that reads a document hierarchically, using coarse attention to select top-level blocks of text and fine attention to read the tokens of the chosen blocks. their proposed summarizer scales linearly with the number of top-level chunks and effectively handles long sequences. however, their model performance lagged behind the standard instantiation baseline of the attention function on rouge similarity metrics. xu and lapata (2020) proposed a coarse-to-fine modeling framework for extractive summarization applied to query focused multi-document. their system incorporates a relevance estimator for retrieving textual segments-such as sentences or longer passages associated with a query-an evidence estimator which further isolates segments likely to contain answers to the query, and a centrality estimator which finally selects which segments to include in the summary. our extractive summary component is resemblant in spirit to their centrality estimator, however we use a sentence transformers (sbert; reimers and gurevych, 2019) model to generate contextual sentence embeddings that follows producing a sentence similarity matrix for computing graph centrality based ranking. @xcite explored transformer language models and proposed an extract-thensummarize computational pipeline for long documents. their model consists of an extractive element comprising a hierarchical neural encoder that outputs sentence representations of either a pointer to input sentences or to the result of sentence classification; and a transformer language model conditioned on the extracted sentences as well as on either a part of or the entire input document to generate the summary. their system was shown to outperform several baselines on similarity metrics, however, a discussion on factual correctness and consistency analyses of experimental results appears relatively sparse. gidiotis and tsoumakas (2020) proposed a divide-and-conquer method by splitting the input into multiple segments, summarizing them separately, and combining the summary pieces. basing on smaller source and target summary pairs that are focused on a specific aspect of the text, results in better alignment and considerable reduction of computation complexity. they used a basic sequence-to-sequence model and incorporated a rotational unit of memory @xcite in its decoder that led to a more stable training and slightly improved f1 similarity scores. content quality of their generated summaries relies entirely on rouge similarity metrics and could benefit from a broader evaluation framework such as offered by @xcite . more recently, @xcite proposed a multi-stage split-then-summarize framework to generate summaries from long-form documents. each source text divides into segments, matching each with a subset of target text. a coarse summary is generated for each segment and further concatenated as input to the next stage. after multiple stages of compression and summarization, a final stage produces a fine-grained summary. their improved performance across baselines renders relatively low bi-gram scores, most likely owing to over-compression of source text. an effective abstractive text summarization approach that first compresses long input text into a relatively short input sequence, and follows with efficient long-form document finetuning demonstrated comparable performance at a significantly lower computational cost @xcite . keen on a specific application, @xcite generate movie plots given movie scripts, by applying heuristic evaluation to extract actions and essential dialogues, a representation that reduces the average length of input movie scripts by 66%. their system outperforms baselines on various automatic metrics. ## chapter summarization our summarization task commences with producing an extractive summary from the source text of a book chapter, and follows with generating an abstractive summary from the salient extractive content ( ## information alignment the goal of a summarization task is to concisely describe the most salient information of the input 2 https://huggingface.co/sentence-transformers 3 https://huggingface.co/sshleifer/distilbart-cnn-12-6 text. thus, the summary generated should be consistent and only contain content from the input, and the included content must be relevant. using the intuition of information alignment, defined as the extent to which the information in one generative component is grounded in another, we can evaluate summary consistency and relevance @xcite . more formally, let x 1:n and y 1:m be our respective extractive and abstractive summary text-sequences for each book chapter. summary tokens are each represented with contextual embeddings we extracted from pretrained bert @xcite . using embedding matching, the alignment vector align(@xmath0, 1] for each token in y, and amount to the maximum cosine similarity with the tokens in x (i, j) = argmax @xmath1:n, j cossim(x i , y j ), where (i, j) is a pair of token indices pointing each to a distinct summary text sequence, and 1 ≤ @xmath2. the consistency metric that measures faithfulness thus follows naturally as the average of the alignment vector scores: mean(align(@xmath3. on the other hand, relevance is implicit in our two-step model that commences with ranking source sentences by their importance. ## conclusion in this paper, we presented a summarization approach that ensures hallucination-free text generation in its first step, and follows by a more regulated and manageable production of a final abstractive summary. on a biographical literary dataset with doubled to quadrupled chapter token complexity, our method achieved superior or similar performance compared to six baseline models. empirical results show that our fact-unaware summarization can produce abstractive summaries with compelling factual consistency. noting that author-created book descriptions are often of less than adequate quality, we encourage not only spanbased but also free-form reader-written chapter summaries that are factually faithful and benefit a plausible load sharing for curating annotations.