Chapter Ordering in Novels (2022)

Abstract: Understanding narrative flow and text coherence in long-form documents (novels) remains an open problem in NLP.To gain insight, we explore the task of chapter ordering, reconstructing the original order of chapters in novel given a random permutation of the text. This can be seen as extending the well-known sentence ordering task to vastly larger documents: our task deals with over 9,000 novels with an average of twenty chapters each, versus standard sentence ordering datasets averaging only 5-8 sentences. We formulate the task of reconstructing order as a constraint solving problem, using minimum feedback arc set and traveling salesman problem optimization criteria, where the weights of the graph are generated based on models for character occurrences and chapter boundary detection, using relational chapter scores derived from RoBERTa. Our best methods yield a Spearman correlation of 0.59 on this novel and challenging task, substantially above baseline.

## introduction novels are a fundamental long-form medium for storytelling, requiring understanding of narrative flow for full comprehension of the text. although neural network-based language models demonstrate amazing performance in comprehending short text windows, the problems of understanding long-form narrative texts such as summarization remain largely open. a major challenge in research on long-form narration is the cost of annotation: the mere act of reading a novel generally requires a 10-15 hour commitment on the part of the annotator, making it clearly cost prohibitive to obtain full annotations of a significant corpus of novels. although published summaries are available for up to a few hundred popular books, annotation cost is the rate limiting step for work in this area @xcite . in this paper, we propose a new task that gets to the heart of narrative flow analysis for novels, without the need for human annotation. most novels are partitioned into chapters as a means of providing thematic breakpoints for the reader. randomly permuting the order of these chapters obviously ruins the coherence of the narrative flow of the underlying story. here we propose the new task of chapter ordering: reconstructing the original order of the book given an input random chapter permutation of the text. this task is clearly related to the task of sentence ordering @xcite @xcite , reconstructing a set of sentences in a tiny story as to maximize their coherence. while there is a large volume of prior work on sentence ordering, to our knowledge, this is the first paper to address chapter ordering. the natural approaches for the sentence task cannot be readily applied to chapter order. first, chapters in novels usually contain several hundred sentences, which cannot easily be encoded with the neural models applied to sentence ordering. second, the number of chapters in a novel (an average of 21.6 chapters/novel in our corpus) versus 5 to 8 sentences in popular datasets. this matters, because the impressive performances of the best sentence ordering models show severe degradation with longer sequences. finally, ordering sentences in a tiny story is a simpler perceptual task for humans, whereas correctly ordering the chapters of even a single novel requires hours of reading and analysis. the combination of these factors makes chapter ordering a difficult task that cannot be tackled with standard methods. the overview of our approach can be found in figure 1 . in this paper, we define the chapter ordering problem and propose a variety of neural-based models to score the relative order of chapter-pairs. our main contributions 1 include: figure 1 : high-level overview of the chapter ordering task: after partitioning and shuffling a novel's chapters, we define a network of coherence scores between chapter pairs, with the permutation reconstructed using an optimization criteria combining the traveling salesman problem (tsp) and minimum arc feedback set. we propose several natural coherence notions implicit in long-form narratives, including: there are subtle characteristics of the start and end of a story that distinguishes it from the middle of the narrative. -neighboring chapter detection: narrative flow implies that the events at the end of one chapter are often connected to the start of the next chapter. this direct correlation provides strong reinforcement in keeping certain chapters paired. -topic correlation: generalizing beyond chapter boundaries, the topical content in nearby chapters is generally more closely related than those further away. the paper is organized as follows. section 2 surveys related work in sentence ordering and whole book-oriented nlp. section 3 formalizes our task. we present our methods for coherent scoring in section 4, and in section 5 our optimization techniques. we present our experimental results in section 6 before concluding with future directions for research. ## related work the sentence ordering task was first proposed to test coherence of sentences @xcite @xcite . traditionally, this was tackled using human designed features and classical machine learning techniques. this included heuristics with markov models @xcite @xcite , k-means clustering @xcite , support vector machines @xcite @xcite and others like latent semantic analysis @xcite and conditional random fields @xcite . we also note the representation of sentence ordering as a graph in many past works as well @xcite @xcite . however, with advances in neural modeling, other approaches have developed. one category are pairwise ordering models, which deal with ordering at a pairwise level and combines the results to get a global ordering. traditionally, the models used in these approaches have been grus and lstms @xcite @xcite , but recently have moved towards transformer-based models like bert @xcite . one common approach to sentence ordering considers the sentences as a graph and optimizes the weights using algorithms such as topological sort @xcite or the asymmetric traveling salesman problem @xcite . the main disadvantage is that a pair-wise ordering of two sentences is determined in isolation from the other sentences. it may be the case that the other sentences provide context that make the ordering sensible. thus, the other main category of sentence ordering deal with set-to-sequence models, which deal with end-to-end encoder-decoder frameworks. most common are pointer networks for sequence prediction @xcite @xcite . this is often combined with other methods such as recurrent neural networks @xcite . recently, works have been incorporating graph-related networks @xcite @xcite as well. additionally, the task has been considered as a text-to-marker generation problem using bart @xcite . these methods have the advantage that all the sentences are taken in context, but lack the interpretability of the previous models discussed. we also rely heavily on literature in natural language processing on books. analysis of books have been streamlined through pipelines like booknlp @xcite as well as datasets of entities @xcite . we also find other bookrelated works that improve the quality of books as well as understanding aspects such as time @xcite . in particular, we focus on chap-ters and require proper chapter segmentation @xcite . ## problem formulation the chapter ordering task can be formulated as follows. given a book of n chapters, let @xmath0, . . . , c n ] be the sequence of chapters where each c i represents the text content of chapter i for ## coherence scoring in this section, we describe our observations of books that help us quantify relations between chapters. since the goal is to provide a benchmark on the feasibility of the chapter ordering task and not an exhaustive comparison on the effectiveness of different models, we primarily use roberta @xcite as our main pretrained language model for all relevant tasks. no hyperparameter tuning was done on any models; default values were run for all models. ## conclusion in this paper, we introduce a new task of chapter ordering as an approach to study long-form narrative understanding without cost-prohibitive human annotation. the standard approaches used in the smaller-scale sentence ordering tasks do not extend directly to chapter ordering, because longer texts exceed the capacity of current neural language models. future work may include incorporating efficient transformers that incorporate longer text windows, although we believe that approaches which explicitly incorporate logical reasoning about events may be necessary to achieve substantial progress. we will release our full dataset on publication to encourage additional research on this task.