Assessing the State of the Art in Scene Segmentation (2025)

Abstract: The detection of scenes in literary texts is a recently introduced segmentation task in computational literary studies. Its goal is to partition a fictional text into segments that are coherent across the dimensions time, space, action and character constellation. This task is very challenging for automatic methods, since it requires a high-level understanding of the text. In this paper, we provide a thorough analysis of the State of the Art and challenges in this task, identifying and solving a problem in the training procedure for previous approaches, analysing the generalisation capabilities of the models and comparing the BERT-based SotA to current Llama models, as well as providing an analysis of what causes errors in the models. Our change in training procedure provides a significant increase in performance. We find that Llama-based models are more robust to different types of texts, while their overall performance is slightly worse than that of BERT-based models.

## introduction research in the area of computational literary studies (cls), where one of the main goals is to provide a computational understanding of literary texts, often has to deal with texts that are much longer than other areas of nlp. this leads to challenges for tasks like co-reference resolution, where models scale very badly with input length @xcite and their performance deteriorates on longer texts. therefore, splitting these texts into coherent segments can be very helpful for further analysis. additionally, a segmentation that is rooted in narrative theory can also be used to gain insights into the development of the plot of a story @xcite . in this paper, we address several questions with regard to the state of the art in scene segmentation. our main contributions are as follows: (1) we identify a problem in the current state of the art model based on sequential sentence classification and propose a modified training sample generation scheme designed to alleviate this issue. (2) we analyse how the relation between training and test data influences the performance of scene segmentation by employing training and test sets composed of different types of literature (dime novels, high literature and an additional out of distribution test set). (3) we question the previous evaluation metrics used for scene segmentation, arguing that the exact f1-score underestimates the performance of the models. (4) we evaluate the performance of current llms, specifically versions of llama-3 and gpt-4o, in both a zero-shot prompting and a fine-tuning setting to see how they compare to the sota in scene segmentation, which is still based on bert. (5) finally, we analyse the errors made by the best model and identify open challenges. 1 ## task description the task of automatic scene segmentation was formally defined by @xcite . the goal is to segment a literary text into scenes, which are parts of the text with a consistent pattern in the four dimensions time, space, character and action. a break in these dimensions corresponds to a scene change. in addition to these scenes, texts can also contain non-scenes, which are (usually short) parts of the text without a consistent pattern in the four dimensions. these non-scenes are commonly used for introductory segments before a scene or to summarise the events of a longer period of time. @xcite define the task of scene segmentation as one of two sentence-level classification tasks: the simpler, binary version only uses the classes noborder and border (ignoring the distinction between scenes and non-scenes), while the full task has the four classes scene-scene, scene-nonscene, nonscene-scene or noborder. in this paper, we focus on the binary task for several reasons: (a) the number of non-scenes in a text is usually very small compared to the number of scenes (cf. note that our labels can directly be mapped to iob2 labels @xcite : the scene-begin label b is identical to our border label and the within-scene label i corresponds to noborder. we decided to use the border/noborder labels for consistency with previous work @xcite . ## state of the art the concept of scenes has existed in narratology for a long time @xcite as a part of the narration where the amount of time that passes in the narrative (story time, or histoire) and the amount of time covered by its narration (narrated time, or discours) are roughly equal. while there have been some earlier computational approaches to scene segmentation @xcite , the task has more recently been discussed in detail @xcite and finally been formalised, along with the introduction of an annotated dataset and experiments based on a bertmodel @xcite . consequently, there has been a shared task @xcite introducing several new ideas. the existing approaches can be divided into multiple categories: unsupervised baseline approaches @xcite present two baseline approaches using standard text segmentation techniques, namely text-tiling @xcite and topictiling @xcite . unsurprisingly, both of these approaches are not able to capture the notion of scenes at all, as they are not designed with the notion of scenes in mind. ## approaches to scene segmentation we compare different approaches to our task, based on either a bert-style model or a current llm. ## data we use an extended version of the dataset from @xcite , containing annotations for 41 texts or text fragments, with the assignment of texts to the different sets is given in ## experiments 6.1 evaluation metrics the evaluation of scene segmentation poses some challenges of its own, wherefore multiple evaluation metrics have been used in the past. in their original task description paper, @xcite propose to use both the f1-score for an exact match of predicted and annotated scene boundaries, and one component of mathet's γ @xcite for a more lenient score. in this paper, we propose the use of a less strict modification of the f1-score, trying to capture the advantages of both metrics. we provide a discussion of additional possible metrics in appendix a.3. ## analysis we are interested in finding out what causes our models to either miss (false negative) or add (false positive) scene borders. to this end, we analyse our best model's predictions on the test data in detail. ## discussion our experiments reveal several interesting findings regarding scene segmentation. first, we find that building samples for ssc using the half-stride strategy greatly improves the performance. secondly, we find that bert-based models are sensitive to the type of texts in training and test set -mixing dime novels and high literature in the training set leads to better performance on non-dime novels. however, we also see that the model's performance on dime novels does not improve with the additional training data. this is very interesting, as it suggests that, contrary to our assumption, the performance of scene segmentation is not limited by the amount of available training data, but rather by the capabilities of the model. our analysis reveals challenging cases and suggests some changes in the way the task is approached: since borders to non-scenes cause a notable amount of errors, our decision to not include this distinction may have to be revised: as it is, the model does not have any indication that there are structurally different segments in the texts. distinguishing between scenes and non-scenes may therefore actually be beneficial. manual analysis of the errors shows that, in many cases, the predictions of the model can be argued to be correct, depending on how fine-grained the segmentation is supposed to be. this suggests that, even though the metrics still leave room for improvement, the produced segmentation can be sufficient for downstream tasks. ## conclusion we have explored the state of the art for automatic scene segmentation in fictional texts. we achieve a new best result for the task, with a relaxed f1-score of 0.68 on a diverse test set with the best model. while this still leaves room for improvement, it shows that it is feasible to train models that achieve good results for automatic scene segmentation. our analysis shows that many of the best model's wrong predictions are still understandable and could be argued to be correct in many cases, depending on how fine-grained one wants the scene segmentation to be. we have also demonstrated that the task is challenging even for current large language models, making it a valuable addition to benchmarks aiming to explore the limits of these models.