Topical Segmentation of Spoken Narratives: A Test Case on Holocaust Survivor Testimonies (2022)

Abstract: The task of topical segmentation is well studied, but previous work has mostly addressed it in the context of structured, well-defined segments, such as segmentation into paragraphs, chapters, or segmenting text that originated from multiple sources. We tackle the task of segmenting running (spoken) narratives, which poses hitherto unaddressed challenges. As a test case, we address Holocaust survivor testimonies, given in English. Other than the importance of studying these testimonies for Holocaust research, we argue that they provide an interesting test case for topical segmentation, due to their unstructured surface level, relative abundance (tens of thousands of such testimonies were collected), and the relatively confined domain that they cover. We hypothesize that boundary points between segments correspond to low mutual information between the sentences proceeding and following the boundary. Based on this hypothesis, we explore a range of algorithmic approaches to the task, building on previous work on segmentation that uses generative Bayesian modeling and state-of-the-art neural machinery. Compared to manually annotated references, we find that the developed approaches show considerable improvements over previous work.

## introduction proper representation of narratives in long texts remains an open problem in nlp @xcite @xcite . high-quality representations for long texts seem crucial to the development of document-level text understanding technology, which is currently unsatisfactory @xcite . a common modern approach for modeling narratives is as a sequence of neural states @xcite @xcite . however, a drawback of this 1 code is provided at https://github.com/ eitanwagner/holocaust-segmentation . approach is the lack of interpretability, which is crucial in some contexts. a different approach represents and visualizes a narrative as a sequence of interpretable topics @xcite . inspired by this approach, we seek to model the narrative of a text using topic segmentation, dividing long texts into topically coherent segments and labeling them, thus creating a global topical structure in the form of a chain of topics. topic segmentation can be useful for the indexing of a large number of testimonies (tens of thousands of testimonies have been collected thus far) and as an intermediate or auxiliary step in tasks such as summarization @xcite and event detection @xcite . unlike recent supervised segmentation models that focus on structured written text, such as wikipedia sections @xcite or book chapters @xcite , we address the hitherto mostly unaddressed task of segmenting and labeling unstructured (transcribed) spoken language. for these texts, we don't have large datasets of divided text. moreover, there may not be any obvious boundaries that can be derived based on local properties. this makes the task more challenging and hampers the possibility of taking a completely supervised approach. we propose an unsupervised alternative for segmentation, based on two assumptions: (1) segment boundaries correspond to places with low mutual information between sentences over the boundary; our contributions are the following: (1) we present the task of topical segmentation for running, unedited text; (2) we propose novel algorithmic methods for tackling the task without any manual segmentation supervision, building on recent advances in language modeling; (3) comparing to previous work, we find substantial improvements over existing methods; (4) we compile a test set for evaluation in the case of holocaust testimonies; (5) we develop domain-specific topical classifiers to extract lists of topics for long texts. typically, narrative research faces a tradeoff between the number of narrative texts, which is important for computational methods, and the specificity of the narrative context, which is essential for qualitative narrative research @xcite . holocaust testimonies provide a unique case of a large corpus with a specific context. our work also communicates with holocaust research, seeking methods to better access testimonies as the survivor generation is slowly passing away @xcite . we expect our methods to promote schemabased analysis and browsing of testimonies, enabling better access and understanding. ## previous work text segmentation. considerable previous work addressed the task of text segmentation, using both supervised and unsupervised approaches. proposed 2 https://sfi.usc.edu/ methods for unsupervised text segmentation can be divided into linear segmentation algorithms and dynamic graph-based segmentation algorithms. linear segmentation, i.e., segmentation that is performed on the fly, dates back to the texttiling algorithm @xcite , which detects boundaries using window-based vocabulary changes. recently, @xcite proposed an improvement to the algorithm, which, unlike texttiling, uses the vocabulary of the entire dataset and not only of the currently considered segment. topictiling @xcite uses a similar approach, using lda-based topical coherence instead of vocabulary only. this method produces topics as well as segments. another linear model, bats @xcite , uses combined spectral and agglomerative clustering for topics and segments. in contrast to the linear approach, several models follow a bayesian sequence modeling approach, using dynamic programming for inference. this approach allows making a global prediction of the segmentation, at the expense of higher complexity. implementation details vary, and include using pretrained lda models @xcite , online topic estimation @xcite , shared topics @xcite , ordering-based topics @xcite , and context-aware lda @xcite . following recent advances in neural models, these models have been used for the task of supervised text segmentation. @xcite introduced chaptercaptor which relies on two methods. the first method performs chapter break prediction based on next sentence prediction (nsp) scores. the second method uses dynamic programming to regularize the segment lengths towards the average. the models use supervision for finetuning the model for boundary scores, but can also be used in a completely unsupervised fashion. they experiment with segmenting books into chapters, which offers natural incidental supervision. another approach performs the segmentation task in a completely supervised manner, similar to supervised labeled span extraction tasks. at first, the models were lstm-based @xcite , and later on, transformer based @xcite . unlike finetuning, this approach requires a large amount of segmented data. all of these works were designed and evaluated with structured written text, such as book chapters, wikipedia pages, or artificially stitched segments, where supervised data is abundant. in this work, we address the segmentation of texts of which we have little supervised data regarding segment boundaries. we, therefore, adopt elements from the unsupervised approaches combined with supervised components and design a model for a novel segmentation task of unstructured spoken narratives. narrative analysis. much work has been done in the direction of probabilistic schema inference, focusing on either event schemas @xcite @xcite or persona schemas @xcite . recently, neural models were utilized for story modeling. @xcite presented a neural gpt2-based model for suspense in short stories. this work follows an information-based framework, modeling the reader's suspense by different types of predictability. due to their strong performance in text generation, neural models are commonly used for story generation, with numerous structural variations @xcite @xcite . narrative analysis can help in conveying the essence of stories, without all the details. this can aid the meta-analysis of stories. @xcite visualized plot progressions in stories in various ways, including the progression of character relations. @xcite analyzed birth stories, using simplistic, uniform segmentation with topic modeling to visualize the frequent topic paths. ## methods we have a document x consisting of n sentences x 1 • • • x n , which we consider as atomic units. our task is to find k -1 boundary points, defining k segments, and k topics, where every consecutive pair of topics is different. ## results we evaluate our models for both the segmentation and the resulting topic sequence. we do not report scores for the lda-based model since it did not produce a reasonable number of segments, and its runtime was prohibitively long (in previous work, it was run on much shorter text). we also implemented the models with different sizes of gpt2. observing that the size had no significant effect, we report the results with the base model ("gpt2") only. we emphasize that our models are only weakly supervised, as the topic classifier was trained with arbitrary boundaries and the topics were implicitly derived from the data. ## discussion our results show that topic assignment given the real segmentation gold + cl gives better topics than all other models. this suggests that a good segmentation does contribute to the topic assignment, which motivates tackling the segmentation and topic assignment jointly, in principle. the gold + cl model actually achieves higher topic similarity than the inter-annotator agreement. this might be explained by the fact that the gold + cl model was given the exact number of segments, while this was not specified for the annotators. regarding the full models, our results show that the pmi methods show better performance for the segmentation task, compared to previous methods. this supports our hypothesis that segment boundaries correspond to low mutual information between the segments. this connects to common unsupervised methods and trends, showing that general-purpose self-supervised models can perform strongly on various tasks. however, we find that the automatic segmentation results do not contribute to the topic assignment (topic assignment scores are comparable with uniform segmentation). it seems then that although the pmi methods show improvement over previous work for the segmentation task, their results are still not sufficient to contribute to the topic extraction. within the different pmi models, we see that additional length and topic scores do not yield substantial improvements, neither for the segmentation nor for the topics. this is somewhat surprising and might mean that the sensitivity of our classifier to exact boundaries is low, or that the produced segments did not yet cross a usefulness threshold for topic classification. another surprising result is that larger sizes and domain fine-tuning of the gpt2 model do not improve the performance, sometimes actually hurting it. inspecting the produced segments, it seems that these models do produce meaningful segments with good boundaries, but they don't always match the manual boundaries, as the exact segmentation depends also on the given set of topics. that said, the fact that the models still perform better than baseline models shows that it is possible to produce reasonable segmentations even without specifying a set of topics. ## conclusion we presented models for combined segmentation and topic extraction for narratives. we found that: (1) local pmi scores are sufficient to infer a segmentation with better quality than previous models; our work addresses the segmentation and topic labeling of text in a naturalistic domain, involving unstructured, transcribed text. our model can segment noisy texts where textual cues are sparse. in addition to the technical contribution of this work, it also makes important first steps in analyzing spoken testimonies in a systematic, yet ethical manner. with the imminent passing of the last remaining holocaust survivors, it is increasingly important to design methods of browsing and analyzing these testimonies, so as to enable us to use the wealth of materials collected in the archives for studying and remembering the stories.