Lessons Learnt from Linear Text Segmentation: a Fair Comparison of Architectural and Sentence Encoding Strategies for Successful Segmentation (2023)

Abstract: Recent works on linear text segmentation have shown new state-of-the-art results nearly every year. Most times, however, these recent advances include a variety of different elements which makes it difficult to evaluate which individual components of the proposed methods bring about improvements for the task and, more generally, what actually works for linear text segmentation. Moreover, evaluating text segmentation is notoriously difficult and the use of a metric such as Pk, which is widely used in existing literature, presents specific problems that complicates a fair comparison between segmentation models. In this work, then, we draw from a number of existing works to assess which is the state-of-the-art in linear text segmentation, investigating what architectures and features work best for the task. For doing so, we present three models representative of a variety of approaches, we compare them to existing methods and we inspect elements composing them, so as to give a more complete picture of which technique is more successful and why that might be the case. At the same time, we highlight a specific feature of Pk which can bias the results and we report our results using different settings, so as to give future literature a more comprehensive set of baseline results for future developments. We then hope that this work can serve as a solid foundation to foster research in the area, overcoming task-specific difficulties such as evaluation setting and providing new state-of-the-art results.

## introduction linear text segmentation, also known as topic segmentation, is a well known problem in natural language processing, and the first step for a number of downstream applications. the task consists in the automatic segmentation of a text into topically coherent units and this has many use cases: a long transcript from a news show, e.g., could be divided into single news stories so as to help an end user in retrieving more relevant and specific information @xcite or a long article could be divided into subsections to aid its reading @xcite . recent works have presented a series of advancements in the field, from which a number of conclusions could be drawn, such as the fact that transformer architectures work better than traditional recurrent models @xcite and that fine-tuned llms need no additional contextual information to perform the task @xcite . the results of different recent works, however, can be contradictory and not pointing towards a clear direction forward in terms of what works and what does not in text segmentation. part of the reason for this, we show, is the fact that existing and popular metrics such as p k @xcite might lead to very different results under different conditions and, therefore, the final results from which to draw our conclusions are unstable. based on this, we draw on existing literature to present our own topic segmentation models. we show that carefully designed recurrent neural networks are still relevant in the field as they can obtain state-of-the-art results in most occasions given a fixed and fair evaluation setting. we draw conclusions on why this might be the case and we show that this evidence makes sense given previous literature on the subject. ## conclusion in this work, we have given a systematic, fair comparison of three state-of-the-art models for linear text segmentation with two fine-tuned sentence encoders as feature extractors for the task, so as to highlight what techniques proposed by recent literature work in a fair setting. consistent with existing literature, we have shown that the popular p k metric is not very stable. specifically, the influence of different k used in the metric is noticeable; with the result that if models are compared under different evaluation settings, the conclusions that could be drawn are very different and potentially misleading. by keeping the evaluation setting fixed, however, we show that bilstm-based models actually outperform transformers, at least on the current datasets, and that fine-tuning the sentence encoders does bring improvements but not necessarily as big as previously suggested. restricting the context available to transformer models leads to performance gains, as previously noticed by @xcite and @xcite ; but bi-lstm-based systems always outperform even the best performing transformer models, perhaps suggesting that some architectural element of lstms makes them more apt for the task at hand. this is indeed interesting evidence, which we aim to develop further in future work.