Improving Long Document Topic Segmentation Models With Enhanced Coherence Modeling (2023)

Abstract: Topic segmentation is critical for obtaining structured documents and improving down- stream tasks such as information retrieval. Due to its ability of automatically exploring clues of topic shift from abundant labeled data, recent supervised neural models have greatly promoted the development of long document topic segmentation, but leaving the deeper relationship between coherence and topic segmentation underexplored. Therefore, this paper enhances the ability of supervised models to capture coherence from both logical structure and semantic similarity perspectives to further improve the topic segmentation performance, proposing Topic-aware Sentence Structure Prediction (TSSP) and Contrastive Semantic Similarity Learning (CSSL). Specifically, the TSSP task is proposed to force the model to comprehend structural information by learning the original relations between adjacent sentences in a disarrayed document, which is constructed by jointly disrupting the original document at topic and sentence levels. Moreover, we utilize inter- and intra-topic information to construct contrastive samples and design the CSSL objective to ensure that the sentences representations in the same topic have higher similarity, while those in different topics are less similar. Extensive experiments show that the Longformer with our approach significantly outperforms old state-of-the-art (SOTA) methods. Our approach improve F_{1} of old SOTA by 3.42 (73.74 \rightarrow 77.16) and reduces P_{k} by 1.11 points (15.0 \rightarrow 13.89) on WIKI-727K and achieves an average relative reduction of 4.3% on P_{k} on WikiSection. The average relative P_{k} drop of 8.38% on two out-of-domain datasets also demonstrates the robustness of our approach.

## introduction topic segmentation aims to automatically segment the text into non-overlapping topically coherent 1 our code is publicly available at https://github.com/ alibaba-damo-academy/spokennlp/ parts @xcite . topic segmentation makes documents easier to read and understand, and also plays a key role in many downstream tasks such as information extraction @xcite and document summarization @xcite . topic segmentation methods can be categorized into linear segmentation @xcite , which yields a linear sequence of topic segments, and hierarchical segmentation @xcite , which produces a hierarchical structure with top-level segments divided into subsegments. we focus on linear topic segmentation in this work, especially for long documents. based on the definition of topics, each sentence in a topic relates to the central idea of the topic, and topics should be discriminative. hence, two adjacent sentences from the same topic are more similar than those from different topics. exploring this idea, prior unsupervised models mainly infer topic boundaries through computing text similarity @xcite or exploring topic representation of text @xcite . different from the shallow features carefully designed and used by unsupervised methods, supervised neural models can model deeper semantic information and explore clues of topic shift from labeled data @xcite . supervised models have achieved large gains on topic segmentation through pre-training language models (plms) (e.g., bert) and fine-tuning on large-scale supervised datasets @xcite @xcite . recently, @xcite @xcite improve topic segmentation performance by explicitly modeling text coherence. however, these approaches either neglect context modeling beyond adjacent sentences @xcite , or require additional label information @xcite @xcite , or impede learning sentence-pair coherence without considering both coherent and incoherent pairs @xcite . moreover, compared to short documents, topic segmentation becomes more critical for understanding long documents, and coherence modeling for long document topic segmentation is more crucial. coherence plays a key role in understanding both logical structures and text semantics. consequently, to enhance coherence modeling in supervised topic segmentation methods, we propose two auxiliary coherence-related tasks, namely, topicaware sentence structure prediction (tssp) and contrastive semantic similarity learning (cssl). we create disordered incoherent documents, then the tssp task utilizes these documents and enhances learning sentence-pair structure information. the cssl task regulates sentence representations and ensures sentences in the same topic have higher semantic similarity while sentences in different topics are less similar. experimental results demonstrate that both tssp and cssl improve topic segmentation performance and their combination achieves further gains. moreover, performance gains on out-of-domain data from the proposed approaches demonstrate that they also significantly improve generalizability of the model. large language models such as chatgpt foot_0 have achieved impressive performance on a wide variety of nlp tasks. we adopt the prompts proposed by @xcite and evaluate chatgpt on the wiki-50 dataset @xcite . we find chatgpt performs considerably worse than fine-tuning bert-sized plms on long document topic segmentation (as shown in appendix a). our contributions can be summarized as follows. ## methodology in this section, we first describe our baseline model for topic segmentation (section 3.1), then introduce our proposed topic-aware sentence structure prediction (tssp) module (section 3.2) and contrastive semantic similarity learning (cssl) module (section 3.3). figure 1 illustrates the overall architecture of our topic segmentation model. ## analysis effect of context size to study the effect of the context size, we evaluate longformer with max sequence length of 512, 1024, 2048 and 4096 on wikisection. we evaluate the effectiveness of our proposed methods using the corresponding max sequence length to investigate whether they remain effective with different context sizes. as can be seen from figure 3 , the topic segmentation f 1 gradually improves as the context length increases. among them, the effect of increasing the input length from 512 to 1024 is the largest with f 1 on en_city and en_disease improved by +2.54 and compared to sso and cats, tssp helps the model learn better intersentence relations and both intra-and inter-topic baseline cats sso tssp w/o inter-topic tssp w/o intra-topic tssp 80.0 80.2 80.4 80.6 80.8 81.0 81.2 81.4 81.6 81.8 f1 80.43 80.49 81.08 81.21 81.29 81.48 (a) tssp 0 1 2 3 4 the number of negative pairs 80.43 80.50 80.60 80.70 80.80 80.90 f1 80.53 80.71 80.88 80.93 80.78 baseline cssl (b) cssl structure labels are needed to improve performance. figure 4(b) illustrates the impact of varying numbers of negative sample pairs for cssl on longformer. we find that adding similarity-related auxiliary tasks improves the performance. compared with csc, cssl focuses on sentences of the same and different topics when learning sentence representations. as the number of negative samples increases, the model performance improves and optimizes at k 2 =3. gain from tssp is slightly larger than that from cssl, indicating that comprehending structural information contributes more to coherence modeling. we speculate that encoding the entire topic segment into a semantic space to learn contrastive representation may help detecting topic boundaries, which we plan to explore in future work. ## conclusion comprehending text coherence is crucial for topic segmentation, especially on long documents. we propose the topic-aware sentence structure prediction and contrastive semantic similarity learning auxiliary tasks to enhance coherence modeling. experimental results show that longformer trained with our methods significantly outperforms sotas on two english long document benchmarks. our methods also significantly improve generalizability of the model. future work includes extending our approach to spoken document topic segmentation and other segmentation tasks at various levels of granularity and with modalities beyond text.