Words That Stick: Using Keyword Cohesion to Improve Text Segmentation (2024)

Abstract: Text Segmentation (TS) is the idea of segmenting bodies of text into coherent blocks, mostly defined by the topics each segment contains. Historically, techniques in this area have been unsupervised, with more success recently coming from supervised methods instead. Although these approaches see better performance, they require training data and upfront training time. We propose a new method called Coherence, where we use strong sentence embeddings to pull representational keywords as the main constructor of sentences when comparing them to one another. Additionally, we include a storage of previously found keywords for the purposes of creating a more accurate segment representation instead of just the immediate sentence in question. With our system, we show improved results over current state-of-the-art unsupervised techniques when analyzed using Pk and WindowDiff scores. Because its unsupervised, Coherence requires no fine-tuning.

## introduction we present coherence, a method that utilizes related words and their contextual meanings within sentences for effective text segmentation (ts). in the past decade, advancements in the field of ts have been primarily dominated by supervised techniques @xcite , @xcite , @xcite , @xcite , @xcite , and @xcite ), which require training data and are computed on a sentence-wise basis (i.e., each sentence is compared with adjacent sentences for evaluation). in contrast, coherence uses contextual keyword embeddings for comparison, reducing potential noise and unnecessary sentence-level information that may not be helpful to the ts task. coherence uses a sliding window technique, traditionally used in supervised ts, to predict segment breaks (e.g., p (s n-1 , s n , s n+1 ) = 1). however, coherence enhances this method by incorporating contextual information (through contextual keyword embeddings). coherence demonstrates performance improvements, particularly in p k scores, and does not require fine-tuning. by leveraging pre-trained sentence encoders like bert, labse, and s-bert, coherence leverages extracted keywords to form an end-to-end flow. the core of coherence lies in collecting and utilizing important keywords during the segmentation process. these keywords are represented as contextual embeddings, capturing essential information about their usage within sentences (for example, differentiating "bridge" in the context of crossing a river from "bridge" in the context of a human's nose). this process is inspired by the multi-headed attention mechanism in the transformer architecture, providing a nuanced understanding of sentence relationships without the need for extensive training and data. we show that without the need for expensive fine-tuning and highly-dimensional sentence embeddings as training data, we can achieve performance improvements in a space that has been more recently dominated by advancements in supervised learning. using orthogonal keywords in addition to similar keywords provides more breadth in keyword representation to further bolster results. all our code can be found on the human-machine lab github repository foot_0 . ## related works initially, hearst (1997) introduced texttiling, an unsupervised algorithm that identifies segment boundaries through lexical overlaps. similarly, @xcite demonstrated the efficacy of unsupervised methods by analyzing sentence similarities, categorizing their work within linear ts methodologies. these initial contributions set a new standard in the field. the landscape of ts shifted with the advent of advanced word and sentence embeddings, paving the way for supervised techniques. @xcite explored the potential of processing large ts datasets through a bi-lstm, analyzing three sentences at a time to understand their interrelations. building on this, @xcite proposed a sentence-wise model utilizing attention mechanisms to enhance performance further. recent supervised approaches have increasingly incorporated lstms and transformers as foundational components, as seen in works by @xcite , @xcite , @xcite , and @xcite . these studies have showcased the effectiveness of adding topic information and emphasizing sentence contextuality in achieving top-tier results. despite the dominance of supervised models, unsupervised ts techniques continue to show promise. @xcite revisited the classic texttiling approach, refining it with lda to identify more precise keywords. riedl and biemann (2012) combined lda and texttiling for another innovative unsupervised solution. furthermore, @xcite introduced a novel unsupervised graph-based method, analyzing sentences as nodes within a graph to predict segment boundaries. these unsupervised models underscore the ongoing exploration and diversity in ts methodologies. while unsupervised approaches in the field continue to be important due to their flexibility and lack of need for domain-specific training data, more research has recently focused on supervised approaches. @xcite 's approach to ts relied upon within-segment word similarity and prior information about segment length, but does not incorporate inter-sentence comparisons. in contrast, @xcite approaches unsupervised ts by using probabilistic latent semantic analysis (plsa) to identify similar words an in intersentence level. they then apply a texttiling based approach for identifying changes in frequency between sentences. another technique by @xcite takes a unique approach to unsupervised ts by grouping a series of sentences together, stacking them on top of each other, and performing max pooling. the resulting matrix is a mixture of sentences, which can then be used to compare to other matrices. they perform their analysis on meeting data, which shows improvements upon other techniques. more recently, john et al. @xcite utilize an lda-based texttiling approach that produces strong results. the boundary adjustment technique proposed in this work is a retroactive solution to topictiling (riedl and biemann, 2012) that helps improve results. ## methodology the core of coherence is its ability to pull out keywords from provided sentences. to accomplish this, we use a library called keybert foot_1 . this library goes through each word in a sentence, creates an embedding for the word and compares it with the embedding of the sentence at hand. keywords are identified as the ones with a higher similarity to the sentence embedding. because of this, there is no need to globally scan the document beforehand, which other techniques like tf-idf and lda require. utilizing bert allows the keybert library to effectively look into the attention being paid at every word and phrase to identify important words. keybert has been shown to outperform other topic modelling and keyword extraction techniques like lda and yake @xcite . we also consider the use of an lda based ap-overcast, precipitation, cloudy astigarraga, ametzagaña, paleolithic oiasso, varduli, irun san sebastián features an oceanic climate (köppen cfb) with warm summers and cool winters. like many cities with this climate, san sebastián typically experiences cloudy or overcast conditions for the majority of the year, typically with some precipitation. the first evidence of human stationary presence in the current city is the settlement of ametzagaña, between south intxaurrondo and astigarraga. the unearthed remains, such as carved stone used as knives to cut animal skin, date from 24,000 to 22,000 bc. san sebastián is thought to have been in the territory of the varduli in roman times. figure 1: topics gathered throughout the keyword extraction phase within the "wiki" dataset. due to the natural pruning of the keyword map, only the most pertinent topics are retained. additionally, importance of the keyword to its original sentence is also maintained. results shown here are extracted from the "wiki" dataset starting at sample 643. coherence is broken down into two major phases. we use sentences to denote the important keywords derived from said sentences -sentences are not compared verbatim, rather the keywords that make up the sentence are compared. at every step, the current sentence is compared against prior sentences, as long as they exist in the keyword map (we elaborate on conditions where a sentence's keywords would not end up in the keyword map later in this section). ## metrics two popular metrics that exist solely to benchmark ts systems are p k and windowdiff (wd), which have become commonplace for work in the ts field. p k is the probability that a pair of chosen sentences with a distance of k are incorrectly classified. both the wd and p k metrics use a sliding window of fixed size w over the document and compare the predicted segments with the reference ones. k is determined as half of the average true segment size of the document. since p k and wd are both penalty metrics, lower values indicate better performance. while p k is the most widely and still is the most accepted metric in the ts space, wd was originally proposed as an update to the p k metric. p k can be thought of as the probability that two segments drawn from a document are incorrectly identified as belonging to the same segment. wd operates almost identically, but uses a sliding window to penalize systems that tend to overpredict, resulting in false positives -something that p k does not acknowledge as an errant prediction. both p k and wd thus lie between zero and one and an algorithm that assigns all boundaries correctly receives a score of zero. wd is considered a better measure than p k as the p k metric suffers from issues such as a lack of false positive prediction penalization @xcite . ## data unsupervised ts methods are often evaluated using constructed datasets, which amalgamate segments from varied sources into composite documents, as evidenced by studies from choi @xcite and galley et al @xcite . ## results coherence shows and improvement over sota unsupervised results in the space. results are reported on the choi, manifesto, clinical, fiction, and wiki datasets @xcite , @xcite , @xcite ). our performance on these datasets shows the versatility of coherence. this gives us hope that with the use of pre-trained models, unsupervised approaches can prove to be viable in the ts space. results on the choi and manifesto datasets are reported against pre-existing sota unsupervised ts approaches. results for the clinical, wiki, and fiction datasets are compared against badjatiya et al. (2018)'s work. results on the clinical and fiction datasets are competitive with @xcite 's preexisting supervised approach. coherence does not do as well on the wiki dataset however. we believe this is due to the subjectivity in ts datasets at the labelling level. the wiki dataset has an average segment length of 26 sentences for example. on choi's dataset, coherence performs extremely well, outperforming all previous sota unsupervised ts techniques. coherence also performs competitively, with stronger results in the wd metric on the manifesto dataset. this performance improvement on wd versus p k indicates that coherence makes less false positive predictions than pre-existing techniques. we show that, in comparison to previous sota unsupervised techniques, coherence outperforms in a variety of datasets using both the p k and wd metrics as benchmarks. this comes without the need for fine-tuning or domain adaptation. since the keyword extraction phase of coherence is modular, we believe that as sentence and word embedding technology continues to improve, so will the results of coherence. the lack of need for fine-tuning a model is advantageous and as of such, each round of inference takes roughly 25ms -125ms on a cloud-based a100 gpu. additionally, coherence provides utility without the need for training or domain adaptation. the lightweight lift of coherence allows it to be used against various datasets, due to the strength of the sentence encoder. the applicability of coherence to new and unseen test datasets can prove to be useful in production settings. ## limitations coherence shows improvements over pre-existing sota unsupervised systems such as @xcite and graph-seg @xcite . the authors for the works found in some reliance for coherence comes from the pre-trained sentence encoder (keybert) in the keyword extraction phase. although this seems like a limitation, it can be a strength in the flexibility of the system. future iterations of pre-trained sentence encoders can be used to replace keybert and enhance coherence's output. we show the flexibility of our system by achieving superior results on a wide array of available datasets without the need for tedious fine-tuning. this implies that as keyword extraction techniques become stronger, so shall our system. most of the processing time comes from the keyword extraction phase, due to the keyword extraction library. roughly 90% of this time comes from the keyword extraction phase, whereby keybert needs to compare every keyword with its parent sentence embedding. the majority of the ram utilization also comes from this phase, as the embedding model (labse in our case) is loaded into memory for inference. in our experiments, coherence required less than 3gb ram throughout testing. with techniques like quantization, smaller models can perform this keyword extraction step more efficiently. this limitation is due to the selected keyword extraction library; keybert in our case. the majority of processing time in the keybert library comes from creating contextual embeddings for each sentence before comparing each word in the sentence it was pulled from with the sentence itself. ## conclusion in this work, we present coherence, which is a novel approach to unsupervised ts that leverages contextual keywords from sentences to represent text segments. we show that the emphasis on contextual keywords can build representations of clinical wiki fiction p k ↓ wd ↓ p k ↓ wd ↓ p k ↓ wd ↓ badjatiya et al. (2018) 33.0 31.0 34.0 32.0 38.0 31.0 coherence 37.1 38.9 50.2 53.4 35.7 61.6 our results on the choi, manifesto, clinical, wiki, and fiction datasets show that coherence can perform well in a variety of domains. while we also include our results on the newer wikisection dataset, other supervised ts approaches show superior results. future work will focus on enhancing coherence to consider the contextual relationship between extracted keywords, while exploring the method's applicability across various domains and datasets. coherence offers a solution that can be adapted to various domains without the need for fine-tuning.