A Reference-free Segmentation Quality Index ( S eg R e F ree) (2023)

Abstract: Topic segmentation, in the context of natural language processing, is the process of finding boundaries in a sequence of sentences that separate groups of adjacent sentences at shifts in semantic meaning. Currently, assessing the quality of a segmentation is done by comparing segmentation boundaries selected by a human or algorithm to those selected by a known good reference. This means that it is not possible to quantify the quality of a segmentation without a human annotator, which can be costly and time consuming. This work seeks to improve assessment of segmentation by proposing a reference-free segmentation quality index (SegReFree). The metric takes advantage of the fact that segmentation at a sentence level generally seeks to identify segment boundaries at semantic boundaries within the text. The proposed metric uses a modified cluster validity metric with semantic embeddings of the sentences to determine the quality of the segmentation. Multiple segmentation data sets are used to compare our proposed metric with existing reference-based segmentation metrics by progressively degrading the reference segmentation while computing all possible metrics; through this process, a strong correlation with existing segmentation metrics is shown. A Python library implementing the metric is released under the GNU General Public License and the repository is available at https://github.com/evan-person/reference_free_segmentation_metric .

## related work existing segmentation evaluation metrics, proposed by @xcite ; @xcite ; @xcite ; @xcite are reviewed, along with a brief introduction to cluster validity metrics. we include a review of the davies-bouldin index @xcite , which was used as a starting point for this work. a brief review of some unsupervised segmentation methods that use sentence-bert (reimers and gurevych, 2019) is also presented. ## method conceptually, our proposed metric seeks to quantify segmentation quality by assuming that the desired segmentation for a given text occurs at topical boundaries. if the segmentation boundary separates two different semantically different groups of text, the semantic vector representations of the sentences in those segments should generally be close to the other sentences within the segment and separated from the sentences in adjacent text segments. at a high level, the proposed metric works by splitting text into sentences or utterances (the term split text into chunks at segmentation boundary generate sentence embeddings use segmentation algorithm to assign class ids to each embedding compute modified davies-bouldin score chunks is used as a synonym for both), generating embeddings of the chunks being segmented, and using a modified cluster validity metric where we treat segments as clusters. a block diagram is shown in figure 1 . explanations of each step are elaborated upon in the rest of this section. ## experiments a variety of tests were devised to evaluate the performance of our metric. tests were designed to mimic segmentation errors that could be encountered in realistic scenarios. ## conclusions in this work, a new topical text segmentation evaluation metric that requires no reference segmentation set is proposed and evaluated. to our knowledge, this is the first reference-free segmentation quality metric that has been proposed. strong correlation to existing reference-based metrics was demonstrated for the worst case errors of missing boundaries and boundary insertion in the middle of a coherent segment. a weaker, yet present correlation is demonstrated for the harder case of boundary transposition (so-called "near misses"). the use of our reference-free metric can help enable future segmentation efforts on data that do not have existing human annotation.