Segment Any Text: A Universal Approach for Robust, Efficient and Adaptable Sentence Segmentation (2024)

Abstract: Segmenting text into sentences plays an early and crucial role in many NLP systems. This is commonly achieved by using rule-based or statistical methods relying on lexical features such as punctuation. Although some recent works no longer exclusively rely on punctuation, we find that no prior method achieves all of (i) robustness to missing punctuation, (ii) effective adaptability to new domains, and (iii) high efficiency. We introduce a new model — Segment any Text (SaT) — to solve this problem. To enhance robustness, we propose a new pretraining scheme that ensures less reliance on punctuation. To address adaptability, we introduce an extra stage of parameter-efficient fine-tuning, establishing state-of-the-art performance in distinct domains such as verses from lyrics and legal documents. Along the way, we introduce architectural modifications that result in a threefold gain in speed over the previous state of the art and solve spurious reliance on context far in the future. Finally, we introduce a variant of our model with fine-tuning on a diverse, multilingual mixture of sentence-segmented data, acting as a drop-in replacement and enhancement for existing segmentation tools. Overall, our contributions provide a universal approach for segmenting any text. Our method outperforms all baselines — including strong LLMs — across 8 corpora spanning diverse domains and languages, especially in practically relevant situations where text is poorly formatted. Our models and code, including documentation, are readily available at https://github.com/segment-any-text/wtpsplit under the MIT license.

## introduction sentence segmentation is defined as the task of identifying boundaries between sentences in a given text. high-quality sentence boundaries are crucial in many nlp tasks and systems since models often expect individual sentences as input @xcite we average over all 23 languages and show the average time (10 runs) for variants with different sizes (@xmath0,000 sentences using consumer hardware (1 nvidia gtx 2080 ti gpu). existing sentence segmentation tools predominantly rely on punctuation marks. this limitation renders them impractical for text lacking punctuation. to address this issue, some recent methods aim to overcome this dependency @xcite . specifically, during the training of their model, wtp @xcite randomly removes punctuation characters to increase robustness against missing punctuation. however, the performance of wtp as the current state-of-the-art (sota) model and all other segmenters is still poor on texts from more challenging domains. this includes, among others, user-generated text such as tweets and highly heterogeneous domains such as lyrics. segmenting these texts is challenging because of missing and/or extra punctuation, inconsistent spacing, and espe-sat also gut. = all right. alles klar? = all right? na also! = all right! they are the same don't believe me let's watch it again we just wanna see yaa everyday i'm shuffelin shuffelin shuffelin "also gut. = all right.", "alles klar? = all right?", "na also! = all right!" "they are the same", "don't believe me", "let's watch it again" "we just wanna see yaa", "everyday i'm shuffelin", "shuffelin shuffelin" figure 2 : examples of our model's predictions from (i) asr output, (ii) multilingual text, and (iii) verse segmentation. (i) shows part of a transcribed ted talk, demonstrating our method is agnostic to punctuation and casing. (ii) is from a reddit post of german-english translations; existing rule-based systems would segment at nearly every punctuation, and existing neural systems are too reliant on punctuation or need a language code. (iii) shows segmentation of lyrics into verses, showing our model's predictions in a distinct domain. to address these challenges, we present a sentence segmentation method that does not rely on language codes or punctuation marks, making it universally applicable across a broad range of languages, corpora, and domains. specifically, we train subword-based multilingual encoder language models (lms) in a self-supervised way to predict naturally occurring newlines on web-scale text. we then continue training models on sentencesegmented data in a second, supervised stage to further improve sentence segmentation performance. we deal with several major issues with previous tools: to ensure robustness against missing punctuation and noise, we propose a set of corruptions, applied randomly to the input during training. crucially, our method does not rely on language codes. in addition, we mitigate issues observed with short sequences via a novel limited lookahead mechanism. furthermore, we recognize the variability of sentence boundaries across domains and sentence definitions. to address this, we show how our models can be efficiently adapted to target domains via lora @xcite , outperforming previous adaptation methods, especially in data-constrained settings. further, we improve efficiency by shedding the upper layers of the base model for our default 3-layer models, which segments 1000 sentences in approx. 0.5 seconds on our hardware. figure 1 shows that the standard 3-layer version of sat outperforms the current open weights stateof-the-art, wtp, while achieving a ≈ 3x reduction in inference time. overall, we present several innovations that overcome each of the shortcomings of previous methods, culminating in a universal model for sentence segmentation. we provide some examples of our model's predictions in figure 2 . ## background and related work we start by providing an overview of existing sentence segmentation systems. following @xcite , we categorize them into 1) rule-based, 2) supervised statistical, and 3) unsupervised statistical approaches. then, we discuss the recently introduced state-of-the-art approach, wtp. moreover, we discuss domain-specific segmentation approaches. lastly, since we are the first to evaluate large language models (llms) for sentence segmentation broadly, we briefly survey them and discuss their usage in sentence segmentation tasks. ## sat: segment any text to create a reliable and effective system across various scenarios, we pre-train a model on paragraph segmentation as in @xcite . in the following, we outline how we solve each of the major issues of wtp discussed earlier, leading to a universal model for sentence segmentation. efficiency. we resort to models using subword tokenization, processing tokens consisting of multiple characters at a time, making them considerably faster than their character-level counterparts. unlike minixhofer et al. (2023) , we do not rely on language adapters. in addition to improving inference time and storage requirements, this also improves multilinguality since no language has to be specified at inference time. ## discussion llms. contrary to our expectations, our evaluation results reveal that llms generally underperform, particularly in non-english languages. notably, when using llms for sentence segmentation via prompting, each sentence is processed twiceonce as part of the input, appended to the prompt, and once within the output. this redundancy leads to inefficient processing, needing to copy the input verbatim to the output, ideally only adding newlines. however, in practice, llms are highly prone to alter their input @xcite . we found this issue to be particularly severe for noisy text and lyrics. 8 this is highly problematic for a specific task requiring input and output characters to remain the same. still, we tried to address this by using the needleman-wunsch sequence alignment algorithm to make pure segmentation performance comparable to other methods. aiming to improve the segmentation performance of llms, we experimented with few-shot prompting. however, this did not yield the desired improvements; in fact, it degraded performance. additionally, we tested varying the number of input-output sentences. the results of both of these ablation studies are presented in appendix a.1. efficiency. for our method, we rely on xlm-r as the lm backbone. operating on subwords makes sat considerably faster than wtp. we compare sentence segmentation performance and time on ersatz across model sizes from 1-12 layers, illustrated in figure 1 . additional datasets are shown in figure 4 . the standard 3-layer variations of sat take ≈ 0.5 seconds to segment 1000 sentences on the hardware specified in appendix a.2, making them 3 times faster than wtp, while also outperforming wtp models in all sizes on ersatz. furthermore, for sat, performance plateaus with @xmath1, whereas sat +sm continues to improve when scaling up its size, making it by far the best model, despite never being exposed to ersatz. ## conclusion we proposed sat, an efficient, robust, and highly adaptable multilingual sentence segmentation method that neither relies on language codes nor punctuation. further, we introduced sat +sm , improving sat via supervised adaptation using multiple corruption schemes. our method consistently achieves state-of-the-art performance among open weights models in experiments across 85 languages and eight diverse corpora, even outperforming newly introduced and optimized strong llm baselines. we also demonstrated that sat can be efficiently domain-adapted via lora, setting new performance standards on segmentation of lyrics and code-switching text. overall, we hope sat will unlock significantly improved text data (pre-)processing across a range of nlp applications for multiple languages and domains via its robust and consistently strong performance, versatility, and high efficiency.