Where’s the Point? Self-Supervised Multilingual Punctuation-Agnostic Sentence Segmentation (2023)

Abstract: Many NLP pipelines split text into sentences as one of the crucial preprocessing steps. Prior sentence segmentation tools either rely on punctuation or require a considerable amount of sentence-segmented training data: both central assumptions might fail when porting sentence segmenters to diverse languages on a massive scale. In this work, we thus introduce a multilingual punctuation-agnostic sentence segmentation method, currently covering 85 languages, trained in a self-supervised fashion on unsegmented text, by making use of newline characters which implicitly perform segmentation into paragraphs. We further propose an approach that adapts our method to the segmentation in a given corpus by using only a small number (64-256) of sentence-segmented examples. The main results indicate that our method outperforms all the prior best sentence-segmentation tools by an average of 6.1% F1 points. Furthermore, we demonstrate that proper sentence segmentation has a point: the use of a (powerful) sentence segmenter makes a considerable difference for a downstream application such as machine translation (MT). By using our method to match sentence segmentation to the segmentation used during training of MT models, we achieve an average improvement of 2.3 BLEU points over the best prior segmentation tool, as well as massive gains over a trivial segmenter that splits text into equally-sized blocks.

## introduction sentences are ubiquitous in nlp. many datasets are made up of annotated sentences @xcite @xcite and models often expect individual sentences as input @xcite @xcite . this mandates a need for tools to segment text into sentences: a requirement that * work done during the time bm interned at cohere. † equal senior authorship. collection sentences ud this is the high season for tourism; | between december and april few people visit and many tour companies and restaurants close down. opus100 'i couldn't help it,' said five, in a sulky tone; 'seven jogged my elbow.' | on which seven looked up and said, 'that's right, five! always lay the blame (...)!' ersatz "a lot of people would like to go back to 1970," before program trading, he said. | "i would like to go back to 1970. | but we're not going back (...)" ## method in light of the ambiguity in what makes up a sentence, we resort to the following definition: this pragmatically driven definition corresponds closely to our intuitive understanding of a sentence due to two statements we assume to be true about sufficiently clean text: (i) a newline can generally not occur within a sentence and (ii) a newline can generally occur after any sentence. this definition turns sentence segmentation into a character-level language modeling task. however, a causal language modeling objective as used in contemporary generative lms @xcite @xcite would restrict the model to unidirectional context. we devise a corruption method to allow using a bidirectional lm to model newline-likelihood. let c denote the sequence of characters making up some corpus. c is preprocessed by stripping consecutive newline characters, and adding a space after every newline in languages which use whitespace to separate sentences. first, we corrupt the text by removing newline characters (\n) from c, resulting in x: the target is to identify which characters were followed by a newline in the original sequence: foot_0 the output ŷ can be interpreted as an estimate for the probability of a newline to occur after any character. this objective is comparable to objectives used in text-editing models @xcite . 2 it remains to find a suitable threshold α such that characters with ŷi ≥ α are considered a sentence boundary. in the simplest case, α can be set to a small constant value such as 0.01, where a higher α gives rise to more conservative segmentation. we denote the model variants with constant threshold as wtp u . wtp u models can segment text according to the general definition d1. in practice, models are trained on different corpora following different definitions of what makes up a sentence. this can be addressed to some extent by selecting the threshold α to maximise performance on the target corpus (wtp t ). to allow for more sophisticated adaptation, we introduce an auxiliary objective. ## results and discussion main results are shown in ## conclusion we have introduced wtp, a method for multilingual sentence segmentation without relying on punctuation or sentence-segmented training data. we have demonstrated strong performance of our selfsupervised sentence segmentation method across 85 languages and 3 different corpora, matching performance of the best prior (supervised) tools. we have further improved wtp in a data-efficient way by means of inexpensive supervised adaptation, which leads to state-of-the-art scores outperforming the best prior tools by an average 6.1% f1 points. we have also found that, though often an overlooked component of nlp systems, sentence segmentation can have a strong impact on downstream tasks by showing that matching the segmentation at inference to the segmentation used during training benefits sentence-level mt models.