A unified approach to sentence segmentation of punctuated text in many languages (2021)

Abstract: The sentence is a fundamental unit of text processing. Yet sentences in the wild are commonly encountered not in isolation, but unsegmented within larger paragraphs and documents. Therefore, the first step in many NLP pipelines is sentence segmentation . Despite its importance, this step is the subject of relatively little research. There are no standard test sets or even methods for evaluation, leaving researchers and engineers without a clear footing for evaluating and selecting models for the task. Existing tools have relatively small language coverage, and efforts to extend them to other languages are often ad hoc. We introduce a modern context-based modeling approach that provides a solution to the problem of segmenting punctuated text in many languages, and show how it can be trained on noisily-annotated data. We also establish a new 23-language multilingual evaluation set. Our approach exceeds high baselines set by existing methods on prior English corpora (WSJ and Brown corpora), and also performs well on average on our new evaluation set. We release our tool, ersatz, as open source.

## introduction in many ways, the sentence is the fundamental unit of text in natural language processing (nlp). from the user perspective, tasks such as sentiment analysis, pos tagging, or machine translation consume sentences and emit classifications, annotations, or transductions of those inputs. even tasks that operate at the paragraph or document level, such as coreference resolution or summarization, often make use of sentences internally. yet at the same time, sentences in the wild rarely exist with marked sentence boundaries. for many languages, punctuation serves as a cue for these examples of ambiguity in punctuated contexts secondly, there is not a well-understood methodology for training segmenters that do not make narrow assumptions about the features or characteristics of the languages they support. at the heart of this is the lack of labeled training data. manually-split datasets that accompany annotation projects tend to be small, and larger datasets are typically (imperfectly) segmented by the very tools whose performance is under question. tools such as nltk @xcite , which packages punkt @xcite , provide an unsupervised method to train a model, but it is unclear what the effect is when switching to non-latin-script languages, or how a more supervised approach would handle such noisy data. finally, and perhaps most importantly, there are no standard test sets or even metrics for evaluating segmenter performance, leaving researchers and engineers with no objective way to determine which one is best. the work described in this paper is aimed at these problems. we propose a simple window-based model and semi-supervised training paradigm for the segmentation of punctuated text ( §3). we frame the task as binary classification applied to a set of candidate punctuation locations defined by a regular expression. leveraging the similarity of the task across languages ( ## background a sentence is a sequence of grammatically linked words that conveys a complete thought. the term can be difficult to define in a precise manner that will not admit any exceptions, and in applications like machine translation, there are many times where the basic input unit is not a sentence, but a sentence fragment, such as a headline or an item from a list. in this work, we skirt these complexities, choosing instead to focus on the most common scenario, in which we are dealing with standard written language. for this, we adopt a functional definition: a sentence is a group of words that ends with a sentence-ending punctuation mark, such as (for many languages) a period, question mark, or exclamation point. since punctuation is often used for non-sentence-ending purposes as well, the primary challenge for sentence segmentation is resolving this ambiguity for each segmentation candidate. research in sentence segmentation foot_1 has been limited in scope. prior work either introduces methods that work under a set of assumptions unique to latin-script languages (the existence and importance of casing, word length, or whitespace), or tackles new languages ad hoc, making adaptation to new languages and domains difficult. statistical methods use text-based features such as casing, punctuation, or length of surrounding words to make decisions around punctuation. the earliest work we found @xcite ) considered all sentence boundaries and used decision trees based on these features. @xcite trained two statistical models in the form of an svm and naive bayes classifier. @xcite introduced satz and shifted the approach by only focusing potential sentence boundaries being near sentence-ending punctuation, using part-of-speech distribution vectors as input to a feed-forward neural network and additionally applied their technique to german and french. in order to work without labeled data, @xcite used heuristics to craft scores based on likelihood values of occurrences of tokens, punctuation, casing and token length, and then manually tune a threshold of score to indicate a sentence boundary. this work expanded the most multilingually, considering 10 indo-european languages as well as estonian and turkish. other work has focused on specific non-english languages. @xcite study chinese and dissect the theoretical reasons behind segmenting chinese sentences to match their english equivalents. to segment thai, which lacks punctuation, @xcite use pos-taggers. some work has tackled the problem of domains. @xcite approaches the problem of legal text, which has a set structure without punctuation; other approaches @xcite have investigated speech, which lacks both punctuation and written textual structure. a popular splitter is packaged in the moses toolkit @xcite ), 3 which works by splitting on all sentence-final punctuation unless the preceding context is a "non-breaking prefix"-a hand-built, language-specific list of acronyms and abbreviations. this approach cannot resolve the ambiguity where punctuation legitimately exists at the end of a sentence and is indifferent to novel abbreviations at inference time. it produces a conservative segmenter that is high precision (unlikely to oversegment) but low recall (prone to undersegmenting). this raises the question of what effect reliance on this tool has had on construction of recent massive bitexts, such as ccmatrix @xcite . @xcite credit a 0.75% increase in accuracy to reduction of summarization error by a factor of four. errors in segmentation may therefore affect the top matches for a sentence when doing bitext construction. another popular splitter is spacy, which has not been described or evaluated anywhere, as far as we could tell. with sentence splitting being a crucial piece of modern corpus creation for machine translation and other tasks, the lack of approaches and rigorous comparisons between tools limits the field. additionally, the research field moving towards (often massively) multilingual settings, the need to build multilingual tools compare them in a proper scientific framework is both important and evident. ## approach our general approach is to treat sentence segmentation as a binary classification problem, predicting sentence-internal (⊗) or sentence-ending ( ) positions. the input to the model ( §3.1), shown in figure 1 , is the concatenated left and right token contexts, as depicted in ## evaluation: metric for evaluation, we begin by removing sentences that do not end in punctuation, since none of the tools are able to segment these. we then concatenate the test set into a single line, joining sentences with a space. evaluation among different tools contains subtle complexities. first, some tools normalize or tokenize the input text, complicating alignment between the input and the output. second, different tools may attempt to segment at different subsets of input string locations, which might unfairly bias the results in favor of conservative tools. finally, if we permit segmentation at any point of the input, there is a large class imbalance between ⊗ and . the class imbalance advocates for f1 as a natural metric. the use of f1 also addresses the second issue, since only the gold positive class ( ) factors into the score. the first two issues also require that we align a segmenter's output with the gold standard segmented text. since the texts are largely similar, we can do this efficiently using a modified levenshtein distance foot_6 that only considers a fixed maximum distance between any two characters. once the text is aligned, we compute f1 against the set of symbols in the gold text. an example is depicted in figure 2 . ## evaluation: data we have noted the difficulty with making use of imperfect training data, and how we hope to work around it ( §3.3). unfortunately, this workaround cannot be used for evaluation, where we need goldstandard data. we construct test sets from the wmt news translation test sets @xcite , which . for scoring, system outputs are independently aligned to the gold text, which accounts for text transformations made by some tools and allows precision and recall to be computed. ## experimental setup we consider three language settings: (i) monolingual english, (ii) a multilingual setting that includes the set of recent wmt languages plus arabic, and (iii) a much larger multilingual setting that includes the previous languages plus all languages with at least 10k lines in the wikimatrix @xcite dataset. starting with the english setting, we investigate the performance of a basic model and vary parameters such as context size, embedding size, and vocabulary size. after finding an optimal setting, we expand to the first multilingual setting and repeat. we train a single multilingual model that is agnostic of language and does not need language specification as input. similar to the monolingual setting, we vary the aforementioned parameters, and compare the best model to baselines ( §6.3). in order to test expandability, we then train with the same parameters on the largest set of languages (using the additional wikimatrix data), and compare to the previous model's performance. while we do not widely experiment with additional monolingual settings, we train monolingual models in each language to compare against the multilingual models' performance. we report the 8 iu was left uncorrected due to the fact that available bitext often aligned "sentences" with singular or compound sentences in english and a lack of automatic translation corresponding to sentences. comparison of these three settings to baselines in ## monolingual experiments we first explore common questions and concerns while focusing on english data and results. we have three main parameters to study: context size, embedding size, and vocabulary size. we additionally consider how the training data affects resultsboth in relative noise in class labels in addition to in general, we find our technique creates a monolingual english model ( ## multilingual experiments after outperforming current baselines in a monolingual english setting, we generalized our approach to work multilingually. the multilingual model can segment text irrespective of input language. in parallel to the monolingual conditions, we train two-layer transformer models with 6 tokens of left context, and 4 tokens of right context with 128 embedding size. while we did experiment with scaling these for the multilingual model, we found little effect. we additionally scale the vocabulary size to 12,000 to accommodate the larger character sets in chinese and japanese. because more of the additional languages have undersegmented data, we searched over potential lambda weights for the class and report the best configuration (λ = 20.0) in ## summary as one of the earliest steps in nlp pipelines, sentence segmentation is an important task. however, it has not to this date received proper experimental attention, relying instead on ad hoc methods. it is a good time to correct this oversight, as nlp moves to the use of larger and larger corpora covering more and more languages. even as the field moves towards processing text at the paragraph or document level directly, it is likely that sentence processing will be with us for some time. we show here that a simple context-based model can produce state-of-the-art results with a modest hyperparameter search, trained on noisy annotations from imperfectly-segmented data. together with a straightforward multilingual approach to identifying candidate split points and training on noisy segmented data, our single model performs well across a range of languages. more fundamentally, we have defined an experimental framework for benchmarking and future comparative work. missing from our paper is an evaluation of the effect of these tools on downstream tasks. an obvious candidate for future work is to conduct this evaluation. it is possible that some tasks will not be affected by small differences among the best performing models, but this work at least sheds light on those differences. another obvious direction is to look at approaches that would work for unpunctuated text (e.g., @xcite ). this would expand the functionality of segmenters into other important areas, such as speech translation, and to languages, like thai, that do not mark ends of sentences.