Sentence Identification with BOS and EOS Label Combinations (2023)

Abstract: The sentence is a fundamental unit in many NLP applications. Sentence segmentation is widely used as the first preprocessing task, where an input text is split into consecutive sentences considering the end of the sentence (EOS) as their boundaries. This task formulation relies on a strong assumption that the input text consists only of sentences, or what we call the sentential units (SUs). However, real-world texts often contain non-sentential units (NSUs) such as metadata, sentence fragments, nonlinguistic markers, etc. which are unreasonable or undesirable to be treated as a part of an SU. To tackle this issue, we formulate a novel task of sentence identification, where the goal is to identify SUs while excluding NSUs in a given text. To conduct sentence identification, we propose a simple yet effective method which combines the beginning of the sentence (BOS) and EOS labels to determine the most probable SUs and NSUs based on dynamic programming. To evaluate this task, we design an automatic, language-independent procedure to convert the Universal Dependencies corpora into sentence identification benchmarks. Finally, our experiments on the sentence identification task demonstrate that our proposed method generally outperforms sentence segmentation baselines which only utilize EOS labels.

## introduction the sentence, which we refer to as the sentential unit (su), is a fundamental unit of processing in many nlp applications including syntactic parsing @xcite , semantic parsing @xcite , and machine translation @xcite . existing works mostly rely on sentence segmentation (a.k.a. sentence boundary detection) as the first preprocessing task, where we predict the end of the sentence (eos) to split a text into consecutive sus @xcite . this approach relies on a strong assumption that the text only consists of sus; however, real-world texts like web contents often contain non-sentential units (nsus) such as the metadata of attachments embedded in the email body, repetition of symbols for separating texts, irregular series of nouns, etc. (just to name a few). such nsus may cause detrimental or unexpected results in the downstream tasks if considered as parts of the sus and are more desirable to be distinguished from sus in the first preprocessing step. to tackle this problem, we formulate a novel task of sentence identification, where the goal is to identify sus while excluding nsus in a given text ( §3). this can be regarded as an su span extraction task, where each su span is represented by the beginning of the sentence (bos) and the eos labels. 1 we illustrate the difference between sentence segmentation and sentence identification in to conduct sentence identification, we propose a simple method which effectively combines the bos and eos probabilities to determine both sus and nsus ( §4). to be specific, we first train the bos and eos labeling models based on either the sentence identification dataset (with sus and nsus) or sentence segmentation dataset (only sus). then, we search for the most probable spans of sus and nsus using a simple dynamic programming framework. theoretically, our method can be considered as a natural generalization of existing sentence segmentation algorithms. input text thank you. -text.@xmath2: text.@xmath3. i might just sell the car and get you to drive me around all winter. e sentence thank you. -text.@xmath4: text.@xmath5. i might just sell the car and get you to drive me around all winter. b e b sentence thank you. -text.@xmath6: text.@xmath7. i might just sell the car and get you to drive me around all winter. to conduct our experiments, we focus on the english web treebank @xcite as the primary benchmark for sentence identification and train the bos/eos labeling models by finetuning roberta @xcite . we also propose techniques to develop these models using a standard sentence segmentation dataset, i.e. the wall street journal corpus @xcite , which only contains clean, edited sus without any nsus. based on our experimental results, we demonstrate that our proposed method generally outperforms sentence segmentation baselines which only utilize eos labels ( §7). these results highlight the importance of combining the bos labels in addition to the eos labels for accurate sentence identification under various conditions. ## background sentence segmentation, a.k.a. sentence boundary detection, is the task of segmenting an input text into the unit of sentences. despite the long history of study @xcite and its importance in the entire nlp pipeline @xcite , this area has received relatively little attention. for one reason, the task has been recognized as "long solved" @xcite with the most recent approach reporting 99.8% f1 score on the standard english wall street journal (wsj) dataset @xcite . their state-of-the-art method er-satz combines (i) a regular-expression based detector of candidate sentence boundaries, followed by (ii) a transformer-based @xcite binary classifier which predicts whether the candidate boundary is eos based on the local context, i.e. surrounding few words. this modern context-based approach has been shown to outperform competitive, widely used baselines such as splitta @xcite , punkt @xcite , and moses @xcite . however, two important aspects are not fully addressed in the current literature. first is the coverage of diverse domains, genres, and writing styles. existing works (including @xcite focus on formal/edited text and assume the existence of sentence ending punctuations (e.g. full stops) at the sentence boundaries. however, social media texts often lack such punctuations and contain various types of non-linguistic noise, which can lead to a substantial degradation in the segmentation performance @xcite . speech transcription texts also usually contain disfluent, ungrammatical, or fragmented structures and lack both punctuations and casing @xcite . considering the amount of such informal or nonstandard texts in the real world, it is compelling to expand the capability of sentence segmentation beyond formal, standardized text. the second aspect is the coverage of multiple languages. different languages involve different complexities in sentence segmentation, e.g. chinese requires the disambiguation of commas as the sentence ending punctuation @xcite and thai does not mark eos with any type of punctations @xcite . to advance nlp from a multilingual perspective, it is crucial to develop and evaluate models in multiple languages: @xcite make an important step in this direction, proposing a language-agnostic, unified sentence segmentation model covering a total of 87 languages. based on these observations, we first propose to extend the task of sentence segmentation to sentence identification, which expands the capability of sentence segmentation beyond formal, standardized text ( §3, §4). secondly, we propose a crosslingual method of benchmarking sentence identification based on the ud corpora, considering every word or character as the candidate boundary to cover diverse domains, genres, and languages that lack sentence ending punctuations ( §5). finally, we follow @xcite to develop modern neural-based models that require no languagespecific engineering and can be developed for different languages in a unified manner ( §6). ## evaluation due to the novelty of the task, currently there exists no benchmark for evaluating sentence identification. to address this issue, we propose a fully automatic procedure to convert the universal dependencies (ud) corpora (de marneffe et al., 2021) into sentence identification benchmarks. concretely speaking, we conduct the following two steps based on the gold ud annotation: (i) the detection of unit (su and nsu) boundaries and (ii) the classification of each unit into su or nsu. as for (i), we simply use the original sentence boundaries in the ud annotation, where ud uses the term sentence in a broader sense including both sus and nsus (e.g. sentence fragments). note that the exact boundaries between consecutive nsus (which we call nsu-nsu boundaries) do not need to be accurate or consistent, since we are only interested in extracting the spans of sus. however, we do expect that the original boundaries are generally reliable in all other cases (su-su and su-nsu boundaries), which seems to be the case. the main problem is (ii), i.e. how to classify due to the reliance on ud, our conversion procedure can be applied to a wide variety of languages supported in ud (currently over 100 languages). however, as a first set of experiments, we focus on the english web treebank (ewt) @xcite as the primary benchmark of sentence identification. this dataset comprises five genres of web media texts: namely weblogs, newsgroup threads, emails, product reviews, and q&a websites. consequently, the dataset contains formal sus, informal sus (e.g. without capitalization or punctuations) as well as a wide variety of nsus. we show some examples of nsus in finally, we summarize the dataset statistics of ewt in overall, we could verify that there exists a nonnegligible amount of nsus in the ewt dataset, which we aim to exclude with sentence identification in our experiments. ## results firstly, we take a look at the results when we have the in-domain sentence identification dataset (ewt train/dev) for model development. in this setup, we can verify that our proposed method (bos&eos) significantly outperforms the baselines (eos-only) in all metrics. for instance, our method achieves consistently high performance of 84∼89% f1 for the exact su span extraction, both at the word-and character-level. this is a very promising result that demonstrates the effectiveness of our method when we can leverage the gold sus and nsus from the target domain. secondly, we focus on the results where we only utilize the standard sentence segmentation dataset (wsj train/dev) for model development. in this setup, we also report the results of applying our data augmentation (+aug) and unidirectional model (+uni) techniques from §6. 1. 7 due to the data discrepancy between wsj and ewt, we find a natural drop in performance compared to the previous setup using in-domain ewt train/dev. however, we can verify that our techniques (+aug, +uni) generally help to alleviate this issue, and our proposed method performs on par or slightly better than the eos-only baselines when applying these techniques. it is especially worth noting the improvement in the exact su span extraction task (reaching 64∼72% f1), where the advantage of our method is the most conspicuous and consistent in both word-and characterlevel evaluation. this improvement can also be explained by the higher performance in the b-label prediction with our method (appendix e), which is a prerequisite for accurate su span extraction. finally, we note that the eos-only baseline without forcing the last eos can be quite competitive with shorter inputs (p @xmath8.5 and postprocessing) but performs considerably worse when the input texts are longer (p @xmath9. this is because the baseline can only predict the last segment of the input as an nsu, which is less problematic when the input texts are shorter but becomes increasingly problematic with longer inputs (since most nsus will not be able to be removed). in contrast, our proposed method performs much more robustly under various input lengths. through further experiments and analyses, we 7 we did not observe any improvement from applying these techniques to the in-domain dataset (ewt train/dev), which is consistent with our motivation and expectation. ## conclusion in this paper, we introduced a novel task of sentence identification, where we aim to identify sus while excluding nsus in a given text ( §3). through sentence identification, we can clearly distinguish the portions of the text that are appropriate (or not) for the prediction and evaluation of sophisticated linguistic analyses, such as dependency parsing, semantic role labeling, etc. to conduct sentence identification, we proposed a simple yet effective method of combining the bos and eos labeling models to determine the sus and nsus ( §4). to evaluate sentence identification, we designed an automatic, languageindependent procedure to convert the ud corpora into sentence identification benchmarks ( §5). in our experiments, we developed the bos/eos labeling models by finetuning pretrained roberta ( §6). based on the experimental results, we showed that our proposed method combining the bos and eos labels outperforms sentence segmentation baselines which only utilize eos labels in all of the considered settings ( §7). overall, we expect sentence identification to be a fundamental framework for the preprocessing of noisy, informal, or non-standard texts in the real world.