P y SBD : Pragmatic Sentence Boundary Disambiguation (2020)

Abstract: We present a rule-based sentence boundary disambiguation Python package that works out-of-the-box for 22 languages. We aim to provide a realistic segmenter which can provide logical sentences even when the format and domain of the input text is unknown. In our work, we adapt the Golden Rules Set (a language specific set of sentence boundary exemplars) originally implemented as a ruby gem pragmatic segmenter which we ported to Python with additional improvements and functionality. PySBD passes 97.92% of the Golden Rule Set examplars for English, an improvement of 25% over the next best open source Python tool.

## introduction sentence boundary disambiguation (sbd), also known as sentence boundary detection, is a key underlying task for natural language processing. in many nlp pipelines, gold standard sbd often assumed, and acts as a primary input to downstream nlp tasks such as machine translation, named entity recognition and coreference resolution. however, in real world scenarios, text occurs in a variety of input modalities, such as html forms, pdfs and word processing doccument formats. although sbd is considered to be a simple problem, it becomes more complex in other domains due to unorthodox use of punctuation symbols. for example, drug names in medical documents, case citations in legal text and references in academic articles all use punctuation in ways which are uncommon in newswire documents. simple sbd approaches for english web text (treating -"?!:;." -as end of sentence markers) covers a majority of cases, but an ideal sbd system should be able to disambiguate these edge case scenarios and be robust to known textual variation. our contributions in this paper are describing an open-source, freely available tool for pragmatic sentence boundary disambiguation. in particular, we describe the implementation details of pysbd, evaluate it in comparison to other open source sbd tools and discuss it's natural extensibility due to it's rule based nature. ## related work sentence segmentation methods can be broadly divided into 3 approaches: i) rules-based, requiring hand crafted rules/heuristics; ii) supervised machine learning, requiring annotated datasets, and iii) unsupervised machine learning, requiring distributional statistics derived from raw text. @xcite use decision trees and neural networks in a supervised, feature based sbd model, requiring part of speech information and training data. @xcite design punkt, an unsupervised sbd model centered around the observation that abbreviations are the main confounders for rule based sentence boundary models. although it is unsupervised, punkt requires the computation of various co-occurence and distributional statistics of a relevant corpora; as pysbd is rule-based, it does not require a initial corpus of text. @xcite cast sbd as a character sequence labelling problem and use features from a recurrent neural network language model to train a crf labelling model. many sbd papers reject rule-based approaches due to non-robustness, maintainability and performance. we reject these conclusions, and instead focus on the positive features of rule-based systems -namely that their errors are interpretable, rules can be adjusted incrementally and their performance is often on-par with learnt statistical models. issues with benchmarks on ptb/wsj corpora sbd systems have historically been benchmarked on the wall street journal/penn treebank corpora @xcite . the majority of the sentences found in the penn tree bank are sentences that end with a regular word followed by a period, testing the same sentence boundary cases repeatedly. in the brown corpus 90% of potential sentence boundaries come after a regular word. although the wall street journal corpus is richer with numerical values, abbreviations and only 53% according to @xcite of sentences end with a regular word followed by a period @xcite . given that commonly used training/evaluation corpora do not contain a particularly large amount of sentence marker variation, we use a golden rule set to enumerate edge cases observed in sentence boundaries. the golden rule set contains 48 hand-constructed rules, designed to cover sentence boundaries across a variety of domains. the grs is interpretable (each rule targets a specific type of sentence boundary) and easy to extend with new examples of particular sentence boundary markers. ## implementation pysbd is divided into four high level components: the segmenter, processor, language and cleaner sub-modules. the segmenter class is the public api to pysbd. it allows a user to set up a segmenter in their language of choice, as well as specify additional options such as text cleaning and char span functionality. the segmenter requires a two character iso 639-1 code 2 to process input text. text extracted from a pdf or obtained from ocr systems typically contains unusually formatted text, such as line breaks in the middle of sentences. this can be handled with the doc type option, or for more aggressive text cleaning, the clean functionality performs additional pre-filtering of the input text, removing repeated and unnecessary punctuation. the processor contains the sentence segmentation logic, using rules to segment the input text. the processor contains several groups of sentence segmentation rules, some of which are universal across languages, and some of which are language specific. these are grouped as follows: the processor identifies sentence boundaries by manipulating input text in 3 stages. firstly, rules are applied to alter the input text by adding intermediate unicode characters as placeholders to signify that particular pieces of punctuation are not sentence boundaries. the segment stage identifies true sentence boundaries by bypassing unicode characters and segments text into sentences using a much simpler regex rule. finally, the manipulated text is transformed into original text form by replacing the unicode placeholders with their original characters. the language holds all the languages supported by pysbd. each language is built on top of two sub-components -common and standard -involving basic rules prevalent across languages. common rules encompass the main sentence boundary regexes; am-pm regexes handle numerically expressed time periods; number regexes handle period/newline characters before or after single/multidigit numbers and additional rules handle quotation, parenthesis, and numerical references within the input text. the standard rule set contains regex patterns to handle single/double punctuation, geolocation references, fileformat mentions and ellipsis in input text. the listitemreplacer rule set handles itemized, ordered/unordered lists; the abbrevia-tionreplacer contains language specific common abbreviations. finally, the exclaimationwords and betweenpunctuation rules handle language specific exclamations and more complicated punctuation cases. in practice, text encountered in the wild is noisy, containing extraneous line breaks, unicode characters, uncommon spacing and hangovers from document structure. in order to handle this, pysbd provides an optional component to handle such texts. the cleaner is passed as an option through the top-level segmenter component and provides text cleaning rules for cases like irregular newline characters, tables of contents, urls, html tags and text involving no space between sentences. as the text cleaning rules perform a destructive operation, this feature is incompatible with the char span functionality, as mapping back to character indices within the original text is no longer possible. ## experimental setup data contrary to the wsj, brown and genia datasets mentioned in @xcite , we use language specific golden rules sets for our experiments. there are total 22 golden rules sets for following languages -english, marathi, hindi, bulgarian, español (spanish), russian, arabic, amharic, armenian, persian, urdu, polish, chinese, dutch, danish, french, italian, greek, burmese, japanese, deutsch (germen), kazakh. the golden rules sets are devised by considering possible sentence boundaries per language as well as considering different domains. for example the english language grs is comprised of 48 golden rules foot_1 from formal and informal domains to cover a wide variety of phenomena. for example, news articles are grammatically and punctually correct; scientific literature often involves numbers, abbreviations and bibliography references, and informal domain like web text -e-mail, social media text involves irregular punctuation and ellipses. to ensure our rules-based system built with respect to the golden rules set generalizes well in the real world, we have also performed a benchmark comparison on the genia corpus @xcite , a dataset of linguistic annotations on top of the abstracts of biomedical papers. the genia corpus provides both raw and segmented abstracts, which we use as natural data for our evaluation. setup we evaluate pysbd and other segmentation tools on two corpora -the english golden rules set and genia corpus. ## comparison to alternatives ## conclusion in this paper, we have described pysbd, a pragmatic sentence boundary disambiguation model. pysbd is open source, has over 98% test coverage and integrates easily with existing natural language processing pipelines. pysbd currently supports 22 languages, and is easily extensible, with 57 projects depending on it at the time of writing. although slower than some alternatives implemented in low level languages such as c++, pysbd successfully disambiguates 97% of sentence boundaries in a golden rule set and is robust across domains and noisy text.