Efficient Deep Learning-based Sentence Boundary Detection in Legal Text (2022)

Abstract: A key component of the Natural Language Processing (NLP) pipeline is Sentence Boundary Detection (SBD). Erroneous SBD could affect other processing steps and reduce performance. A few criteria based on punctuation and capitalization are necessary to identify sentence borders in well-defined corpora. However, due to several grammatical ambiguities, the complex structure of legal data poses difficulties for SBD. In this paper, we have trained a neural network framework for identifying the end of the sentence in legal text. We used several state-of-the-art deep learning models, analyzed their performance, and identified that Convolutional Neural Network(CNN) outperformed other deep learning frameworks. We compared the results with rule-based, statistical, and transformer-based frameworks. The best neural network model outscored the popular rule-based framework with an improvement of 8% in the F1 score. Although domain-specific statistical models have slightly improved performance, the trained CNN is 80 times faster in run-time and doesn’t require much feature engineering. Furthermore, after extensive pretraining, the transformer models fall short in overall performance compared to the best deep learning model.

## introduction from linguistic theory, a sentence is a textual segment or span of one or more grammatically correct words representing a complete thought. declarative statements, questions, exclamations, requests, commands, and suggestions can all be expressed in sentences. a grammatical subject and grammatical predicate are typically present in an expressive sentence. the person, place, or object (including abstract concepts) that the sentence is about is the grammatical subject, which is typically a noun phrase, and a verb phrase serves as the grammatical predicate @xcite . it is quite simple for a person to separate a given text into sentences. however, one realizes how difficult this task is when attempting to condense this segmentation into rules that a machine could follow. many natural language processing (nlp) applications, including part-of-speech taggers, named entity recognition, document indexing, and questionanswering, use sentence boundary detection as a crucial pre-processing step. the pre-processing requirements depend on both the nature of the corpus and the nlp application. thus sbd faces challenges when working with a specialized corpus like legal text because it offers issues distinguishing between citations, abbreviations, and law-specific keywords. existing sbd systems work well for generic corpora but pose serious issues when dealing with legal domain. the language, structure, and content are very different and more challenging to understand. when working with sbd, the fundamental presumptions used with the generic corpora are not always applicable. legal documents typically consist of smaller components like paragraphs, sentences, etc. sentences can be lengthy and contain intricate structures like lists. there is no standard formatting structure or style for legal documents, even those of the same category (such as statutes or judgments). a sentence segmentation system must resolve these issues to create pure sentences from the mixture of such diverse textual elements. one main difficulty in sbd is to pinpoint potential boundary spots(e.g. "."). the uncertainty of the delimiter period "." symbol, which has multiple purposes in the legal domain, makes it challenging. it could denote the end of a sentence, an acronym, an initialism, a numerical number @xcite , or part of a citation in legal text. to determine if a punctuation character is actually a sentence end marker, a sentence boundary detection system must to resolve the issue of using ambiguous punctuation characters. with this motivation of identifying the potential end-of-sentence marker, we build a deep-learning architectural framework using the context at the character level surrounding the period as input. the main contribution of this paper is that we use a context window-based deep learning framework for sentence boundary detection in legal text. we compared various deep learning models, identified the best framework, and compared the models with the existing state-of-the-art rule-based and statistical models. we also compared the deep learning model with various transformer architectures like legal-bert @xcite and xl-net @xcite . the rest of the paper is organized as follows: section 2 provides an existing literature summary of earlier studies about sentence boundary detection. section 3 gives a brief description of the dataset and its associated pre-processing. we describe our methodologies and the suggested neural network architecture in section 4. section 5 covers results with evaluation techniques and performance comparisons. finally, the paper concludes with some future steps in section 6. ## related work sentence boundary detection in a normal english text is regarded as an answered problem with robust methods. in the nlp literature, several methods for identifying sentence borders have been studied, encompassing algorithms and models ranging from rule-based, statistical, and machine learning-based models. several techniques for recognizing sentence boundaries across various corpora are presented in the seminal paper @xcite . decision tree algorithm, support vector machines (svm) @xcite , bayesian networks, and unsupervised methods like punkt @xcite are among the various algorithms for generic sbd. the identification of sentence boundaries in speech transcriptions is crucial for enhancing readability and supporting later language processing modules. in @xcite , prosodic and textual information sources are used to identify speech sentence boundaries, and @xcite detect sentence boundaries and speaker changes in the unpunctuated text. recently a wellknown rule-based model called pragmatic sentence boundary disambiguation(pysbd) @xcite with more than 98 percent test coverage was developed as an open-source pack-age. pysbd supports 22 languages and is robust in noisy text and domains. we have used pysbd as our baseline model for comparison. in general, these algorithms work well for processing text that adheres to the rules of normal english but operates poorly in other areas. to yield acceptable findings in fields like medical @xcite , scientific @xcite , legal, and finance areas @xcite , the algorithms used are heavily customized. the main issues with current legal sbd systems are a lack of compliance with known sentence patterns, sentence length, and the use of punctuation, particularly periods as non-sentence ending characters. legal experts use "linguistic indications, structure, and semantic interpretations which interact with domain knowledge" @xcite . @xcite examines the use of conditional random fields (crf) models @xcite for sentence boundary identification in legal documents since these models are frequently used for sequence modeling, or assigning labels for the items in the connected input sequence. here they developed many crf models using basic textual features. they employed an aggressive tokenization technique that divides the text into more tokens than normal. the tokens are represented using simple features like the length of the token, whether it is a digit or space, or is written in upper-case or lower-case, etc. a token's features are a combination of its characteristics and features obtained from nearby tokens. for the final model's training, these corresponding feature extractors were utilized, which led to an increase in the inference time of the models. later @xcite examined the same dataset with punkt, crf, and bidirectional lstm neural network architecture. they enhanced the performance of the punkt model by training it with an updated abbreviation set based on the legal text domain. in neural network architecture, the sentence token is represented by a concatenation of word2vec @xcite embeddings using a three-word window and eight features fed as input to the stacked bilstm network with a softmax layer as output. they concluded that this token classification architecture did not result in an sbd that was superior to the crf model. the issue of detecting sentence boundaries can be viewed as a classification problem. so our work is inspired by @xcite ## dataset the algorithm was trained using a dataset @xcite of 80 court decisions in four different domains: cyber crime (cc), intellectual properties (ip), board of veterans (bva), and the united states supreme court (sc). these decisions were put in four json files of 20 decisions each, along with the list of offsets that denotes the sentence boundaries. the complete dataset of 26052 annotated sentences is publicly available. 1 the summary of the statistics of the four decision sets is specified in the dataset composition of the delimiter occurrences is shown in ## model architecture the architecture of the deep learning framework is depicted in fig. 1 . this context window-based model architecture is used for training sequence classification neural network models. once the file identifies the delimiter, the corresponding left and right contexts with efficient window size are ## results and discussion we organize the results into three subsections. the first two sections focus on evaluation patterns used in our research to compare with the existing stateof-the-art models. in the first section, we compare the deep learning models against the baseline models based on offset boundaries. the second evaluation provides an inference time-based performance analysis of all the models. the final result section covers the performance assessment of the deep learning models to the architecture for the binary classification task. ## conclusion this paper uses a context window-based deep learning model framework for efficient sentence boundary detection in legal text. we compared various deep learning models, including transformers, for analysis. we showed that cnn showed a better per-formance when compared to other deep learning models. this model also outperformed the popular rule-based pysbd framework. even though the statistical model has a minor performance improvement, the trained cnn had a decent performance without the requirement of exhaustive feature engineering compared to domain-specific crf models. also, cnn is faster than the state-of-the-art crf models by multiple folds compared to the running time. as a result, the convolutional neural network is the model with the best performance. in the future, we plan to broaden the scope of our architecture by including the different delimiters found in legal text. also, we aim to chain multiple models together to improve the sbd performance. however, we could demonstrate that our model had an excellent performance and could thus be incorporated into nlp pipelines for various downstream legal tasks.