Automatic sentence segmentation of clinical record narratives in real-world data (2024)

Abstract: Sentence segmentation is a linguistic task and is widely used as a pre-processing step in many NLP applications. The need for sentence segmentation is particularly pronounced in clinical notes, where ungrammatical and fragmented texts are common. We propose a straightforward and effective sequence labeling classifier to predict sentence spans using a dynamic sliding window based on the prediction of each input sequence. This sliding window algorithm allows our approach to segment long text sequences on the fly. To evaluate our approach, we annotated 90 clinical notes from the MIMIC-III dataset. Additionally, we tested our approach on five other datasets to assess its generalizability and compared its performance against state-of-the-art systems on these datasets. Our approach outperformed all the systems, achieving an F1 score that is 15% higher than the next best-performing system on the clinical dataset.

## introduction sentence segmentation is the task of automatically identifying the boundaries of sentences in a written document, where a sentence is commonly defined as a sequence of grammatically linked words ending with a punctuation mark (pm). it is often the first pre-processing step for other natural language processing (nlp) tasks such as sentiment analysis @xcite , information extraction @xcite @xcite , semantic textual similarity @xcite , question answering ( @xcite , and machine translation @xcite . even tasks that operate at the paragraph or document level, such as coreference resolution @xcite or summarization @xcite , often make use of sentences internally. errors in segmentation * these two authors contributed equally. despite its importance, sentence segmentation has received much less attention in the last few decades than other linguistic tasks. for non-clinical text, high-performing baseline systems use simple rule-based @xcite or machine learning-based @xcite approaches that capture obvious and frequent sentence ending pms (epms) such as @xcite . such baselines leave little room for further improvement on traditional benchmarks derived from formal news(wire) sources or published articles. the focus on formal or edited text assumes epms as sentence boundaries, which is not directly applicable to real-world data such as clinical text @xcite or web text. these type of texts often contain fragmented and incomplete sentences, complex graphemic devices (e.g. abbreviations, and acronyms), and markups, which present challenges even for state-of-the-art sentence segmentation approaches, e.g., 70-85% f1 score on english web treebank @xcite . another comprehensive evaluation of sentence segmentation in the clinical domain reveals that four standard sentence segmentation tools perform 20-30% worse on clinical texts compared to general-domain texts @xcite . here, we present a sentence segmentation approach specifically tailored for real-world data, particularly clinical notes. our method uses a sequence labeling classifier to predict sentence spans over a sliding window. during inference, we dynamically slide the window based on the predic-tion of each input sequence, such that the window always starts with a complete predicted sentence. this allows our approach to segment long text sequences on the fly without needing to pre-split the text. moreover, the sequence labeling classifier does not rely on pms for segmentation. to evaluate our approach on real-world clinical texts that can be shared, we annotated 90 clinical notes from mimic-iii. additionally, we extensively tested our method on five other datasets to assess its generalizability. unlike other studies @xcite that have modified datasets for sentence segmentation, we retained the original raw text, preserving their form and document structure. our work makes the following contributions: ## related work existing sentence segmentation approaches can be categorized into rule-and learning-based approaches. rule-based approaches @xcite @xcite utilize handcrafted rules, abbreviation lexicons, and linguistic features to decide whether a pm belongs to a token (an abbreviation or a number), or indicates the end of a sentence. for instance, stanford corenlp toolkit @xcite utilizes rules such as sentence ending pms, or two consecutive line breaks to segment text. however, one major limitation of rule-based approaches is that the handcrafted rules are language-or domain-specific, making them dif-ficult to maintain and adapt to new texts. as an alternative, other systems aim to automatically learn segmentation rules through machine learning algorithms. when working with unlabeled data, unsupervised approaches @xcite automatically curate information about abbreviations and proper names from large corpora and use them to determine whether the token preceding a period is an abbreviation and whether the token following a period is a proper name. one representative algorithm of the approach is in the punkt system @xcite , as it computes the likelihood ratio of the truncated words and the following periods to identify abbreviations. an implementation of punkt is bundled with the nltk tool @xcite . although these unsupervised approaches do not require extensive lexical resources or manual annotations and are easily adaptable to new domains, they can only segment sentential units (sus) that use periods as sentence boundaries. with the increasing availability of annotated corpora, supervised learning approaches have become predominant. one type of supervised approach combines a regular-expression-based detector to generate candidate sus with a binary classifier. for generating candidate sus, researchers have focused on only periods @xcite , multiple epms @xcite @xcite , or more complex regular expressions @xcite . for classifying candidate sus, most approaches employ binary classifiers with various features, e.g., a feedforward neural network with pos tags features @xcite , an svm classifier with features such as length and the case of the words occurring before and after the pms @xcite , deep neural models using characters from the surrounding context @xcite of candidate sus, or a two-layer transformer encoder using the surrounding context words @xcite . however, all these approaches focus on proofread and edited documents, always assuming the existence of epms in all sus. this assumption does not hold for informal, user-generated text or clinical notes with minimal proofreading and post-editing. as a consequence, several studies noted a substantial decline in performance when these systems move to texts with less formal language @xcite . pred sw-3 sw-2 sw-1 text neuro : <\@xmath0: sedated . <\@xmath1. b i o o o b i i b i i i i o b i b i i i i i b i o o o b i i i i o b i i i i i sentence-1 sentence-2 sentence-3 figure 1 : sliding window algorithm for sentence segmentation. we segment the text using three sliding windows sequentially (sw-1, sw-2, and sw-3). each sliding window contains up to 8 tokens. the final sentence segmentation tags are at the top (pred) of the diagram. ## methods we approach sentence segmentation as a sequence labeling task using a bio tagging scheme (shown in figure 1 ). in this scheme, each token in an input sequence is assigned a tag to mark sentence boundaries: b indicates the beginning of a sentence, i represents inside of a sentence, and o denotes outside of a sentence. we chose this tagging schema as it allows not only to segment sentences from a document but also to differentiate sus (labelled as b and i) from non-sus (labelled as o), also known as sentence identification task @xcite . non-sus typically include metadata from email attachments, markups in web text, irregular series of nouns, repetition of symbols for separating texts, and plain text tables in clinical notes, among other examples. all these non-sus require additional text cleaning for downstream tasks. unless otherwise specified, we do not differentiate between sentence identification and sentence segmentation in the following sections. formally, let @xmath2, t 1 , ..., t n-1 ] represent an input sequence that consists of n tokens; @xmath3, y 1 , ..., y n-1 ] represent a sequence of bio labels. so the goal of sentence segmentation task is to find a label sequence y which satisfies: ## results on the mimic-iii dataset, across five other standard benchmark datasets, mimic-iii p i2b2-2010 genia ewt brown wsj 60 80 100 87.03 97.71 99.91 71.86 80.07 90.76 60.6 69.65 97.16 91.1 98.39 93.55 segmenter-data and segmenter-domain, consistently achieve the best f1 on four datasets (except the wsj dataset), for an average rank of 1.6 and 2, respectively. trankit achieves the best performance on the wsj dataset, with an average rank of 3.5. compared against segmenter-data models that are trained on each individual datasets, segmenter-domain models that are trained on the combination of datasets from each domain, achieves nearly identical performances. this suggests that instead of maintaining six separate models, we can effectively use just two models for the segmentation task. on another clinical dataset -i2b2-2010, all tools except ctakes achieve less than 58.68% of f1; while on a well-formed dataset -genia, all tools except ctakes achieve more than 96.93% of f1. along with the evaluation on mimic-iii dataset, we find that tools developed on the general domain texts struggle with clinical texts; however, they still achieve great performances on biomedical articles. this indicates that sentence segmentation is influenced not only by domain-specific language, such as terminology and abbreviations, but also by sentence structure and text form. surprisingly, comparing the performances of ctakes on i2b2-2010 and mimic-iii, we see a big performance drop. this is probably because the training data used in ctakes is more similar to the i2b2-2010 corpus. ). this is likely because both corenlp and syntok fail to account for characteristics of web language, such as fragmented text and the absence of epms. besides ctakes, which is designed specifically for the clinical domain, both nltk and spacy achieve the worst performance on one of the three general domain datasets. we analyzed the sentence segmentation experiments with spacy on the wsj corpus, and found that newline characters caused many segmentation errors. after removing newline characters from the documents, we achieved an f1 score of nearly 96%. ## discussion from the evaluation of off-the-shelf tools, we can see inconsistent performances on different datasets. this is expected because of language variation, sentence structures, and text form. to check whether such a phenomenon also exists in our approach, we conducted a cross-domain evaluation for our segmenter-domain models, i.e., evaluating models trained on biomedical domain datasets on the general domain datasets, and vice versa. figure 3 shows similar findings as other tools: except on ge-nia and wsj, there are around 27% of f1 drop on biomedical datasets, and around 20% of f1 drop on general domain datasets. we also performed cross-dataset evaluation (models that are trained on one dataset and then evaluated on other datasets) for segmenter-data models, but the decline in performance was even more pronounced. we posit that segmenter-domain models hold better applicability in real-world scenarios due to their ability to generalize across multiple datasets. to understand how each tool and our approach work on different text form, we compute the recall of top 4 off-the-shelf tools (based on their average rank in ## conclusion in conclusion, our proposed sentence segmentation approach addresses the challenges posed by realworld, ungrammatical, and fragmented text used in the daily, often harried and hectic hospital environment when typing clinical notes. utilizing a sequence labeling classifier with a dynamic sliding window, our approach effectively segments long text sequences on the fly without requiring presplitting or relying on pms. additionally, we contribute a new sentence segmentation dataset derived from the mimic-iii corpus, providing a valuable resource for future research in this domain. the evaluation on our annotated clinical notes, along with extensive testing on five additional datasets, demonstrated the generalizability and effectiveness of our approach over seven commonly used tools.