Quantifying Clinical Outcome Measures in Patients with Epilepsy Using the Electronic Health Record (2022)

Abstract: A wealth of important clinical information lies untouched in the Electronic Health Record, often in the form of unstructured textual documents. For patients with Epilepsy, such information includes outcome measures like Seizure Frequency and Dates of Last Seizure, key parameters that guide all therapy for these patients. Transformer models have been able to extract such outcome measures from unstructured clinical note text as sentences with human-like accuracy; however, these sentences are not yet usable in a quantitative analysis for large-scale studies. In this study, we developed a pipeline to quantify these outcome measures. We used text summarization models to convert unstructured sentences into specific formats, and then employed rules-based quantifiers to calculate seizure frequencies and dates of last seizure. We demonstrated that our pipeline of models does not excessively propagate errors and we analyzed its mistakes. We anticipate that our methods can be generalized outside of epilepsy to other disorders to drive large-scale clinical research.

## introduction the electronic health record (ehr) is a longitudinal catalog that describes patient visits, conditions, treatments, and well-being; thus, the ehr has significant potential for use in clinical informatics. unfortunately, much of the data in the ehr is stored as unstructured text in the form of hand-typed doctor's notes, which makes rapid information extraction traditionally difficult. however, recent developments in neural models, namely transformers @xcite like bert @xcite , have opened up exciting new avenues of research. such developments have been applied to epilepsy, a neurological disease characterized by recurrent unprovoked seizures. in epilepsy, seizure frequency and the date a seizure most recently pipeline steps raw note text extracted sentences summaries quantities long text short text x per y [unit] x [unit] ago date string two episodes of staring in the past 6 months hpi: patient presents today for evaluation of seizures… their last seizure was on 7/9/2013 last event was 4 or 5 years ago extractive qa models 2 per 6 month july 9 2013 4 years ago 0.33 2013-07-09 00:00:00 2012-11-21 00:00:00 x (per month) datetime object summarizers (t5 models) previous work current study rules-based quantifier occurred on are among the most important clinical outcome measures for patients. in @xcite , we previously used specially finetuned bio_clinicalbert @xcite and roberta @xcite models to extract, with human-like performance, sentences with a patient's seizure frequency and date of last seizure from clinical progress notes. these sentences contain temporal information and can thus be considered time expressions (timex). however, such timex only simplify the problem of extracting such information from a document to extracting such information from a sentence, and ## methods this retrospective study was approved by the institutional review board of the university of pennsylvania with a waiver of informed consent. in @xcite , we finetuned bio_clinicalbert @xcite and roberta @xcite on a combination of public datasets and proprietary clinical notes to extract sentences with seizure frequency and dates of last seizure from clinical notes. we framed this task as an extractive question-answering problem, where we asked the model to identify statements that answered the questions "how often does the patient have seizures," and "when was the patient's most recent seizure?" we demonstrated that our models achieved human-like performance relative to clinicians and researchers working in epilepsy-related research. quantifying seizure frequency and dates of last seizures from these sentences is therefore a timex normalization task, which seeks to convert a timex statement like "their last seizure was on 7/9/2013" into the datetime object 2013-07-09 00:00:00. more difficultly, seizure frequency and date of last seizure are represented in a number of nonstandardized ways by clinicians, precluding the use of simple rules-based quantification. we characterize broad categories and provide illustrative examples of such representations in we finetuned two t5-large models @xcite using huggingface @xcite , on the training sets and made predictions on the test sets. one t5-large model summarized sentences of seizure frequency, while the other summarized sentences of last seizure. we used huggingface's default parameters for text summarization and did not perform any hyperparameter optimization. we then developed a rules-based quantifier that normalizes a frequency summary into a numerical value, and converts a date summary into a datetime object. for summaries of seizure frequency, we take the "x" value in "x per y we also created a rules-based quantifier specifically for the seizure calendars, as the summarizer was unable to produce an accurate summary of this format of frequency. this seizure calendar quantifier identifies a sentence as a seizure calendar if it has at least two months, and at least two numbers. it then associates a month to its number of seizures by assuming that the number of seizures either directly follows the month in the text (e.g. "january: 1"), or precedes the month within three words (e.g. "1 seizure in january")." it counts the number of months and accumulates the number of seizures in that time span to calculate a monthly seizure frequency. we manually calculated the accuracy of each step of our approach in an all-or-nothing approach by comparing a statement to its downstream summary or quantity; a step was correct only if both its format and value given the context were correct. ## results we finetuned our t5 models for text summarization using a training set of 700 annotations, and a testing set of 300 annotations. to determine how much error we were propagating through our pipeline, we calculated the accuracy of each step in our method using the testing set ( reason times erred times erred (seizure frequency) (last seizure) competitive temporal statements 2 2 "since last visit: ... one ... seizure in the past year" "on the same day as their last appointment ..." no temporal reference 2 2 "they think they only had two ... seizures" "two weeks later they had another seizure" using month as value 2 0 "since 4/2012 they have had a few seizures" "since last office visit, they have had seizure 8/12/16" finally, we attempted to identify patterns of errors in our incorrect summaries. we manually catalogued these errors for both sentences of seizure frequency and dates of last seizure, and determined potential reasons for such problems ( ## discussion in this study, we normalized timex containing seizure frequency and date of last seizure by simplifying them with text summarization models, and applying simple rules-based quantifiers to extract quantitative outcome measures for patients with epilepsy. we demonstrated that this pipeline can accurately calculate quantitative seizure frequencies and dates of last seizure. though applied specifically to epilepsy, our methods are not constrained just to neurological disorders, and can be easily adapted to other medical conditions as well. our findings pave the way for large-scale clinical informatics research through extracting and quantifying textual information from the ehr. our full pipeline, including our previous work from @xcite , extracts timex from clinical documents, simplifies them using neural models, and normalizes them with rules-based methods to obtain quantitative outcome measures. the overall process is reminiscent of other temporal understanding studies. for example, @xcite developed a pipeline for temporal understanding that involves a begin-inside-outside (bio) tagging scheme with machine learning to extract timex, and a rules-based method to normalize them. meanwhile, @xcite formulated timex normalization as a sequence of operations that selects and applies normalization rules, and @xcite extracted timex from clinical text using machine learning-based bio taggers on two clinical datasets. additionally, to our knowledge, we are the first to use neural text summarization as an intermediate step to simplify variable timex into a standardized template for easy rules-based quantity extraction in the clinical domain. however, similar approaches exist in other domains and tasks. for example, @xcite used a seq-to-seq model to normalize the often complex and non-standard text found in social media into more standard forms. additionally, @xcite tested how various sentence simplification methods improved the informativeness of extractive text summarization methods, while @xcite compressed sentences in a manner that simplified the sentence but preserved its sentiment as a preprocessing step for aspect-based sentiment analysis. our categories of errors are also in line with what has been seen in the literature for transformers. for example, @xcite found that in extractive question-answering tasks, bert models showed remarkably lower performance on competitive i-don't-know questions (where a plausible but incorrect answer of the correct type exists in the context), mirroring our summarization errors when competitive time frames were presented. our study does have limitations. first and foremost, our methodology was developed using data from a single institutional healthcare center. while we used a neural summarizer in the hopes of improving overall generalizability to the various ways of representing outcome measures in text, it is still possible that the summarizer will fail to generalize to text from other health centers. we are actively evaluating of our methods at a collaborating institution to access this effect. additionally, 21 of 22 summaries that involved previous visits could not be actively quantified with this dataset, as the date of the previous visit did not exist in the 300 test notes. this can easily be corrected by performing a larger longitudinal study across our patients that would allow us to track them through their visits. ## conclusions we created a generalized two-step system that rapidly and accurately extracts and quantifies seizure frequency and date of last seizures. we used the t5 model to create standardized summaries of sentences of these outcome measures, and then applied a rules-based algorithm to extract and quantify the desired information. we anticipate that our methods can be used to quantify important clinical outcome measures not only for patients with epilepsy, but other disorders as well, allowing for large-scale clinical research in the future.