Extracting Epilepsy Patient Data with Llama 2 (2024)

Abstract: We fill a gap in scholarship by applying a generative Large Language Model (LLM) to extract information from clinical free text about the frequency of seizures experienced by people with epilepsy. Seizure frequency is difficult to determine across time from unstructured doctors’ and nurses’ reports of outpatients’ visits that are stored in Electronic Health Records (EHRs) in the United Kingdom’s National Health Service (NHS). We employ Meta’s Llama 2 to mine the EHRs of people with epilepsy and determine, where possible, a person’s seizure frequency at a given point in time. The results demonstrate that the new, powerful generative LLMs may improve outcomes for clinical NLP research in epilepsy and other areas.

## introduction advances in natural language processing (nlp), in particular pre-trained transformers @xcite and large language models (llms), create opportunities to develop new methodologies to mine free-text electronic health records (ehrs) for clinical research. one such opportunity is to investigate associations between anti-seizure medications (asms) and the frequency of seizures suffered by people with epilepsy, which is typically recorded in free text in the uk's national health service (nhs). mostly, this text consists of doctors' and nurses' reports of outpatients' ambulatory visits; the reports are shared with a patient's primary care physician in the form of a letter. the majority of hospital care episodes for people with epilepsy occur in ambulatory care. yet these reports are unstructured and typically noisy as they include a range of medical and administrative information, such as the patient's medication, other therapies, and details disclosed during previous clinic visits. moreover, the reports often do not include any information about seizure frequency and, if they do, the language is often imprecise so that the nature of the frequency is vague or unclear. these factors make the application of nlp to ehrs to research seizure frequency challenging. epilepsy affects about 1% of the general population @xcite . around 30% of people with epilepsy do not respond to asms and are therefore regarded as refractory to treatment @xcite . while there are more than 30 individual asms and a much larger number of possible combinations of asms taken together, it is not feasible to try them in every refractory patient. this underlines the importance of research in predicting which asms would have the greatest impact on epileptic seizures for individual patients. although there is some published research on applying pre-trained transformers to investigate epileptic seizure frequency among ehrs, the more recent opportunity of applying the new, generative llms for the same task is under-explored. however, it is expected that the application of generative llms to epilepsy research will increase significantly @xcite . the paucity of published research in this field may largely be due to the fact that these models are so new. the most extensive relevant research we found was a long-term study @xcite @xcite ) that used a different methodology from ours to extract seizure frequency information from ehrs. the university of pennsylvania researchers applied the pre-trained transformers bio_clinicalbert (for text classification), roberta (for text extraction), and a t-5 model (to summarize sentences with seizure frequency data) to free-text data in ehrs to determine the seizure frequency of a person with epilepsy or whether that person was seizure free. for seizure frequency, they framed the task as an extractive question-answering problem, asking the language model to identify statements that answered the question: "how often does the patient have seizures?" they then simplified each sentence into a standardized format, "x per y [day/month/year/visit]"; for example, "1 per 1 week". they subsequently manually annotated 1,000 sentences of seizure frequency generated by their models with the formatted summaries, then split them into training (700 sentences) and testing (300 sentences) datasets. finally, they fine-tuned a t5-large model using huggingface on the training dataset and made predictions on the test dataset. the researchers declared an "overall accuracy" score of 0.88 for seizure frequency, which comprised scores for each of "sentence accuracy", "summary accuracy", and "quantity accuracy". that study follows a large body of research applying pre-trained transformers to a wide variety of clinical tasks, such as predicting the risk of seizure recurrence among children with epilepsy @xcite , inferring cancer disease response from free-text radiology reports @xcite , or detecting dementia with inhospital clinical notes @xcite . two other studies used rules-based nlp approaches to identify seizure frequency in unstructured clinic 1 https://cogstack.org letters @xcite . our objective was to apply a new, generative llm to the task of determining seizure frequency from free-text data. llms are built on the architecture of the transformers but are much larger and more powerful language models. we were encouraged by recent research that demonstrates the benefits of using llms with clinical texts (for example, @xcite @xcite . our research, however, was restricted to using only an open-source language model because we used confidential nhs medical data that had to remain within the hospital's secure it network for regulatory reasons. therefore, we could not experiment with llms such as openai's chatgpt that are only available via an api to an off-site service. we found that meta's llama 2 @xcite performed best for our purposes within our limitations (see details in section 2.4). the llm was run on up to eight nvidia v100 gpus. ## results by comparison, llama 2 performed better than bert large and roberta large, although it must be noted that our testing methodology for llama 2 was different to that for the pre-trained transformers. llama 2 was mainly tested against the full annotated dataset of 3,000 ehrs (llama 2 does not require fine-tuning), whereas bert large and roberta large, which required 80% of the balanced annotated dataset of 1,720 ehrs as a training dataset, were tested against a much smaller test dataset of 172 ehrs (or 10% of 1,720). under this scenario, llama 2's model f1 score of 0.73 was higher than roberta large's 0.58 and bert large's 0.55 (the results of the pre-trained transformers are the average of three different runs with the same random states). moreover, llama 2 recorded a positive f1 score in all nine seizure frequency categories, whereas the pre-trained transformers both posted f1 scores of zero in at least four categories, suggesting llama 2 is better at identifying seizure frequency in the sparse categories. however, we also tested llama 2's performance on the same smaller test dataset of 172 ehrs used for bert large and roberta large. in this case, llama 2's model f1 score dropped to 0.54, broadly in line with the pre-trained transformers, and llama 2 recorded f1 scores of zero in three of the nine seizure frequency categories. there are two possible explanations for this apparent difference in performance. first, the small test dataset furthermore, other metrics demonstrate that even when evaluated on the small test dataset, llama 2 was more reliable than the other two models. llama 2 predicted that 59% of the ehrs in the test dataset contained either no, or vague, information about seizure frequency, which was higher than the annotators' 40% but lower than roberta large's 65% and bert large's 71%. also, while llama 2 always made a prediction on every ehr, roberta large failed to make a prediction on average on 17% of the test ehrs and bert large failed on 30%. it is difficult to compare the results of our study to those of @xcite @xcite because they provided an "overall accuracy" score of 0.88 for seizure frequency and did not break down accuracy for individual seizure frequency categories. however, in broad terms in appears our llama 2 methodology produced at least similar performance given its model accuracy was 0.94 and its accuracy rate for the infrequent category was 0.92 and for the frequent category 0.85. ## discussion while our initial aim to determine whether the llm could identify the frequency of seizures in unstructured outpatient reports for eight temporal categories proved too ambitious, when the temporal categories were reduced to frequent and infrequent the output of llama 2 was much improved. the key objective of our broader epilepsy project is to track the effects of different combinations of anti-seizure medications on seizure frequency in individual patients and consequent changes. in this respect, llama 2's f1 scores of 0.87 for the unknowns and 0.62 for frequent seizures is useful. although the model's f1 score was a lower 0.30 for infrequent seizures, we are mindful that the number of observations of frequent seizures is roughly three times that of infrequent seizures, as previously stated, and so while more work is required to optimize the model's output for infrequent seizures, its overall performance aids our broader objective. during experimentation it was clear that llama 2's pre-training on vast general corpora had imbued it with a noticeable degree of expert knowledge about epilepsy. this may be one reason why llama 2 proved superior to the pre-trained transformers in identifying seizure frequency in unstructured, free-text ehrs. another reason is that llama 2 is a much bigger language model -we used the 13b parameter version -than bert large with 336m parameters and roberta large with 356m parameters. llama 2, like other generative llms, has three advantages over pre-trained transformer language models. first, llama 2 does not have to be finetuned on an annotated dataset, which saves substantial time and resources by obviating annotations for a training dataset. second, llama 2 does not have a built-in maximum token length for processing long texts. third, llama 2 is 'guided' on a particular language task by prompt engineering, which typically takes less time than adjusting multiple hyperparameters to optimize the performance of a pre-trained transformer model. on the other hand, llama 2 has a distinct disadvantage: because of its large size, the llm requires a longer running time. in this case, llama 2 took on average 3.6 seconds to process one ehr, or about one hour for 1,000 ehrs. a drawback of this particular study, however, is that the results are not reproduceable by other researchers because the patient ehrs are confidential and can only be accessed via the hospital's secure it network. ## conclusion llama 2, as a popular llm widely regarded as producing impressive performance on a variety of nlp tasks, performed well on the specific clinical nlp task of identifying seizure frequency from unstructured, free-text ehrs. this demonstrates that the new, generative llms are useful for epilepsy research in particular and clinical nlp research in general. the key question for our broader epilepsy research project was whether a new, generative llm could identify seizure frequency among the ehrs to a sufficient degree to use the model's predictions as a basis for further research into different anti-epilepsy medications and their effects on seizure frequency. our conclusion is that llama 2 can.