Lexicans at Chemotimelines 2024: Chemotimeline Chronicles - Leveraging Large Language Models ( LLM s) for Temporal Relations Extraction in Oncological Electronic Health Records (2024)

Abstract: Automatic generation of chemotherapy treatment timelines from electronic health records (EHRs) notes not only streamlines clinical workflows but also promotes better coordination and improvements in cancer treatment and quality of care. This paper describes the submission to the Chemotimelines 2024 shared task that aims to automatically build a chemotherapy treatment timeline for each patient using their complete set of EHR notes, spanning various sources such as primary care provider, oncology, discharge summaries, emergency department, pathology, radiology, and more. We report results from two large language models (LLMs), namely Llama 2 and Mistral 7B, applied to the shared task data using zero-shot prompting.

## introduction electronic health records (ehrs) are a rich repository of patient information, encompassing a wide array of formats and sources including physician notes, laboratory results, radiology images, and pathology reports. due to the heterogeneous and unstructured nature of clinical data, it is cumbersome to visualize patient journeys or extract meaningful information from electronic health records (ehrs) to help guide clinical decision making @xcite . ehr data is often dispersed, recorded in free text with substantial variability in terminology, and embedded in narrative formats that are not easy to process or normalize across healthcare settings and systems. in addition, privacy concerns further limit the use of clinical data across hospitals and geographical borders further compounding complexity @xcite @xcite and difficulty to leverage ehr data for insights generation. large language models (llms), with their advanced natural language @xcite @xcite understanding capabilities, offer a transformative solution to these challenges. they can be trained to interpret complex language found in ehrs, extracting relevant clinical events and concepts, and mapping these onto a coherent information or treatment timelines which can be difficult to realize manually by humans. llms are appropriate for handling the variability and ambiguity that arise in medical documentation, enabling them to identify and organize critical information such as chemotherapy treatments, such as drug names, dosages, administration dates, and associated clinical outcomes @xcite . moreover, by leveraging the latest advancements in transfer learning and domain-specific finetuning, llms can be programmed in such a way to understand the specific lexicon and data structures unique to domains as complex as oncology and chemotherapy treatment regimes @xcite . all in all, this can help with the generation of comprehensive, accurate, and personalized chemotherapy treatment timelines that are an essential component for advancing precision oncology, and also supporting the development and assessment of patient-centric therapeutic strategies @xcite . to better understand the impact of various factors on tumor behavior and responsiveness, particularly in the context of precision oncology, the chemotimelines 2024 shared tasks has been proposed @xcite . in this work, we describe our submission to subtask 1, which aims to build timelines of chemotherapy treatments for individual patients using their electronic health records (ehr) notes. we achieved a 5th place ranking in subtask 1, with an averaged accuracy across breast, ovarian, and melanoma indications. the contributions of our paper can be outlined as follows: ## related work in recent years, there has been a growing body of research demonstrating the effectiveness of large language models (llms) in comprehending medical text data and extracting valuable insights from electronic health records (ehrs) across various clinical domains @xcite @xcite . prior investigations have shown the application of natural language processing (nlp) in healthcare, encompassing tasks such as clinical text classification, medical entity recognition, and patient risk prediction. efforts to construct clinical timelines from ehr data have predominantly focused on structured data such as procedure codes, diagnosis codes, and laboratory results @xcite . within oncology, nlp methodologies have been employed to analyze cancer-related textual data, including pathology reports, clinical notes, and research articles @xcite . researchers have investigated the utility of nlp in extracting treatment regimens, identifying adverse drug events, and predicting treatment outcomes among cancer patients @xcite . techniques for temporal event extraction and sequence modeling have been explored extensively to develop patient timelines for disease progression tracking and treatment monitoring @xcite . temporal reasoning techniques have found applications in healthcare to analyze the temporal associations between clinical events, treatments, and patient outcomes @xcite . studies have explored temporal logic, temporal abstraction, and probabilistic models to represent and analyze temporal data in healthcare contexts @xcite . transformer based large language models have demonstrated remarkable performance improvements across various nlp benchmarks @xcite . furthermore, healthcare-specific models @xcite have exhibited state-of-the-art accuracy in biomedical entity recognition @xcite and relation extraction @xcite . the current state of the art lies in several notable large language models (llms), each featuring distinct model architectures and sizes @xcite . prominent examples include llama 2 ( touvron et al., 2023 3 task and dataset details 3.1 task details chemotimelines 2024 at naacl-clinicalnlp workshop is a shared task @xcite that focuses on building a timeline of chemotherapy treatment for each patient given all the available electronic health records (ehrs) notes of that patient. the shared task has 2 subtasks. subtask 1 involves using provided gold annotations of chemotherapy events (events) and time @xcite ## approach we aimed to significantly contribute to the development of advanced cutting-edge methodologies and techniques for automatically constructing chemotherapy treatment timelines from electronic health records (ehrs) clinical notes of individual patients. we leveraged current state of the art large language models (llms) for this shared task. we tested various llms with different sizes and archi-tectures to determine which model works best for relation extraction (see figure 1 ). ## results and discussion as previously stated, our study utilizes the two large language models, llama 2 @xcite and mistral 7b @xcite . we utilized the default parameters for both llama 2 and mistral 7b, except for the chunk size, which was set to 256, temperature set to 0.1, and seed set to 123 (refer to error analysis error analysis in large language models (llms) involves scrutinizing the model's prediction errors to discern their types, frequency, and underlying causes. this entails evaluating the model's performance on a test dataset and categorizing errors into various types, including false positives, false negatives, ambiguous cases, outof-distribution errors, and conceptual errors. by analyzing these errors, insights can be gleaned regarding patterns and areas for improvement in the model. this analysis guides strategies for enhancing the model's performance through fine-tuning, refining training data, and optimizing input representations. furthermore, error analysis is crucial for establishing confidence in the model's predictions and comprehending its limitations in realworld scenarios. figure 3 shows the error analysis presented compares llama 2 and mistral for the baseline established by the organizers, as well as prediction ## conclusion and future work in this paper, we present our submission to the chemotimelines 2024 shared tasks @xcite to build chemotherapy treatment timelines using electronic health records (ehrs) notes from various sources, such as primary care providers, oncology departments, discharge summaries, emergency department, pathology, radiology, and more. we used zero shot prompted relation extraction @xcite driven by the thyme ontology @xcite and temporal relation annotations @xcite . we evaluated pre-trained large language models (llms) like llama 2 @xcite @xcite , meditron-70b @xcite , and mixtral of experts @xcite with different sizes and architectures. we only reported results on llama 2 @xcite and mistral 7b @xcite . we conducted a series of experiments with different setups to improve the system's performance. from our analysis, we conclude that our approach helped us determine which model works best for this shared task. we 8 conclude that llms provide a promising path forward for extracting timelines that contextualize cancer treatment, which were previously unavailable. we also show that our model provides high recall that is beneficial for instances where high sensitivity is required such as with output-sensitive predictions like cancer prediction models. however, due to the auto-generation approach and minimizing human intervention, the models we developed demonstrated relative low precision. precision was evaluated with two physicians validating the accuracy of the generated chemotherapy timelines. our proposed methodology represents a significant advancement in the field, providing a flexible and efficient solution for relation extraction tasks in natural language processing. large language models (llms) offer a promising approach for auto-generating chemotimelines from electronic health records (ehrs) due to their advanced natural language understanding capabilities, contextual understanding, and semantic representation of medical information. llms can comprehend complex medical texts, capture contextual relationships between different medical events, and generate rich semantic representations of medical concepts and events mentioned in ehrs. as we see in our current study, our effort to attempt fully auto-generated chemotherapy timelines have shown great promise in terms of recall but have a negative impact on precision. in future studies we will explore further training rounds or human-in-the-loop models to explore the right balance between automation and human guided outputs. nevertheless, our study demonstrates great promise in integrated llmgenerated chemotherapy timelines that have the potential to alleviate documentation and data harmonization burdens, potentially easing clinician workload and enhancing quality of patient care. exploring the potential of llms is an emerging area in research. we have experimented with two state-of-the-art llms (llama 2 and mistral 7b) for this task, comparing each with the gold standard for various cancer types. our approach was to maintain a broad, domain-agnostic perspective, treating it as a high-level nlp relation detection task. we assumed that the underlying llms were generalpurpose. in the future, we aim to explore domainspecific llms tailored for biomedical texts, such as jsl-medmnx-7b (jsl-med-sft-llama-3-8b, 2024), which could offer improved accuracy by better handling specialized language and data struc-tures inherent in this domain. furthermore, we aim to validate the effectiveness of our llm-based system across diverse healthcare datasets to enhance its performance. additionally, we intend to conduct comprehensive analysis of the generated chemotherapy timelines to fine-tune them further and improve precision. this includes conducting in-depth error analyses to pinpoint the root causes of false positives. our goal is to identify any consistent patterns, words, or phrases that the model may misinterpret, facilitating targeted improvements to enhance its accuracy.