NYUL angone at Chemotimelines 2024: Utilizing Open-Weights Large Language Models for Chemotherapy Event Extraction (2024)

Abstract: The extraction of chemotherapy treatment timelines from clinical narratives poses significant challenges due to the complexity of medical language and patient-specific treatment regimens. This paper describes the NYULangone team’s approach to Subtask 2 of the Chemotimelines 2024 shared task, focusing on leveraging a locally hosted Large Language Model (LLM), Mixtral 8x7B (Mistral AI, France), to interpret and extract relevant events from clinical notes without relying on domain-specific training data. Despite facing challenges due to the task’s complexity and the current capacity of open-source AI, our methodology highlights the future potential of local foundational LLMs in specialized domains like biomedical data processing.

## introduction the extraction of structured information from unstructured clinical narratives is a crucial task in healthcare informatics, enabling better patient care and clinical decision-making. the chemotimelines 2024 shared task focuses on extracting chemotherapy treatment timelines from clinical narratives, a challenging task for understanding oncology patients' treatment paths. our team, nyulangone, participated in subtask 2, aiming to leverage the general reasoning capabilities of large language models (llms) for this purpose. ## related work clinical narrative processing traditionally relies on rule-based systems or machine learning models trained on domain-specific annotated data. recent advances in nlp have seen the rise of transformer-based models and llms, offering powerful general-purpose language understanding capabilities. however, their application in domainspecific tasks like chemotherapy timeline extraction remains in the infancy of exploration. ## system description our system builds upon a locally deployed instance of mixtral, an open-weights llm. the system comprises two rounds of text inference: the first round is an extraction of chemotherapy events from individual notes, and the second round is the aggregation of events from multiple notes to a single timeline. algorithm 1 patient chemotherapy summary algorithm 1: for each patient do 2: for each note of the patient do 3: prompt mixtral to read the note and extract chemotherapies 4: end for 5: end for 6: prompt mixtral to combine the extracted chemotherapies from every note to create a patient-level summary of all chemotherapies ## results on the dev set, our system achieved an average f1 score of 0.35. on the validation set, our system achieved an average f1 score of 0.23 across different cancer types, as shown in ## discussion while our performance was well below the baseline and leading teams, it provided valuable insights into the challenges and potential of using locally hosted llms in clinical nlp tasks without domainspecific training. the opaque inner workings of llms preclude an exact understanding of why certain chemotherapy events are more easily extracted than others. the errors our system demonstrates could largely be grouped into several types: despite the objectively poor performance, our results highlight a future potential for llms to be used in biomedical nlp tasks. local llms that can competently perform general reasoning are still a new technology, with expert opinion suggesting that local models like mixtral currently perform at a gpt-3 (openai, united states) level of performance. ## conclusion and future work our exploration into using local llms for chemotherapy treatment timelines extraction offers a starting point for further research in this area. future work will focus on enhancing model understanding of clinical contexts through retrieval augmented generation (rag) and ensemble prompting techniques such as "tree of thought."