Team NLP eers at Chemotimelines 2024: Evaluation of two timeline extraction methods, can generative LLM do it all or is smaller model fine-tuning still relevant ? (2024)

Abstract: This paper presents our two deep learning-based approaches to participate in subtask 1 of the Chemotimelines 2024 Shared task. The first uses a fine-tuning strategy on a relatively small general domain Masked Language Model (MLM) model, with additional normalization steps obtained using a simple Large Language Model (LLM) prompting technique. The second is an LLM-based approach combining advanced automated prompt search with few-shot in-context learning using the DSPy framework.Our results confirm the continued relevance of the smaller MLM fine-tuned model. It also suggests that the automated few-shot LLM approach can perform close to the fine-tuning-based method without extra LLM normalization and be advantageous under scarce data access conditions. We finally hint at the possibility to choose between lower training examples or lower computing resources requirements when considering both methods.

## introduction the advent of auto-regressive large language models (llms) has taken the nlp field by storm and has been diffusing to more specialized domains, such as clinical nlp ever since. while the most powerful models are still only available as private owned services -oftentimes precluding their use with sensitive medical data -open source and open weight models have been catching up, mostly since the release of the llama model family @xcite . with such open models, in-context learning strategies became more viable. on top of those llms, an ecosystem of tools and frameworks has also emerged to provide more robust and efficient ways to use them. one such framework is dspy @xcite , whose ambition is to provide a principled and automated way to search llm prompts and weights and ultimately build robust llm pipelines. while this latter technology still evolves, older and more established deep learning models coexist, and the comparative advantages of the two approaches are being assessed. a prominent example of those predecessors is bert-based models, which can still be considered llms, although they are usually an order of magnitudes smaller than their auto-regressive counterparts. with such masked language models (mlms), fine-tuning of the model weights can be more easily performed due to their usually smaller size. temporal relation extraction (tre) is a crucial task for several domains, particularly the clinical domain, requiring a deep understanding of natural language. with the rise of llms, recent research efforts attempt to apply these models to the tre task, but results are still debatable @xcite @xcite . in this paper, we address the clinical event-to-time expression relation extraction task by evaluating two timeline extraction methods. the main contributions of this paper are: ## task description and data we participated in the first subtask of the chemotherapy treatment timelines extraction shared task foot_0 @xcite , which aims to extract temporal relations between chemotherapy events and time expressions and then produce the final patient-level timelines by resolving duplica-tion and conflicts in the pairwise temporal relations. the types of relations to extract are mainly con-tains, begins-on, and ends-on. the data provided by the university of pittsburgh/umpc, through a data use agreement during this shared task, includes a list of available de-identified electronic health record (ehr) notes for patients with breast, ovarian, and melanoma cancer. further details about the subtask and the data distribution are described in @xcite . the organizers provide a baseline system based on the entity-bert model @xcite . ## methods this section describes our proposed methods for the tre task and the post-processing, normalization, and summarization steps used to construct patient-level timelines. ## evaluation metrics for the final evaluation of patient timelines, the organizers provide an evaluation code foot_3 . the evaluation process covers strict and relaxed evaluation settings by calculating the average f1 score across all patients. the official score is an arithmetic mean of two types of macro f1 measure, type a and type b, in a relaxed to-month setting. the type a evaluation includes the patients with no gold timelines, while the type b evaluation excludes the patients with no true relations. the relaxed to-month setting means only the month must match the gold annotation. more details about the evaluation process are presented in the shared task website foot_4 . while selecting the different models we tried, we evaluated them based on the official score provided by the organizers. for the optimization of our automated few-shot llm approach and to ensure quality few-shot examples and demos, we defined a strict f1 measure. indeed, the dspy optimizer will only keep the fewshot examples that maximize this evaluation metric. note that for our first configuration setting, i.e., predicting an ordered list of (event, relation, time), the system prediction will not be considered a match if the model correctly predicts the relation type but fails to output the required format. ## results & discussion to participate in this shared task, we submitted two runs. the first run is the mlm fine-tuning approach (nlpeers 1), and the second run is the automated few-shot prompting llm approach (nlpeers 2). in this section, we begin by discussing the overall performance of our systems on the test and the development sets. since the gold annotations of the test set will not be released, we then present a more in-depth review of each of our methods on this set and the impact of adding the llm-based normalization on performance. ## conclusion in this work, we showed that an llm-based automated prompting method could, with no weight fine-tuning, give good results on a temporal relation extraction task. we also showed that a smaller fine-tuned mlm likely performs better while requiring less computing resources, thus confirming that smaller model fine-tuning is still relevant for such classification tasks. given the low number of examples retained at the end of the selection procedure by the few-shot prompting approach, it can be inferred that a smaller set of examples could be used to reach better performances, effectively making it an interesting choice when access to annotated data is scarce. another finding demonstrated the effective use of a simple llm approach for a general domain task such as time normalization.