UTSA - NLP at C hemo T imelines 2024: Evaluating Instruction-Tuned Language Models for Temporal Relation Extraction (2024)

Abstract: This paper presents our approach for the 2024 ChemoTimelines shared task. Specifically, we explored using Large Language Models (LLMs) for temporal relation extraction. We evaluate multiple model variations based on how the training data is used. For instance, we transform the task into a question-answering problem and use QA pairs to extract chemo-related events and their temporal relations. Next, we add all the documents to each question-answer pair as examples in our training dataset. Finally, we explore adding unlabeled data for continued pretraining. Each addition is done iteratively. Our results show that adding the document helps, but unlabeled data does not yield performance improvements, possibly because we used only 1% of the available data. Moreover, we find that instruction-tuned models still substantially underperform more traditional systems (e.g., EntityBERT).

## introduction extracting chemotherapy treatment timelines from clinical notes is crucial in clinical natural language processing (clinicalnlp) for enhancing patient care and advancing cancer research @xcite . researchers can construct detailed treatment timelines within electronic health records (ehr) across various medical domains by identifying and extracting events related to chemotherapy treatments and their temporal information from medical documents. this work aims to develop an end-to-end system utilizing large language models (llms) in a question-answer format for chemotherapy timeline extraction. such a system will aid healthcare professionals in comprehending patient histories, thereby improving clinical textmining efforts and assisting physicians in making more informed care decisions. additionally, it will contribute to research in personalized cancer treatment development. the main approach in clinical entity and relation extraction tasks heavily relies on pre-trained, domain-specific models like biobert @xcite , clinicalbert @xcite , pub-medbert @xcite , and entitybert @xcite . these models are trained on a broad range of biomedical corpora, like pubmed articles and clinical notes, to understand the complex language of the clinical domain, which is often succinct and laden with phrases, jargon, non-standard expressions, abbreviations, assumptions, and specialized knowledge. these models are then adapted or fine-tuned for specific tasks such as named entity recognition (ner), relation extraction @xcite , and event extraction (ee), which often employ strategies like multi-task learning (mtl) and an all-inone scheme to enhance performance across multiple tasks by leveraging shared knowledge and representations @xcite . however, there are still challenges, such as a drop in performance when these models are used for out-of-domain tasks or very different sub-domains in terms of context and terminology, revealing their limitations in adaptability @xcite . recently, large language models (llms) have shown remarkable potential in natural language processing (nlp) tasks, including text generation, reasoning, text classification, summarization, and question answering, through their ability for zeroshot or few-shot learning @xcite . this capability allows them to adapt to new tasks quickly with minimal fine-tuning. this adaptability has resulted in their outstanding application performance, including ner and re within the general domain. models like cot-er @xcite , gpt-re @xcite , and promptner @xcite show that through few-shot learning or zero-shot learning, these generative llms can achieve performance levels competitive with the state-of-the-art methods in entity or relation extraction @xcite @xcite . this achievement is primarily due to their capability to use task-specific and concept-level knowledge stored during pre-training, which is then effectively leveraged through prompting to generate relevant evidence for the tasks. however, challenges arise when adapting llms from the general to the medical domain, primarily due to their lack of domain-specific knowledge and the difficulty in incorporating new, relevant factual information over time @xcite @xcite . while llms have shown potential in biomedical natural language processing tasks through innovative in-context learning strategies, their application in specific tasks like ner and re remains problematic. this is partly because current few-shot learning methods, which trained on large amounts of source data and fine-tuning on exemplars from the target domain, do not perform well in the medical context @xcite @xcite . the discrepancy arises from significant differences in entity and relationship definitions between general and medical texts @xcite . to address these challenges, researchers have explored various approaches, including the development of domainspecific generative llms like biogpt @xcite , biomedlm @xcite , and bio-bart @xcite . these models are trained from scratch using medical corpora such as pubmed or are continually fine-tuned on medical data. basically, fine-tuning is required for adequate performance on biomedical nlp tasks. these efforts represent steps toward bridging the gap in domain adaptation for llms. however, updating these models for the rapidly changing medical field is still non-trivial due to the risk of catastrophic forgetting during fine-tuning @xcite , highlighting the need for better training methods tailored to medical knowledge. to address this, we explored instruction-tuning methods for large language models, focusing on an open-source language model. traditional finetuning methods risk forgetting previous knowledge, so we adopted a novel training strategy, gradually extending training to include associated documents and unlabeled datasets. initially, we instructiontuned on question-answer (qa) pairs before integrating complete ehr documents. then, we trained on qa pairs and documents simultaneously. finally, we continue pre-training on the large un-labeled corpus. @xcite demonstrates that this integration strategy ensures the retention of acquired knowledge. in the inference stage, our system directly generates output relations from input questions for subtask 1. for subtask 2, we first extract event entities and time expressions before predicting relationships between identified entities using different input questions. our approach provides an end-to-end relation extraction system for extracting chemotherapy treatment timelines. this system formulates the task as a text generation task, using clinical notes as input to generate relational triplets end-to-end, without requiring additional intermediate annotations, as seen in the rebel method @xcite . in summary, this paper makes the following contributions: ## methods in this section, we describe our instruction-tuned llms strategy. figure 1 shows a high-level overview of our approach. we convert the information extraction task to the question-answer instruction format. our strategy has three main components: 1) instruction-tuning llms on taskspecific qa pairs (i.e., named entity recognition (ner) and relation classification (re)); 2) joint first, we perform instruction-tuning on llms using task-specific qa pairs (e.g., ner and re). second, we conduct further instruction-tuning on qa pairs and associated documents to enhance its ability to progressively absorb knowledge from simpler to more complex data. finally, we continue pre-training the model on an unlabeled corpus to refine its capabilities in the clinical domain further. ## experiments in this section, we provide a brief overview of the dataset, discuss the evaluation metrics, discuss our results on the validation dataset, and briefly mention the final model performance in the competition on the test set. ## related work continual knowledge acquisition. in continual knowledge acquisition, several studies have investigated the ability of language models (lms) to retain and update knowledge over time. @xcite and @xcite explore the effectiveness of different pre-training approaches using smaller lms like bart @xcite and entitybert @xcite . zhu and li (2023); @xcite delve into fine-tuning lms on qa pairs related to individuals, with a focus on mixed training settings combining biographies and qa pairs. these studies are a foundation for exploring strategies to incorporate qa data before continued pre-training. additionally, researchers have sought to adapt lms to specialized domains, such as medicine, with @xcite ; @xcite ; @xcite proposing various strategies. however, a common challenge in continual knowledge acquisition is the potential for inaccuracies or difficulties in clinical nlp tasks. models like biogpt @xcite , biomedlm @xcite , and biobart @xcite address these concerns by continuing training specifically within the medical domain. instruction fine-tuning. recently, instruction tuning, also known as supervised fine-tuning, has gained prominence for its ability to draw out knowledge from large language models (llms) using high-quality annotated data or data from proprietary models @xcite @xcite . this process enhances llms' capacity to address user inquiries and improves their factual accuracy, a focal point of our research. additionally, the zero-shot and few-shot in-context learning capabilities of llms, which operate with minimal or no training data, present a significant advantage for efficient learning. these approaches, further discussed by @xcite and highlighted in the works of @xcite and @xcite , underscore the potential of instruction tuning in refining llms' factuality and responsiveness. ## limitation due to the constrained timeline and limited resources of the shared task, our exploration was restricted to basic setups. we did not create negative examples for ner qa pairs and only used a limited set of negative examples for re qa pairs by randomly selecting three unrelated pairs of chemotherapy events and time expressions. additionally, our limited use of just 1% of the unlabeled dataset resulted in decreased performance across all three disease types, suggesting that we didn't fully exploit the continuous training capabilities. furthermore, our experiments only considered entities within the same sentence, overlooking cases where entities span multiple sentences in the chemotimelines dataset. this oversight could significantly impact model performance evaluation. ner and re tasks are sensitive to prompt design, and our initial single prompt strategy may not have been optimal. more comprehensive training and experiments, including ablation tests, will be necessary to evaluate and enhance our system's performance and efficiency thoroughly. ## conclusion and future work this paper presents our end-to-end system for extracting chemotherapy treatment timelines from the clinical nlp chemotimelines share the task. we explored various instruction tuning strategies for open-source generative llms, providing a starting point for developing ner and re models in the medical domain. our future work will explore the implementation of outcome supervision and process-based reward mechanisms in reinforcement learning training to address the issue of false positive predictions @xcite .