LAIL ab at Chemotimelines 2024: Finetuning sequence-to-sequence language models for temporal relation extraction towards cancer patient undergoing chemotherapy treatment (2024)

Abstract: In this paper, we report our effort to tackle the challenge of extracting chemotimelines from EHR notes across a dataset of three cancer types. We focus on the two subtasks: 1) detection and classification of temporal relations given the annotated chemotherapy events and time expressions and 2) directly extracting patient chemotherapy timelines from EHR notes. We address both subtasks using Large Language Models. Our best-performing methods in both subtasks use Flan-T5, an instruction-tuned language model. Our proposed system achieves the highest average score in both subtasks. Our results underscore the effectiveness of finetuning general-domain large language models in domain-specific and unseen tasks.

## introduction patient health records contain a wealth of information that can offer valuable insights to healthcare professionals and researchers, aiding in the enhancement of diagnosis, treatment, and disease prevention. cancer patients often undergo lengthy treatment regimens, resulting in extensive electronic health record (ehr) documentation over time. the sheer volume of data available to healthcare providers is substantial, making manual curation impractical and cost-prohibitive. a crucial aspect of cancer patient records is their chemotherapy treatment status documentation. automatically extracting information regarding the timelines of chemotherapy events offers several advantages, including the ability to evaluate treatment efficacy across various cancer types. this automated extraction process also facilitates the creation of concise summaries for future attending physicians. two main tasks have been defined and addressed in association with temporal relation extraction from clinical notes: doctimerel and tlink detection and classification. the first task is to iden-tify and classify the relation between events in an ehr note and the creation time of the document. tlink detection and classification identify relations between event mentions and time expressions in ehr notes. in this paper, we deal with the latter, the temporal relation extraction on a dataset of three cancer types. the shared task defines two subtasks. subtask one aims at discerning a temporal relationship between a pair, consisting of a chemotherapy event and a time expression, subsequently classifying this relationship into one of the following categories: contains, begins-on, or ends-on. in the second subtask, the only given input is the patient notes. the desired output for both subtasks is patient-level chemotherapy timelines. for detailed information on the definition of the subtasks, baseline methods, dataset, and evaluation criteria, see @xcite . we approach both subtasks using large language models (llms). for the first subtask, we reformulate the relation classification problem into a text generation task and benefit from instruction-tuned language models to predict the relation. in the second subtask, we experiment with a sequence-tosequence fine-tuning method with relations transformed into target sequences using a triplet linearization algorithm and also a pipeline method consisting of a rule-based event and time expression module and our best-performing model on the first subtask to identify and classify the pairwise relations. we achieved the highest average scores on the test results as announced by the organizers @xcite . in the following, we describe how we have utilized llms in detection, classification, and the end-to-end approach to chemotherapy timeline extraction from clinical notes. 2 methodology 2.1 subtask1 with the chemotherapy events and time expressions in each patient's note already provided by the organizer, the first subtask aims to identify temporal relations between them and subsequently generate patient-level timelines. prior to training a model, we need to prepare the dataset to train the model for the temporal relation classification task. the annotated relations with their respective pair of events and time expressions in the gold standard training/development dataset are added as positive instances. we create negative instances labeled as no-relation by pairing events and time expressions within a patient note with no temporal relations in the gold-standard dataset. however, incorporating every potential negative instance would lead to a significant imbalance in the training dataset as well as additional computational costs for training and inferencing the model. to mitigate this, we exclude instances where the positional distance between the event mentions and time expressions in the ehr note exceeds a maximum number of characters. during preprocessing, we first employ the "mimic" model from the stanza library, developed by @xcite , for sentence segmentation. then, we construct the context for the input sequences using two different approaches: concatenated context and bounded context. if the event and the time expression in the pair occur within the same sentence, both methods consider the sentence as the context. otherwise, if the event and time expression are located in different sentences, the two sentences are concatenated to form the concatenated context. in the bounded context method, any sentence occurring between these two sentences is also included in the context. in addition, we add markers denoted @xmath0, @xmath1. we reformulate the temporal relation classification task as a generation task by finetuning a large language model to directly generate a label from the predefined set of relation types: contains, begins-on, ends-on, no-relation. we prepend the instruction describing the task to each input context. this method conditions the model to generate the relation type immediately following an anchor prompt "relation:". figure 1 illustrates our approach to tackling the first subtask, including our model's input and expected output. in the instruction, we use the definition of events, time expressions, and temporal relations provided in the data descriptions of the shared task. our instruction format leverages the prompt structure used in relation extraction tasks, as described by @xcite . we also experiment with finetuning the model without adding the task instruction to the input contexts. during our preliminary experiments, we fine-split cancer type gold relation pairs# max character distance no relation pairs # train brca 455 99 381 mela 48 218 35 ovca 494 143 336 dev brca 113 213 132 mela 201 144 191 ovca 226 173 220 ## results we use the evaluation system provided by @xcite for both subtasks on the development set. 7 the evaluation script receives the gold standard timelines, and the system prediction for all patients in each cancer type as input. all the experiments on the development set have been executed before the test set results were announced. ## error analysis since the gold standard timeline and annotations on the test set have not been released to enable future editions of the task, we will provide the error analysis on the results of the development set. ## related work numerous studies focus on annotation @xcite @xcite , detection and classification @xcite of temporal relations in the general domain. in the medical domain, temporal relation extraction also received attention for its benefits in longitudinal studies of medication, treatments, and diseases, as well as in summarizing clinical notes for physicians' further reference. thyme annotation guidelines and corpus @xcite and its extension (wright-bettner et al., 2020) is a considerable effort in the specification of process of temporal relation annotation process in clinical narratives based on iso-timeml @xcite . prior to the introduction of transformer-based language models a few studies approached various tasks of temporal relation extraction problem with feature-based supervised machine learning algorithms and sequential neural networks @xcite @xcite . moreover, @xcite utilized unlabeled data by self-training neural networks in clinical temporal relation extraction. after the rise of transformer-based models, temporal relation extraction from clinical notes also benefited from this significant development in nlp methods using models such as bert @xcite , biobert and bart @xcite for clinical text representation. @xcite continue training bert using a masking method called entity-centric masking strategy, where they use the mimic iii dataset as their training data. their results on temporal relation extraction shows improvements on baselines using the model pretrained using this approach. most end-to-end systems for temporal relation extraction in the clinical domain have been tackled using a pipeline approach consisting of modules for event and time expression extraction and pairwise temporal relation detection and classification. @xcite on the other hand, explore the use of sequence-to-sequence models in extracting temporal relations from text. they experiment with various input/output representations and adopt those representations, which enable the reconstruction of the snippets with several relations and repetitive event names in a text snippet. they report this approach's results utilizing different sequence-tosequence llms such as bart and t5. @xcite approach temporal relation extraction problem as an end-to-end task without given events and time expressions using a combination of domainspecific pre-trained language model pubmedbert @xcite and a multi-headed attention classifier on thyme2 dataset @xcite . @xcite organized previous shared tasks to incentivize the research on temporal relation extraction from clinical notes. ## conclusions this paper presents our effort in participating in the chemotimeline shared task. we apply an instruction finetuning method for temporal relation detection and classification and a sequence-to-sequence approach for extracting timelines directly from ehr notes to solve the first and second subtasks. our approach, leveraging the power of generaldomain large language models and further finetuning them with parameter-efficient methods, secured the highest average scores across the different cancer types for both subtasks. the results of our approach using flan-t5-xxl + lora underscore the potential of instruction finetuning in enhancing the capabilities of llms for unseen natural language understanding and generation tasks, even on domain-specific data. in future work, we aim at augmenting the data for low-frequency relation types and also harnessing the power of provided unlabeled data to continue pre-training large language models and to investigate the effect on the results of extracting temporal relations from cancer patient ehr notes.