Overview of the 2024 Shared Task on Chemotherapy Treatment Timeline Extraction (2024)

Abstract: The 2024 Shared Task on Chemotherapy Treatment Timeline Extraction aims to advance the state of the art of clinical event timeline extraction from the Electronic Health Records (EHRs). Specifically, this edition focuses on chemotherapy event timelines from EHRs of patients with breast, ovarian and skin cancers. These patient-level timelines present a novel challenge which involves tasks such as the extraction of relevant events, time expressions and temporal relations from each document and then summarizing over the documents. De-identified EHRs for 57,530 patients with breast and ovarian cancer spanning 2004-2020, and approximately 15,946 patients with melanoma spanning 2010-2020 were made available to participants after executing a Data Use Agreement. A subset of patients is annotated for gold entities, time expressions, temporal relations and patient-level timelines. The rest is considered unlabeled data. In Subtask1, gold chemotherapy event mentions and time expressions are provided (along with the EHR notes). Participants are asked to build the patient-level timelines using gold annotations as input. Thus, the subtask seeks to explore the topics of temporal relations extraction and timeline creation if event and time expression input is perfect. In Subtask2, which is the realistic real-world setting, only EHR notes are provided. Thus, the subtask aims at developing an end-to-end system for chemotherapy treatment timeline extraction from patient’s EHR notes. There were 18 submissions for Subtask 1 and 9 submissions for Subtask 2. The organizers provided a baseline system. The teams employed a variety of methods including Logistic Regression, TF-IDF, n-grams, transformer models, zero-shot prompting with Large Language Models (LLMs), and instruction tuning. The gap in performance between prompting LLMs and finetuning smaller-sized LMs indicates that for a challenging task such as patient-level chemotherapy timeline extraction, more sophisticated LLMs or prompting techniques are necessary in order to achieve optimal results as finetuing smaller-sized LMs outperforms by a wide margin.

## introduction cancer treatment is rarely simple. complex protocols involving multiple drugs, given over extended period of times in specified orders, are the norm @xcite . this poses a challenge for clinical researchers. ideally, real-world studies of the impact of specific protocols would require to identify which patients have been given which protocols. in practice, this task is complicated by a dearth of detailed information: although medication records and clinical notes might indicate the administration of a given chemotherapeutic agent to a patient, they rarely, if ever, name specific protocols. furthermore, structured medication administration records are insufficient, as clinical notes may contain mentions of medications in the context of reasons for discontinuing treatment, prior treatments given at differing institutions, or reactions to treatment. extracting chemotherapy timelines from clinical notes involves a series of challenges. individual mentions of relevant drug administrations (chemotherapy events) must be extracted and mapped to appropriate medication terminologies. each event must then be assigned a time extent, based on the date of the note and any temporal modifiers and indicators (e.g. time expressions) identified alongside the medication event @xcite . finally, these individual instances must be ordered into a timeline. each of these tasks involves substantial challenges, several of which have been the focus of previous semeval challenges @xcite @xcite . ## description of the shared task and subtasks the overall goal of the task was to create patientlevel timelines of chemotherapy treatment events from all the notes in the ehr available for a given patient. in general, timelines can be represented in different formats. we can describe a patient's treatment timeline in natural language, such as "2 cycles carboplatin and taxol, 9/30/13, 10/20/13" which is easy to understand by humans, however, it cannot be "understood" directly by machines. over the years, the research community has developed a parsimonious set of relations to express temporality between two events or between an event and temporal/time expression @xcite . we adopt these conventions where an event is any occurrence that can be positioned on a timeline (in our case chemotherapy events) and the set of temporal relations are defined as before, contains (with inverse contains-1 which is the equivalent of contained-by), overlap, noted-on, begins-on, ends-on. we limit events to only chemotherapy treatment events. therefore, for the shared task we represent the chemotherapy treatment timelines in a computable format as a list @xmath0, temporal_relation, @xmath1. thus, the previous ex- with this representation, the construction of chemotherapy treatment timelines can be naturally decomposed into the following stages: chemotherapy event extraction, time expression extraction, temporal relation classification, time expression normalization and patient-level timeline refinement. time expressions are also referred to as temporal expressions and timex3. the shared task defined two subtasks. in sub-task1, gold chemotherapy event mentions and time expressions are provided (along with the ehr notes). participants were asked to build the patientlevel timelines using gold annotations as input. thus, the subtask sought to explore the topics of temporal relation extraction and timeline creation if event and time expression input is perfect. in sub-task2, which is the realistic real-world setting, only ehr notes are provided. thus, the subtask aimed at developing an end-to-end system for chemotherapy treatment timeline extraction from patient's ehr notes. figure 1 is an overview of this task. ## evaluation we used the standard f1 metric to evaluate system performance, with variations to reflect the real world use case of chemotherapy treatment timelines. in consultation with our oncology domain experts it was determined that the level of granularity most useful for both point of care and translational studies is the month and the year for the chemotherapy treatment; the exact date was not deemed critical. therefore, we designed four evaluation strategies with different levels of granularity: strict, relaxed-to-day, relaxed-to-month and relaxed-toyear. strict evaluation requires all elements in a triplet to match the corresponding ones in the gold annotations to count as a match. in all relaxed evaluations, we consider certain temporal relations interchangeable, and only compare the predicted month (relaxed-to-month) or year (relaxed-to-year) with the gold ones. for instance, under relaxedto-month evaluation, we @xmath2, begins-on, 2013-02> correct if the gold timeline @xmath3, begins-on, 2013-02-13>. in this shared task, based on our consultations with our oncology domain experts as described above we use the relaxedto-month metric as the official score for the leader board and rankings. our scoring metrics account for differences in patterns of chemotherapy treatments. most, but not all patients have chemotherapy. some melanoma patients, for example, are treated surgical with no chemotherapy. to handle these differences, we used two types of scores based on relaxed-to-month results as motivated above: type a score aims to catch false positives for these patients. type b score measures the effec- ‡ https://www.globus.org tiveness of the methods on patients with confirmed chemotherapy treatments. the f1 score for each patient was computed and the final f1 score for each type is the average across all patients. the official score used for the rankings in the leader board is the average of type a and type b. a link to the evaluation script § is posted on the shared task website. teams uploaded their systems output into their gated globus collection and the organizers ran the evaluation script to produce the results posted on the leader board on the shared task website. each team was allowed to upload up to three submissions for each task. ## baseline systems the shared task organizers provide baseline results for subtask1 and subtask2. for both subtasks we used apache ctakes ¶ (savova et al., 2010) for sentence boundary detection, tokenization, and pipelining of software components via the python bridge to java (ctakes-pbj) module. we use huggingface transformers @xcite for model training and inference, and clulab timenorm's synchronous context free grammar module @xcite for normalizing time expressions to iso standard. the system processes all the patients and notes for a given cancer type and split of the dataset. we processed patients by cancer type and dataset split since there are overlapping patient identifiers across different cancer types and splits (although the patients are different). ## participating systems in this section, we briefly describe the approaches of participating systems. details of each system can be found in the separate papers by each of the team. the participants explored a variety of methods, including logistic regression, tf-idf, ngrams, transformer models, zero-shot prompting with large language models (llms), and instruction tuning. biocom participated in subtask 1. they utilized scispacy for named entity extraction (ner) and logistic regression to classify temporal relations. they used unigram term frequency-inverse document frequency (tf-idf) to get features from the input text. clinicalrxminers submitted two systems for subtask 1. in submission 1, clinicalrxminers utilized a machine learning (non-deep learning) approach and employed n-grams as features of the input, with a soft voting classifier as the model for making predictions. in submission 2, clinical-rxminers utilized a pretrained language model (lm) named gliner @xcite , which is specialized for ner. kclab @xcite utilized a hybrid method, employing ctakes @xcite for preprocessing and pubmedbert @xcite for post-processing. their system was built on top of the baseline model provided by the organizers. additionally, kclab used the umls @xcite database. kclab participated in both subtask1 and subtask2. lailab @xcite utilized two approaches: supervised fine-tuning of language models and a pipeline approach combining rulebased ner with deep learning based relation classification. for subtask 1, they finetuned teams approach lm or algorithm task biocom_submission1 machine learning logistic regression 1 clinicalrxminers_submission1 machine learning soft voting classifier 1 clinicalrxminers_submission2 deep learning gliner base 1 kclab_submission1 finetuned lm pubmedbert 1, 2 lailab_submission1,2,3 finetuned lm flan-t5-xxl, bart-large 1, 2 lexicans-submission1,2,3 zero-shot prompting llama2, mistral, zephyr, meditron, and mixtral 1 nlpeers_submission1 finetuned lm deberta-v3-base 1 nlpeers_submission2 few-shot prompting mixtral-8x7b-instruct-v0.1 1 nyulangone_submission1 zero-shot prompting mixtral 8x7b 2 utsa-nlp_submission1,2,3 instruction tuning lm, continued pretraining lm openchat-3.5-7b 1, 2 wonder_submission1,2,3 finetuned lm bio-lm 1, 2 lexicans @xcite used llms with zero-shot prompting to extract relations. they also utilized the thyme ontology to formalize the representation of entities and their relationships. a few llms such as llama2, mistral, zephyr, meditron, and mixtral @xcite @xcite were tested under various settings. additionally, a data normalization step was performed to transform time entities into absolute date-time formats. nlpeers @xcite developed two systems, both submitted for subtask1. for submission 1, nlpeers fine-tuned the microsoft/deberta-v3-base model and used it for temporal relation classification. additionally, the heideltime library ** @xcite and an llm-based prompt with the openchat 3.5 model @xcite were used to normalize time expressions. for submission 2, the nlpeers team applied few-shot prompting with the mixtral-8x7b-instruct-v0.1 model @xcite , the prompt was chosen by dspy @xcite , a framework for ** https://github.com/heideltime/heideltime algorithmically optimizing lm prompts. a chain-of-thought @xcite approach was integrated during the prompt searching step by dspy. for time expression normalization, heideltime was also used in submission 2. nyulangone employed an llm-based prompt approach with minimal pre-and post-processing. nyulangone participated only in subtask2, which means the team did not use the gold annotation provided in subtask1. utsa-nlp @xcite ) presented an instruction-tuning based approach. the utsa-nlp team reformulated the task into a question-answering (qa) dataset for both the entity extraction step and temporal relation classification step, then instruction-tuned an llm, openchat-3.5-7b, on the qa dataset. the team continued pre-training the instruction-tuned model on a portion of the unlabeled dataset in one of their submissions. for the temporal relation classification step, they used an open-sourced llm to generate reasoning for the answer. wonder @xcite participated in subtasks 1 and 2. they employed a supervised finetuning approach, formulating the task as a multiclass sentence classification task, where the input was the text between the event and time expression. for subtask 2, medtagger † † was used to identify all the potential event-timex3 pairs. time ex ## results and discussion finetuning lms: for both subtasks, the top teams, i.e. lailab and wonder, employed finetuned pretrained language models as the core technology. lailab finetuned flan-t5-xxl @xcite and bart-large @xcite , which have 11b and 400m parameters respectively. they achieve best performance on all subtasks (overall and per type of cancer) except for subtask2, breast cancer. the wonder team finetuned bio-lm @xcite , yielding top 3 results across all subtasks (excluding the baseline system). the other two teams with good results are nlpeers and kclab, who finetuned deberta-v3-base @xcite and pub-medbert @xcite respectively. overall, the commendable performances of those teams suggest that finetuning lms remains the optimal approach for optimizing system performance if gold labeled data and computing resources are available. ## conclusion the 2024 shared task on chemotherapy treatment timeline extraction is unique in both (1) focusing on a highly complex task, and (2) providing a large corpus of ehr data to the participants. the community embraced the task with enthusiasm and employed diverse methodologies, thus enabling robust comparison of approaches. perhaps surprising in our current era of very large lms, fine-tuned smaller lms achieved superior performance. this discrepancy between prompting llms and finetuning smaller-sized lms suggests that more sophisticated llms or prompting techniques are necessary in order to achieve optimal results for challenging tasks such as patient-level chemotherapy timeline extraction.