Analysing zero-shot temporal relation extraction on clinical notes using temporal consistency (2024)

Abstract: This paper presents the first study for temporal relation extraction in a zero-shot setting focusing on biomedical text. We employ two types of prompts and five Large Language Models (LLMs; GPT-3.5, Mixtral, Llama 2, Gemma, and PMC-LLaMA) to obtain responses about the temporal relations between two events. Our experiments demonstrate that LLMs struggle in the zero-shot setting, performing worse than fine-tuned specialized models in terms of F1 score. This highlights the challenging nature of this task and underscores the need for further research to enhance the performance of LLMs in this context. We further contribute a novel comprehensive temporal analysis by calculating consistency scores for each LLM. Our findings reveal that LLMs face challenges in providing responses consistent with the temporal properties of uniqueness and transitivity. Moreover, we study the relation between the temporal consistency of an LLM and its accuracy, and whether the latter can be improved by solving temporal inconsistencies. Our analysis shows that even when temporal consistency is achieved, the predictions can remain inaccurate.

## introduction reasoning regarding the temporality of events detected in a text (e.g., understanding their duration, frequency, and order) is an essential part of natural language understanding @xcite . event ordering can be approached as identifying temporal relations between two events, a task often referred to as temporal relation @xcite . this task can also be applied to medical text (biotempre), e.g., clinical notes written by clinicians regarding a patient's visit, and various medical events such as symptoms, treatments, tests, and other medical terms (see figure 1 ). biotempre has numerous useful applications in figure 1 : an example of three event pairs annotated with temporal relations. in the right part, the order of the events with respect to time (t) is shown and the consistency of uniqueness and transitivity. in tempre, there are important properties that emerge from the temporal nature of this task and determine the relations between events (see figure 1 ). such properties are symmetry (e.g., a before b ⇒ b after a) and transitivity (e.g., a before b and b before c ⇒ a before c). additionally, we identify the property of uniqueness: each pair of events can have only one temporal relation since they are mutually exclusive. these properties can be utilized to enforce global temporal consistency on a model's predictions: for example, on a unified output of different classifiers @xcite , on a model that operates locally (i.e., with one pair of events as input, @xcite ), or on predicted relations between different types of events @xcite . recently, large language models (llms) have shown remarkable performance in several tasks even in a zero-shot setting, which helps to tackle the need for training data @xcite . numerous works experiment with predictions of llms and study their reasoning abilities and the impact of various prompts in different tasks @xcite @xcite . despite the success of llms, studies show that these models continue to face challenges in temporal reasoning, especially in tempre @xcite @xcite , as well as in biomedical tasks @xcite . in zero-shot tempre, @xcite employed different prompts for chatgpt and found that it has a considerably lower performance compared to standard supervised methods. they also report chatgpt's tendency to provide temporally inconsistent responses, but only in terms of symmetry and did not perform an evaluation of temporal consistency specifically. furthermore, to the best of our knowledge, we are the first ones to investigate the temporal reasoning capabilities of llms on medical data. in this paper, we perform zero-shot biotempre on clinical notes (i.e., medical texts documenting patients visits) by using prompts consisting of a clinical note and questions regarding which temporal relation exists between a pair of events. 1 we experiment with two different prompting strategies (batchqa and cot) and five widely-used llms (gpt-3.5, mixtral 8x7b, llama2 70b, gemma 7b, and pmc-llama 13b). our findings reveal that llms perform poorly in this task, with a difference of approximately 0.2 in f1 score compared to supervised models. furthermore, we calculate consistency scores for uniqueness and transitivity for each llm in order to assess their temporal consistency and its impact on accuracy. consistency is later enforced on the predictions with an integer 1 we do not perform event detection and instead consider the events in each text already known. linear programming (ilp) method, revealing that solving the inconsistencies does not improve the f1 score. overall, our contributions are: ## zero-shot tempre results to evaluate the correctness of the predicted relations, we calculate the precision, recall and f1 scores. for each pair of events p @xmath0, e k ), we check if the predicted relation r i matches the gold relation. hence, we calculate the triple (e j , e k , r i ) match between the predictions and the ground truth. in our experiments demonstrate that the best performing methods are the same for the gold and the candidate pairs. as expected, the f1 score of the methods when the candidate pairs are used is lower, mostly due to the decrease in recall. the best performing method is llama cot + w-order in terms of f1 score. on the other hand, mixtral cot achieves the best precision score and gemma batchqa + w-order the best recall. overall, the supervised methods consistently outperform the methods in the zero-shot setting, with an average difference of approximately 20% f1 score. in general, most llms (except for gemma) exhibit improved performance when the cot prompting approach is used. however, in an llm-based comparison, we observe that the performance varies depending on the type of prompt used. for example, llama with cot has the highest f1 score, but when batchqa is used, the score drops almost in half. moreover, the combination of w-order predictions with the zero-shot methods yields improvements in recall and f1 score, but in most cases, it harms precision. notably, pmc-llama, the medical llm we employed, has low results and is often outperformed by the general domain llms, showing no advantage from pre-training on biomedical text. ## temporal consistency analysis considering the temporal nature of the tempre task, we investigate the impact of incorporating the following two temporal properties in the zero-shot setting: 1. uniqueness, requiring that each event pair has exactly one relation, and 2. transitivity (see transitivity rules in how consistent are different zero-shot methods? we calculate two consistency scores: one for uniqueness c u and one for transitivity c t , which show the percentage of cases where the corresponding temporal property was not violated. the consistency score for uniqueness is calculated based on the number of pairs as follows: and can go up to 97% (gemma batchqa). in this evaluation, we also find that there is no clear winner among the llms or the prompt types, since the same llm shows different levels of consistency with different prompt types. the combination with the highest consistency for uniqueness is llama with batchqa. the consistency score for transitivity is calculated based on triples of event pairs in the following form: ((e i , e j ), (e j , e k ), (e i , e k )). we first find these triples in the dataset and then obtain the relations predicted for them. if r 1 , r 2 and r 3 are the pre-dictions for each respective pair in the triple, then for r 3 , it should hold that r 3 ∈ trans(r 1 , r 2 ). 10 if it does not hold, then we have a transitivity violation. therefore c t is calculated as: how is the temporal consistency of the predictions connected to their correctness? when comparing the consistency scores with f1, we observe a contradiction. models that have high consistency have a lower f1 score. in particular, llama batchqa is the most consistent in terms of uniqueness and transitivity, but has one of the lowest f1 scores. especially for the candidate pairs, the f1 score is even lower than the rule-based baseline, yet c u is 70.58% and c t is 80.05%. moreover, llama cot, which is the best in terms of f1 score, has low consistency with around only 30% of predictions being unique and 60% correct transitivity triples. these insights suggest that temporal consistency does not always mean correctness. can the predictions be improved by a temporal constraint-based ilp approach? following the approach proposed by @xcite , we implemented an ilp step that uses the temporal properties as constraints and changes inconsistent predictions so that the constraints are not violated. 12 this study aims to investigate whether enforcing consistency can improve the accuracy of the predictions. first, we assign a confidence score sc to each triple (e i , e j , r k ), ∀r @xmath1. the score sc for a pair of events @xmath2, e j ) equals 1, if the relation was predicted from the model, and 0.2 otherwise. next, we create a binary vector, which is 10 the transitive relations for the relation set we used can be found in the objective of the ilp method is to maximize the confidence score sc based on the indicator i: as shown in ## pairs distance-based analysis since clinical notes contain long texts (see section 4.1), we perform an analysis based on the dis-tance of event pairs for the best-performing llm (llama cot). first, we calculate the distance in terms of characters between the events for all the gold pairs. then, we sort the pairs by their distances and split them to 10 bins, so that each bin contains roughly the same amount of pairs. finally, the f1 score is calculated for the prediction of the pairs contained in each bin. figure 3 depicts the barplot with the bars representing the pairs in the specific distance range and the corresponding f1 scores. we observe that 37.5% of the pairs have a distance of 0 to 100 characters. larger distances appear less frequently and hence the range of distance is smaller in the first bars, while the last bars have larger ranges. there is no consistent decrease in f1 score as the distance increases, meaning that the model is not affected by the distance of events in the text. ## conclusion in this paper, we performed biotempre on clinical notes in a zero-shot setting employing five different llms. two types of prompts were used, namely batchqa and cot, in order to obtain llms' responses. the zero-shot performance of all llms is lower compared to supervised learning methods. moreover, we perform a temporal evaluation by calculating the consistency score of each llm for the temporal properties of uniqueness and transivity. we find that, in general, llms' predictions are temporally inconsistent and, interestingly, the model with the higher consistency scores (llama batchqa) has one of the lowest f1 scores. an ilp method utilized to enforce consistency on the models' predictions fails to improve their accuracy. these findings indicate the importance of the relation between temporal consistency and correctness, emphasizing the need for further study in order to assist temporal reasoning. r 1 r 2 trans(r 1 , r 2 ) before before before after after after includes includes includes is included is included is included simultaneous simultaneous simultaneous before simultaneous before after simultaneous after includes simultaneous includes is included simultaneous is included before after [before, after, includes, is included, simultaneous] before includes [before, includes] before is included [before, is included] after before [before, after, includes, is included, simultaneous] after includes [after, includes] after is included [after, is included] includes before [before, includes] includes after [after, includes] includes is included [before, after, includes, is included, simultaneous] is included before [before, is included] is included after [after, is included] is included includes [before, after, includes, is included, simultaneous] simultaneous before before simultaneous after after simultaneous includes includes simultaneous is included is included