Will LLM s Replace the Encoder-Only Models in Temporal Relation Classification? (2024)

Abstract: The automatic detection of temporal relations among events has been mainly investigated with encoder-only models such as RoBERTa. Large Language Models (LLM) have recently shown promising performance in temporal reasoning tasks such as temporal question answering. Nevertheless, recent studies have tested the LLMs’ performance in detecting temporal relations of closed-source models only, limiting the interpretability of those results. In this work, we investigate LLMs’ performance and decision process in the Temporal Relation Classification task. First, we assess the performance of seven open and closed-sourced LLMs experimenting with in-context learning and lightweight fine-tuning approaches. Results show that LLMs with in-context learning significantly underperform smaller encoder-only models based on RoBERTa. Then, we delve into the possible reasons for this gap by applying explainable methods. The outcome suggests a limitation of LLMs in this task due to their autoregressive nature, which causes them to focus only on the last part of the sequence. Additionally, we evaluate the word embeddings of these two models to better understand their pre-training differences. The code and the fine-tuned models can be found respectively on GitHub.

## introduction an important ability in understanding information flows such as news is to recognize the temporal relations of events, which happened, are happening, or will happen, to order them into a coherent storyline. indeed, temporal relations are utilized in many natural language processing tasks such as narrative understanding @xcite , story generation @xcite , summarization (liu et al., 2009; gung and kalita, 1 https://github.com/brownfortress/llms-trc 2012) and temporal question answering @xcite . the automatic recognition of temporal relations (e.g. before or after) is referred to as temporal relation classification (trc). figure 1 depicts an example of the trc task, as defined in the tempeval challenges @xcite @xcite , that is to predict the temporal relation after between the two given connected events, e 1 : accused and e 2 : driving. the temporal relations used to annotate the corpora for training and testing models have been originally defined in allen's interval algebra @xcite . in this, events are described as intervals rather than time points to handle explicit and implicit or vague time references. in recent years, several works on the trc task have focused on exploiting a variety of features to best represent the surrounding context of the two events. some of these are syntactic @xcite , semantic (e.g., event arguments) @xcite and discourse @xcite features. to utilize those features, models have been based on complex architectures constructed on top of encoder-only pre-trained language models such as @xcite and roberta @xcite . more recently, large language models (llms) have become widely used in many natural language processing tasks, achieving state-of-the-art performance in sentiment analysis @xcite , named entity recognition @xcite and natural language inference @xcite . although llms have been evaluated on the temporal question answering task @xcite @xcite , in which temporal relations are implicitly used, limited studies have been conducted on the performance of llms on the trc task @xcite @xcite experimenting with closed-sourced llms, limiting interpretability studies. in this paper, we study the performance and the decision process of seven open and closed-sourced large language models (llm) in performing the task of temporal relation classification (trc) over three different publicly available benchmark corpora. along with an example-label in-context learning (icl) approach @xcite , we cast the trc task into a question answering task form to create qa prompts. furthermore, we use the low-rank adaptation (lora) @xcite technique to fine-tune llama2 7b and llama2 13b to measure the upper bound performance of such models. the results show that although the autoregressive llms with qa prompts perform better than example-label prompts, they struggle with the trc task compared to smaller encoder-only models based on roberta in all settings. we further investigate the possible reasons for this by analyzing the most contributing tokens to the prediction extracted with @xcite algorithm as an xai attribute scorer. this analysis shows that llms tend to focus more on the last tokens of the target sentence due to their autoregressive nature, rather than using the entire context as encoder-only models do. we evaluate the word embeddings extracted from llms and roberta to highlight the pre-training differences between these two models. we observe that roberta still performs better than llms, suggesting that the gap between these two models is probably due to their different pre-training strategies. the contributions of the paper are the following: the remainder of the paper is organized as follows. section 2 reviews the related works. in section 3, we formally define the trc task. section 4 presents the encoder-only model based on roberta and the prompts for llms. in section 6, we present the results of the tested models and the explainability studies. in section 7, we present and discuss the error analysis. finally, we present our conclusions. temporal qa the purpose of extracting temporal relations among events proposed in timeml @xcite is to improve the performance of temporal question answering task @xcite @xcite , which is to answer temporal grounded queries such as "which is the current us president?". recently, the temporal qa task has been used for testing and challenging the temporal reasoning ability of llms @xcite by querying the llms parametric knowledge with time-grounded questions. temporal relations corpora one of the first and largest corpora annotated with temporal relations is the timebank @xcite corpus. this corpus has been annotated using the time-ml scheme @xcite . this scheme provides a definition used to identify events and defines a set of thirteen temporal relations which follow in principle the thirteen interval relations of allen's interval algebra @xcite . adaptations and refinements of the iso time-ml @xcite scheme, mainly regarding the event definition and the number of relations, have been used to annotate most of the available corpora such as in thyme-timeml @xcite , timebank-dense @xcite , red (o'gorman et al., 2016), matres, maven-ere @xcite and timeline. ## task definition the temporal relation classification (trc) task can be defined as follows. the corpus comprises a set of documents d. a document @xmath0, ..., s n ], where a sentence is a sequence of tokens i.e. @xmath1, ..., w n ]. a sentence is delimited by a full stop, exclamation or question mark. each document in the corpus contains a set of annotated event triggers @xmath2, ..., e n } where e is a span of tokens of a sentence of a document i.e. @xmath3, ..., w j ] ∈ s with @xmath4, @xmath5. the trc task is to assign a temporal relation r from a predefined set r to a given pair of connected events (e i , e j ), where e i ̸ = e j . the set of relations r changes depending on the annotation scheme of the corpus as described in section 5.1. besides, the temporal relations @xmath6, e j ) and r ′ ∈ (e j , e i ) are always the opposite (e.g. before and after) i.e. @xmath7, where by definition @xmath8. indeed, in allen's interval algebra, the 13 temporal relations are composed of equal plus six temporal relations and their corresponding opposites. ## models in this section, we describe the encoder-only model and llms with fine-tuning and prompting approaches tasked with temporal relation classification (trc). ## evaluation and results we have experimented with seven large language models (llm): five open-source, namely llama2 7b, llama2 13b, llama2 70b @xcite namely gpt-3 @xcite and gpt-3.5 foot_0 (openai, 2023). as described in section 4, we have adopted in-context learning (icl) and fine-tuning approaches using the following prompts: regarding the icl experiments, we have experimented with zero and different numbers of few-shot examples on llama2 7b. we have observed that the models achieve the best performance by using one example for each class of the corpus, i.e. five for tb-dense and three for both matres and time-line. furthermore, to measure the impact on the performance of the few-shot example selection, we have sampled and frozen five sets of few-shot examples to create the context for all three prompt types. the results foot_1 of icl and fine-tuning experiments are reported in regarding the performance of individual llms with icl, llama2 70b outperforms all other open and closed-source models on all corpora. furthermore, despite the 175b (billions) parameters, gpt-3 yields worse results compared with llama2 7b on all corpora and mixtral 8×7b on matres whose numbers of parameters are 7b and 12b foot_2 respectively. furthermore, llama2 7b outperforms mistral 7b on all corpora and mixtral 8×7b on timeline and tb-dense. besides, llama2 13b underperforms llama2 7b in all corpora but tb-dense. to estimate the upper bound performance of llms, we have fine-tuned llama2 7b and 13b on the three corpora using the same prompts of icl but with a zero-shot approach. on matres, llama2 13b fine-tuned using prompt qa 2 achieves close results to the encoder-only model based on roberta . instead, llama2 7b fine-tuned with qa 1 scores the highest micro-f1 among llms on timeline but is 11.0% inferior to the robertabased model. on tb-dense, while achieving the best score compared with prompt p, llama2 13b scores the highest gap of 27.7% w.r.t. robertabased model. a possible reason for this is that tb-dense has two additional temporal relations compared to matres and timeline, making the training and inference more challenging. overall, the results achieved by llms in all settings are always outperformed by the encoderonly model based on roberta. indeed, roberta scores improvements w.r.t icl best models of 22.3% on matres, 25.4% on timeline and 51.7% on tb-dense. although fine-tuning llama2 7b and 13b substantially reduces this gap on ma-tres, the differences are still high in the other two ence time it automatically selects and utilizes a subset of 12 billion. ## error analysis we have analyzed the error of the roberta-based model, llama2 70b and llama2 13b fine-tuned by comparing the performance between intra and inter-sentences on matres as reported in to investigate whether there is a subset of the test set with challenging examples for all the models, we have computed the intersection between the errors of the roberta-based model and the correct predictions of llama2 13b fine-tuned and llama2 70b with icl. the sizes of these intersections are 23.3% for llama2 13b fine-tuned and 33.7% for llama2 70b, which account for 2.8% and 4.0% of the test set, respectively. by manually ## conclusions in this work, we have evaluated seven open and closed-sourced llms on the temporal relation classification task with an in-context learning and fine-tuning approach. we have shown that the encoder-only roberta-based model achieves the highest results compared to llms. explainable studies suggest that one of the reasons for this gap is due to the different pre-training tasks. finally, considering the low performance and the huge amount of computational resources needed at finetuning and inference time, llms might not be the best option for the trc task compared to a more accurate and low-resource demanding robertabased model. future work a possible future work is to further investigate the pre-training task differences, by pre-training two models on the autoregressive and masked language model tasks using the same parameter size and training set. another possible direction is to study a hybrid architecture to join the best performance of the roberta-based models and the large language models such as llama2 13b fine-tuned and llama2 70b.