Temporal Relation Classification: An XAI Perspective (2025)

Abstract: Temporal annotations are used to identify and mark up temporal information, offering definition into how it is expressed through linguistic properties in text. This study investigates various discriminative pre-trained language models of differing sizes on a temporal relation classification task. We define valid reasoning strategies based on the linguistic principles that guide commonly used temporal annotations. Using a combination of saliency-based and counterfactual explanations, we examine if the models’ decisions are in line with these strategies. Our findings suggest that the selected models do not rely on the expected linguistic cues for processing temporal information effectively.

## introduction temporal information processig is a fundamental aspect of natural language and is essential for nlp applications including question answering @xcite ), text summarization @xcite , and information retrieval @xcite . transformer-based pre-trained language models have shown impressive performance in such tasks @xcite @xcite ). yet, their interpretation of time diverges from human interpretation @xcite , making it challenging to evaluate their temporal processing, and whether they indeed interpret the temporal information as expected @xcite . while temporal benchmarks @xcite @xcite have been extensively developed, performance metrics alone do not reveal the under-1 https://github.com/sofitere/trc-xai lying mechanisms or explain how conclusions are reached @xcite . this study contributes a methodology and an evaluation dataset for evaluating nlp models on temporal relation classification. we define valid reasoning strategies, and use a combination of saliencybased and example-based explainability methods to assess whether a model follows these strategies when making decisions. our framework extends the work introduced by ray @xcite . we explore discriminative models of varying sizes to determine if larger models, trained more extensively on more data, are also more likely to base their decisions on valid information retrieval processes. our findings suggest that while larger models show better performance on the task, they frequently deviate from expected reasoning strategies. these results align with broader concerns about the reliability of current popular benchmarks, where high accuracy can mask a reliance on shortcuts or spurious correlations. we discuss the limitations of this framework, together with the opportunities and challenges of extending it to generative models. temporal annotation. timeml @xcite remains the most widespread format for temporal annotation, and it became the basis for iso standard @xcite . timeml includes conventions to identify and describe temporal elements in text, including temporal expressions (timex), events, temporal relations (t-links), signals (signal), and relation types. timebank corpus @xcite has been re-annotated in several projects to increase the density of t-links @xcite @xcite and improve its consistency. its texts have been utilized in subsequent projects providing additional annotation in other formats, including matres @xcite . ## discussion this study investigates selected discriminative models on a temporal relation classification task. while numerous benchmarks have been developed to evaluate models' temporal processing abilities, our experiments highlight limitations in these evaluations. specifically, we adopted one commonly used benchmark dataset and found that models can achieve high accuracy without following the expected reasoning patterns. the framework used in this study offers a step toward improving evaluation methodologies by emphasizing whether models make correct predictions for the right reasons. it establishes clear success criteria for the task and highlights the role of validating "reasoning" to accurately assess model performance. post-hoc and local explainability methods are commonly used to determine if model decisions are justifiable from a human perspective, yet their reliability and utility is often questioned @xcite . counterfactual explanations are considered as more truthful @xcite , but require careful handling to prevent unreliable conclusions. saliency scores, on the other hand, may not reflect the model's decision-making process. different saliency methods can produce conflicting results, meaning that they inconsistently reflect the model's decision process @xcite @xcite . moreover, the lack of a ground truth for saliency evaluation makes it challenging to evaluate whether they correctly approximate the model's processes @xcite . having addressed the truthfulness of these methods, the question of their utility remains. for this study, we must conclude that the models follow some other strategy for correct prediction (rather than relying on the expected reasoning). explainability methods should aim to make a model's decisions understandable to humans. however, this is challenging when a model's reasoning processes do not align with human reasoning @xcite . identifying alternative reasoning strategies or shortcuts through these explanations is challenging because they are not necessarily human interpretable (see figure 3 6 ), raising questions about the practical value of these methods, as they only provide a partial interpretable view of a model's processes, and fail to provide actionable insights. ## extending to generative models extending the experiments to adapt modern generative models, such as llama @xcite , gpt @xcite , and olmo @xcite , presented challenges, particularly in interpreting saliency scores. @xcite provides a taxonomy of explainability methods for transformer-based language models, categorizing them based on training paradigms (e.g., fine-tuning and prompting), which influence their goals and effectiveness. generative models, primarily prompt-based, leverage their extensive scale and learned prompts for task execution. these complex processing strategies @xcite make it difficult to isolate specific components of the model responsible for particular decisions. localized and examplebased explainability methods become less meaningful @xcite . moreover, differences in training objectives (e.g. autoregressive versus masked language), make it challenging to apply explainability methods that work reliably across all model types. trustworthiness of explanations is both task and model-dependent @xcite . variations in how models process and prioritize input can result in inconsistencies in the effectiveness of these methods. this variability underscores that no single explanation method can be universally treated as a standard across all contexts. consequently, conducting meaningful comparisons between different architectures becomes challenging, as the results may be unreliable or even misleading. further research is needed to validate the robustness of such comparative analyses. in contrast, counterfactual explanations provide a promising approach for evaluating generative models. assessments centered on counterfactual instances could help determine whether these models maintain consistent reasoning when confronted with alternative scenarios. we leave the adaption of the presented counterfactual explanations ( § 4.6) to generative models to future work. of particular relevance, roccabruna et al. (2024) highlights the performance gap between generative and discriminative models in temporal relation classification tasks. encoder-only models based on roberta consistently outperform generative models like llama. this performance gap is attributed to roberta's ability to fully utilize input context via masked language modeling, in contrast to llama's autoregressive objective, which tends to prioritize final tokens in the input sequence. this underscores the significance of discriminative models for trc and reinforces the value of evaluating whether their decisions are based on valid and expected reasoning patterns. ## conclusion temporal annotations are used to mark all linguistic features that express temporal information in text. we evaluate selected discriminative models on a temporal relation classification task, examining whether they rely on these features for correct predictions. experiments involve a combination of counterfactual explanations and saliencybased methods. high alignment between these two explanations indicates that a model is following a valid processing strategy. we find that this is not the case for the selected models, meaning that they might learn spurious correlations or shortcuts rather than relying on the defined linguistic features that form temporal meaning. we evaluate the limitations of this framework by examining the utility of the explainability methods used, together with challenges and potential directions for extending the framework to generative models.