Time to Revisit Exact Match (2025)

Abstract: Temporal question answering is an established method for evaluating temporal reasoning in large language models. Expected answers are often numeric (e.g., dates or durations), yet model responses are evaluated like regular text with exact match (EM), unable to distinguish small from large errors. In this investigative work, we frame temporal question answering as a numerical estimation task to assess the shortcomings of EM. We introduce TempAnswerQA , a benchmark distilled from Test of Time and TempTabQA, where all questions require a numerical, temporal answer, allowing us to evaluate models beyond EM. We use the forecasting metrics symmetric mean absolute percentage error (sMAPE) and mean absolute scaled error (MASE). With sMAPE, we find that error size and EM are decoupled. Models with low EM still have low sMAPE (both 20%), and some models have high sMAPE despite high EM. Scaling errors by the deviation of the ground truth data with MASE reshuffles model rankings compared to EM, revealing gaps in models’ understanding of temporal domain knowledge, especially when trained with synthetic data. Lastly, the models’ most frequent error is to deviate by only \pm1 from the ground truth. sMAPE and MASE, unlike EM, adequately weight these errors. Our findings underscore the need for specialised metrics for temporal QA tasks. Our code and data are available on https://github.com/aauss/temporal-answer-qa.

## introduction time is an inherent part of the real world, and reasoning about it is essential for intelligent behaviour @xcite . as such, temporal reasoning is crucial in many domains, including high-stakes areas such as logistics @xcite , finance @xcite , and medicine @xcite , which increases the stakes for adequate evaluation. temporal question-answering (qa) benchmarks are a well-established method for evaluating temporal reasoning in large language models (llms), and the binary string-matching metric exact match (em) is a widely used for this purpose @xcite . while prevalent, em does not consider the continuous nature of time. as illustrated in fig. 1 , em considers model a and model b to be tied, despite model a's error being much smaller (∆2h) than @xcite . although continuous alternatives exist, such as rouge, meteor @xcite and f1 @xcite , they collapse to binary scores when temporal answers consist solely of digits. the limitations of current metrics have been noted across domains. in medication direction systems, for example, these metrics are unable to distinguish acceptable from lethal errors in medication schedules @xcite (e.g., q1 in fig. 1 ). therefore, current benchmarks suffer from a mismatch between evaluation and deployment risk. this work aims to address this mismatch by exploring metrics more suitable for the temporal nature of the task. exploring continuous alternatives to em allows us to differentiate between small and large errors. beyond that, continuous metrics are more suitable for assessing temporal reasoning for two additional reasons. first, studies by jack @xcite and @xcite have shown that llms tend to approximate the answer to a (temporal) arithmetic task. relying solely on em undervalues models that approximate correct answers well. second, answers to temporal questions can be ambiguous, such as calculating a person's age using only their birth year, where two answers with a difference of 1 year could be true @xcite . this ambiguity is caused by transitional times. with em alone, we cannot distinguish relevant errors from transitional time ambiguities. we frame temporal qa as a numerical estimation task and borrow two scale-free error metrics from forecasting to evaluate llms beyond em. the first is the symmetric mean absolute percentage error (smape) @xcite , which measures the percentage error of the model predictions. the second is the mean absolute scaled error (mase) @xcite . this metric scales errors by a sensible baseline derived from the benchmark data, thus aiming to measure the models' temporal domain knowledge. our contributions can be summarised as follows: ## experiments and results we conducted our experiments based on these six selected models of different sizes, with and without few-shot prompting, on tempanswerqa. its questions expect temporal answers that can be assessed in a regression-like fashion. our experiments aim to answer the following questions: ## conclusion in this work, we release tempanswerqa, a distilled benchmark focused on the continuous nature of time. with it, we show that em systematically ignores error magnitude and direction, which are both critical to temporal reasoning. to this end, we use smape and mase, two regression-based metrics that capture properties in the prediction errors of the models that em does not. smape is relatively low, even if em is low. em underestimates the models' understanding of the correct answer. qwen2.5-14b, which was trained on structured data and mathematical reasoning, ranks first according to smape, overtaking llama-3.3-70b, the best model according to em. both llama models perform the best according to mase. they are the only models not trained on synthetic data, suggesting that their temporal domain knowledge is higher and synthetic data distorts this knowledge. answers to duration-related questions can be ambiguous due to transitional times, leading to two answers being correct with a difference of just 1. this leads to an inflation of errors equal @xmath0. smape and mase are continuous metrics and thus provide a more balanced evaluation than em. lastly, we show that mase and smape are valuable additions to em. although smape yields rankings similar to em, its sensitivity to error sizes gives it an edge when large deviations need to be penalised. mase ranks models significantly differently. it attempts to scale errors by plausible ranges of correct answers and thus tries to probe models' temporal domain knowledge rather than their relative errors as with smape. without humanannotated data, mase is a viable alternative to measure prior temporal knowledge.