Evaluating LLM s’ Ability to Understand Numerical Time Series for Text Generation (2025)

Abstract: Data-to-text generation tasks often involve processing numerical time-series as input such as financial statistics or meteorological data. Although large language models (LLMs) are a powerful approach to data-to-text, we still lack a comprehensive understanding of how well they actually understand time-series data. We therefore introduce a benchmark with 18 evaluation tasks to assess LLMs’ abilities of interpreting numerical time-series, which are categorized into: 1) event detection—identifying maxima and minima; 2) computation—averaging and summation; 3) pairwise comparison—comparing values over time; and 4) inference—imputation and forecasting. Our experiments reveal five key findings: 1) even state-of-the-art LLMs struggle with complex multi-step reasoning; 2) tasks that require extracting values or performing computations within a specified range of the time-series significantly reduce accuracy; 3) instruction tuning offers inconsistent improvements for numerical interpretation; 4) reasoning-based models outperform standard LLMs in complex numerical tasks; and 5) LLMs perform interpolation better than forecasting. These results establish a clear baseline and serve as a wake-up call for anyone aiming to blend fluent language with trustworthy numeric precision in time-series scenarios.

## introduction the application of large language models(llms) has expanded from tasks with natural language inputs, such as question answering and summarization, to tasks involving structured data inputs, referred to as data-to-text tasks. within data-to-text, numerical time series generation, such as various sensor data or stock market prices, hold significant potential for practical applications like financial reporting @xcite @xcite and automated com-inflation is an important measure of any country 's economy , and the retail price index ( rpi ) is one of the most widely used indicators in the united kingdom , with the rate expected to fall to 1.2 percent in 2020 . the forecasted inflation rate for this index is estimated to increase in upcoming years , to 3.1 percent by 2022 . ## related work interpreting sequences of numbers is a basic step for any model that needs to generate reliable text about data. a recent survey highlights the growing use of time-series inputs in large language models @xcite . real applications already depend on this skill, for example automated financial reports @xcite , chart-to-text captioning @xcite , and systems that produce natural-language forecasts @xcite . the numerical understanding ability of llms has attracted significant attention from researchers, leading to the release of benchmark datasets for mathematical problems @xcite @xcite @xcite , reasoning with tabular data @xcite , and handling numerical information in free text @xcite . however, these datasets do not examine whether models can transform raw numbers into coherent natural-language prose, especially for time-series inputs-an ability our benchmark is designed to measure. while existing time-series forecasting tasks, such as those introduced by @xcite , could be used as one of benchmarks for time-series understanding to some extent, they fundamentally differ from our approach because of three reasons: 1) the time-series forecasting tasks lack of a definitive answer -forecasting tasks require predicting future values, which inherently allows for multiple plausible outcomes, and even among humans, predictions can vary, 2) dependence on inference -many existing tasks require extrapolation or assumptions beyond the given data, and 3) requirement for domain expertise -some tasks, such as predicting market prices in finance @xcite , rely heavily on domain knowledge. a very recent work by @xcite also proposes a benchmark for evaluating llms on numerical time-series understanding. they primarily use synthetic data for tasks that often integrate numerical series with textual information or analyze multiple series, rather than focusing solely on the interpretation of individual numerical sequences. in contrast, our benchmark provides a more foundational and comprehensive assessment focused on single numerical sequences, quantitatively measuring llms' core numerical comprehension across diverse task types. ## main results however, certain tasks remain challenging. in event detection, detecting peak and dip points proves particularly difficult, with gpt-4o achieving only 0.64 for peak detection and 0.52 for dip detection. additionally, computation tasks also pose challenges-for example, gpt-4o scores only 0.50 on the summation task. these results indicate that while llms perform well on simpler numerical time-series interpretation tasks, more complex computations remain difficult. does model size improve performance?: increasing the model size in both llama and swallow leads to significant performance improvements across all tasks. for example, we observe a substantial accuracy increase from 0.34 (llama3.1-8b) to 0.60 (llama3.1-70b). this size-driven gain was statistically significant (appendix g). however, openai o1 does not always show a clear advantage. in the computation task of averaging within a specific range, it achieved only 0.23, a score comparable to other gpt models. while reasoning-based models excel in many nlp tasks, they fail to outperform others in certain numerical reasoning tasks such as constrained averaging. although both tasks involve linear prediction, their performances differed greatly. most models, particularly high-performing ones like ope-nai o1 (1.00), gpt-4o (0.90), and gpt-4-turbo (0.91), showed strong proficiency in linear imputation. this suggests llms can generally execute linear interpolation, filling missing values using the linear relationship between immediately adjacent known points. conversely, all models performed significantly worse on linear forecasting. even openai o1, excelling at imputation, scored only 0.56 on forecasting. gpt-4o (0.26) and gpt-4turbo (0.27) scores were markedly lower. ## analysis 6.1 examples of generated reasoning tokens one key advantage of reasoning-based models is their explainability. 5 unlike standard llms, distill-llama generates reasoning tokens, allowing us to examine its thought process when solving numerical tasks. to understand how the model reasons through numerical time-series problems, we manually analyzed its reasoning tokens. ## conclusion this paper introduced a set of 18 evaluation tasks for assessing the interpretation abilities of llms on numerical time-series. our benchmark focuses on four aspects: 1) event detection-identifying specific values such as maxima and minima; 2) computation-averaging and summation; 3) pairwise comparison-comparing values across time; and 4) inference-imputation and forecasting. each task expects a numeric answer, making evaluation fully automatic and comparable across models and prompts. our experiments show that even state-ofthe-art llms still struggle with multi-step reasoning, lose accuracy when the calculation must stay within a specific range, gain inconsistent benefits from instruction turning, and interpolation is more easily than forecasting; models with explicit reasoning steps handle the hardest tasks best. these findings establish a clear baseline and highlighting the remaining gap between fluent text generation and dependable numeric precision.