A Picture is Worth A Thousand Numbers: Enabling LLM s Reason about Time Series via Visualization (2025)

Abstract: Large language models (LLMs), with demonstrated reasoning abilities across multiple domains, have been largely underexplored fortime-series reasoning (TsR), which is ubiquitous in the real world. In this work, wepropose TimerBed, the first comprehensivetestbed for evaluating LLMs’ TsR performance.Specifically, TimerBed includes stratified reasoning patterns with real-world tasks, diversecombinations of LLMs and reasoning strategies, and various supervised models as comparison anchors. We perform extensive experiments with TimerBed, test multiple current beliefs, and observe the initial failuresof LLMs in TsR, as evidenced by the ineffectiveness of zero shot (ZST) and performancedegradation of few shot in-context learning(ICL). Further, we identify one possible rootcause: the numerical modeling of data. Toaddress this, we propose a prompt-based solution VL-Time, with visualization-modeled dataand language-guided reasoning. Experimental results demonstrate that VL-Time enablesmultimodal LLMs to be non-trivial ZST andpowerful ICL reasoners for time series, achieving about 140% average performance improvement and 99% average token costs reduction.TimerBed and VL-Time are available at https://github.com/AdityaLab/DeepTime/.

## introduction with the rapid advancement of large language models (llms), they have exhibited powerful reasoning abilities with unique interpretability across multiple domains, including logic, mathematics, and symbolic reasoning @xcite @xcite @xcite . time series, which records the dynamic evolution of data over time, is common in the real world. time-series reasoning (tsr) @xcite has extensive real-world applications, such as pathological diagnosis @xcite , marine biological monitoring @xcite , and human activity recognition @xcite . therefore, in this paper, we aim to explore the question: "can llms be reused for time-series reasoning? if not, how can we enable them?" however, this question has remained largely unexplored. the first key obstacle is "how to effectively evaluate llms on tsr?". existing works @xcite face limitations at each component of evaluation: (1) task structure and datasets. they directly mix multiple tasks without a clear taxonomy of reasoning patterns, making it difficult to understand llms' limitations in tsr; they rely on synthetic data, which fails to accurately represent real-world scenarios. (2) llms reasoning strategies: they only adopt the zeroshot setting, ignoring chain-of-thought @xcite and few-shot in-context-learning @xcite strategies, which are widely used to enhance llms' reasoning abilities. (3) comparison anchors foot_0 : they either lack comparison anchor, thus failing to quantify success, or adopt humanlevel performance, which is costly and difficult to scale to other datasets. to address these issues, we propose timerbed, the first comprehensive and hierarchical testbed for evaluating llms reasoning capabilities on timeseries tasks. using timerbed, we conduct extensive experiments to assess llms' tsr effectiveness, test several current beliefs about llms, and verify the initial failure of llms for tsr. inspired by the fact that humans rely on visualization to analyze complex data @xcite , we propose a possible reason for llms' failure: the numerical modeling of time-series data, which are typically high-dimensional. we empirically validate this hypothesis and introduce a sim-ple yet effective solution, vl-time, which leverages visualization for data modeling and language for reasoning guidance, thereby unlocking the potential of multimodal llms for time-series reasoning. main contributions are summarized as : in this work, we validate that visualization and few-shot icl are key to enabling llms to reason about time series. we envision visual encoding and icl can be important ingredients for the nextgeneration models to move beyond forecasting. additional related works are provided in appendix a. limitations are discussed in appendix b. in this section, we first define time-series reasoning following existing llms reasoning definitions @xcite and then introduce the constructed testbed. let the input numerical time series be @xmath0, where l is the series length and d is the feature dimension. let the textual question be q and the correct answer be a. the goal of time-series reasoning is to derive a reasoning path p , a sequence of intermediate steps, that leads to a given q and x, using zero or a few samples. as discussed in section 1, existing evaluation efforts have limitations at every component of benchmarking. to address the question of "how to evaluate and understand the actual tsr capabilities of llms?", we present timerbed, the first-of-itskind testbed for evaluating llm reasoning about time series, as shown in figure 1 . we further detail each component of timerbed, including reasoning patterns with tasks, llms and reasoning strategies, as well as comparison anchors. ## failure analysis: the numerical modeling of data. we further investigate "what causes the failures of llms for tsr?" focusing on the two aforementioned anomalies, we verify that a key cause is the numerical modeling of time-series data, following most existing works @xcite @xcite , detailed as follows: numerical modeled @xmath1. we begin by checking the feature extraction plans of llms. through prompting, we present verbalized feature extraction plans of each llm for different tsr tasks in appendix j. we notice that these plans are similar to each other and, in most cases, are reasonable. this indicates that llms possess useful knowledge for zero-shot tsr. however, most of these planned features are difficult to extract from numerical time series and can be classified into four types: ## vl-time: enabling llms reasoning about time series via visualization this section aims to answer: "how can we address the original failure of llms in tsr?" by focusing on the two observed anomalies: zst ineffectiveness and icl performance degradation. high-level idea. inspired by the fact that humans rely on visualization to analyze complex data @xcite , we propose to replace existing numerical modeling with visualizationbased modeling. in the context of llms reasoning about time series, the motivation is detailed as: framework and details. as shown in figure 4 , we propose a prompt-based solution, named vl-time, which aims to empower any multimodal llms to reason about time series by data modeling via visualization and task reasoning guided by language. specifically, vl-time adopts a "planthen-solve" framework @xcite inspired by human reasoning processes @xcite to prompt multimodal llms for tsr. the key difference is that vl-time replaces numerical modeling with visualization modeling for time-series data, which enhances feature extraction and reduces context length. vl-time further divides the entire reasoning process into planning and solving stages, mimicking the behavior of human experts. a full example is provided in section ?????????f. for each task, the planning stage needs to be executed only once. the planning stage. vl-time prompts llms to act as domain experts and make two key plans based on the task description: (1) choosing between time-domain or frequency-domain visualization, and (2) proposing the reasoning clues. to facilitate effective feature extraction, our visualization designs include time stamps on the x-axis, semantic labels on the y-axis, and different colors and textual legends for each variable or frequency. figure ???13 and ???12 show examples of our visualization. notably, the planning stage only needs to run once for each task, i.e., shared across samples. the solving stage. vl-time prompts multimodal llms to perform tsr using visualized data (i.e., "picture") and verbalized reasoning clues (i.e., "hints") derived from the planning stage. ## experiments for proposed solution this section aims to answer the question:"how well can vl-time unlock multimodal llms' reasoning ability for time series?" (1) vl-time enables gpt-4o to perform non-trivial zero-shot tsr, with up to 40% relative improvement and consistently outperforming random guessing on all tasks. (2) vl-time enables gpt-4o to achieve powerful few-shot tsr, with up to 433% relative improvement, surprisingly surpassing all supervised models in simple and complex deterministic tsr, and matching their performances in probabilistic tsr. here, few-shot refers to fewer than sixexamples per class. we bold the best result for each task. ## conclusion in this work, we explore llms reasoning about time series with a two-fold contribution: evaluation and methodology. we present the first comprehensive evaluation suite, conduct extensive evaluations, provide insights and verify the initial failures. we identify the direct numerical data modeling as one key issue and propose a prompt-based solution with visualized data modeling. we empirically validate that visualization of data and few-shot icl are key to enabling llms reasoning about time series. we expect that visual encoding and icl could be important ingredients for future time-series models. manqing dong, hao huang, and longbing cao. 2024. can llms serve as time series anomaly detectors? arxiv preprint arxiv:2408.03475. wenjie du, jun wang, linglong qian, yiyuan yang, fanxing liu, zepu wang, zina ibrahim, haoxin liu, zhiyuan zhao, yingjie zhou, et al. 2024. tsi-bench: benchmarking time series imputation. arxiv preprint arxiv:2406.12747. abhimanyu dubey, abhinav jauhri, abhinav pandey, abhishek kadian, ahmad al-dahle, aiesha letman, akhil mathur, alan schelten, amy yang, angela fan, et al. 2024. the llama 3 herd of models. arxiv preprint arxiv:2407.21783. valentin fuster, lars e rydén, david s cannom, harry j crijns, anne b curtis, kenneth a ellenbogen, jonathan l halperin, jean-yves le heuzey, g neal kay, james e lowe, et al. 2006. acc/aha/esc 2006 guidelines for the management of patients with atrial fibrillation: a report of the american college of cardiology/american heart association task force on practice guidelines and the european society of cardiology committee for practice guidelines (writing committee to revise the 2001 guidelines for the management of patients with atrial fibrillation): developed in collaboration with the european heart rhythm association and the heart rhythm society. circulation, 114(7):e257-e354. nate gruver, marc finzi, shikai qiu, and andrew g wilson. 2024. large language models are zero-shot time series forecasters. advances in neural information processing systems, 36. neel guha, julian nyarko, daniel ho, christopher ré, adam chilton, alex chohlas-wood, austin peters, brandon waldon, daniel rockmore, diego zambrano, et al. 2024. legalbench: a collaboratively built benchmark for measuring legal reasoning in large language models. advances in neural information processing systems, 36. manfred jaeger. 1994. probabilistic reasoning in terminological logics. in principles of knowledge representation and reasoning, pages 305-316. elsevier. yixing jiang, jeremy irvin, ji hun wang, muhammad ahmed chaudhry, jonathan h chen, and andrew y ng. 2024. many-shot in-context learning in multimodal foundation models. arxiv preprint arxiv:2405.09798. ming jin, shiyu wang, lintao ma, zhixuan chu, james y. zhang, xiaoming shi, pin-yu chen, yuxuan liang, yuan-fang li, shirui pan, and qingsong wen. 2024a. time-llm: time series forecasting by reprogramming large language models. in the twelfth international conference on learning representations. yiqiao jin, minje choi, gaurav verma, jindong wang, and srijan kumar. 2024b. mm-soc: benchmarking multimodal large language models in social media platforms. in acl. harshavardhan kamarthi and b. aditya prakash. 2024. large pre-trained time series models for crossdomain time series analysis tasks. jun kimura. 2013. 361types of electromyographic abnormalities. in electrodiagnosis in diseases of nerve and muscle: principles and practice. oxford university press. takeshi kojima, shixiang shane gu, machel reid, yutaka matsuo, and yusuke iwasawa. 2022. large language models are zero-shot reasoners. advances in neural information processing systems, 35:22199-22213. gregory yh lip and hung-fat tse. 2007. management of atrial fibrillation. the lancet, 370(9587):604-618. haoxin liu, harshavardhan kamarthi, lingkai kong, zhiyuan zhao, chao zhang, b aditya prakash, et al. 2024a. time-series forecasting for out-ofdistribution generalization using invariant learning. in forty-first international conference on machine learning. haoxin liu, shangqing xu, zhiyuan zhao, lingkai kong, harshavardhan kamarthi, aditya b sasanur, megha sharma, jiaming cui, qingsong wen, chao zhang, et al. 2024b. time-mmd: multi-domain multimodal dataset for time series analysis. in the thirtyeight conference on neural information processing systems datasets and benchmarks track. haoxin liu, zhiyuan zhao, jindong wang, harshavardhan kamarthi, and b. aditya prakash. 2024c. lstprompt: large language models as zero-shot time series forecasters by long-short-term prompting. in findings of the association for computational linguistics, acl 2024, bangkok, thailand and virtual meeting, august 11-16, 2024, pages 7832-7840. association for computational linguistics. jiashuo liu, zheyan shen, yue he, xingxuan zhang, renzhe xu, han yu, and peng cui. 2021. towards out-of-distribution generalization: a survey. arxiv preprint arxiv:2108.13624. nelson f liu, kevin lin, john hewitt, ashwin paranjape, michele bevilacqua, fabio petroni, and percy liang. 2024d. lost in the middle: how language models use long contexts. transactions of the association for computational linguistics, 12:157-173. yong liu, tengge hu, haoran zhang, haixu wu, shiyu wang, lintao ma, and mingsheng long. 2024e. itransformer: inverted transformers are effective for time series forecasting. in the twelfth international conference on learning representations. haoyu lu, wen liu, bo zhang, bingxuan wang, kai dong, bo liu, jingxiang sun, tongzheng ren, zhuoshu li, yaofeng sun, et al. 2024. deepseek-vl: towards real-world vision-language understanding. arxiv preprint arxiv:2403.05525. sarabeth m mathis, alexander e webber, tomás m león, erin l murray, monica sun, lauren a white, logan c brooks, alden green, addison j hu, roni rosenfeld, et al. 2024. evaluation of flusight influenza forecasting in the 2021-22 and 2022-23 seasons with a new target laboratory-confirmed influenza hospitalizations. nature communications, 15(1):6289. mike a merrill, mingtian tan, vinayak gupta, tom hartvigsen, and tim althoff. 2024. language models still struggle to zero-shot reason about time series. arxiv preprint arxiv:2404.11757. yuqi nie, nam h nguyen, phanwadee sinthong, and jayant kalagnanam. 2023. a time series is worth 64 words: long-term forecasting with transformers. in the eleventh international conference on learning representations. joel niklaus, lucia zheng, arya d mccarthy, christopher hahn, brian m rosen, peter henderson, daniel e ho, garrett honke, percy liang, and christopher manning. 2024. flawn-t5: an empirical examination of effective instruction-tuning data mixtures for legal reasoning. arxiv preprint arxiv:2404.02127. judea pearl and dana mackenzie. 2018. the book of why: the new science of cause and effect. basic books. ben shneiderman. 2003. the eyes have it: a task by data type taxonomy for information visualizations. in the craft of information visualization, pages 364-371. elsevier. zayne sprague, fangcong yin, juan diego rodriguez, dongwei jiang, manya wadhwa, prasann singhal, xinyu zhao, xi ye, kyle mahowald, and greg durrett. 2024. to cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning. arxiv preprint arxiv:2409.12183. lichao sun, yue huang, haoran wang, siyuan wu, qihui zhang, chujie gao, yixin huang, wenhan lyu, yixuan zhang, xiner li, et al. 2024. trustllm: trustworthiness in large language models. arxiv preprint arxiv:2401.05561. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, ł ukasz kaiser, and illia polosukhin. 2017. attention is all you need. in advances in neural information processing systems, volume 30. curran associates, inc.