Language Models Still Struggle to Zero-shot Reason about Time Series (2024)

Abstract: Time series are critical for decision-making in fields like finance and healthcare. Their importance has driven a recent influx of works passing time series into language models, leading to non-trivial forecasting on some datasets. But it remains unknown whether non-trivial forecasting implies that language models can reason about time series. To address this gap, we generate a first-of-its-kind evaluation framework for time series reasoning, including formal tasks and a corresponding dataset of multi-scale time series paired with text captions across ten domains. Using these data, we probe whether language models achieve three forms of reasoning: (1) Etiological Reasoning—given an input time series, can the language model identify the scenario that most likely created it? (2) Question Answering—can a language model answer factual questions about time series? (3) Context-Aided Forecasting–does highly relevant textual context improve a language model’s time series forecasts? We find that otherwise highly-capable language models demonstrate surprisingly limited time series reasoning: they score marginally above random on etiological and question answering tasks (up to 30 percentage points worse than humans) and show modest success in using context to improve forecasting. These weakness showcase that time series reasoning is an impactful, yet deeply underdeveloped direction for language model research. We also make our datasets public to support further research in this direction.

## introduction time series measure how systems change over time and contain information that is uncommon in language. they are a critical data modality in healthcare @xcite , finance @xcite , agriculture @xcite , economics @xcite , political science @xcite , astronomy (benson 1 all data and code are available at https://github.com/ behavioral-data/tsandlanguage et al., 2020), signal processing @xcite , and beyond. as the scientific community races to bring language models (lms) to these domains, we must ensure lms can support decisions about these sources of valuable information. if successful, lms could perform novel tasks like citing patterns and events in time series as evidence for observations and inferences, drawing interpretable conclusions from complex dynamical systems, or learning to recognize and respond to temporal patterns. several recent works have shown that lms can be used for zero-shot time series tasks, though nearly all focus on forecasting. these works typically forecast by structuring historical observations as raw text @xcite @xcite or images @xcite . this is promising work, and suggests language models may someday demonstrate the same remarkable zero-shot performance that they do with text and images. but it remains unknown whether non-trivial forecasting implies that lms can reason about time series, as opposed to simply generating matching temporal patterns that appear in their inputs. in fact, recent works indicate that a lm's ability to generate data does not imply deeper reasoning @xcite . in this work, we develop, apply, and release a framework to ultimately find that despite excitement about using lms for time series analysis, current language models are remarkably bad at zero-shot time series reasoning. we propose three components of time series reasoning. first, for a lm to reason about time series it must be able to consider the etiology (the set of possible causes) of a time series through etiological reasoning (figure 1(a) ). second, a successful model should excel at question answering and be able to address queries about time series and how they relate to one another (figure 1(b) ). finally, time series reasoning implies context-aided forecasting, wherein a language model can leverage its world model and natural language context to aid in forecasting (figure 1(c) ). to evaluate lms we create a first-of-its kind dataset that contains 230k time series multiple choice questions and 8.7k pairs of synthetic time series and text captions that describe the series and the context in which it was observed (section 3). these data span a diverse set of time series scenarios across including health data, transport and traffic trends, finance, and more. we use this dataset to evaluate etiological reasoning by tasking models to select the most probable time series caption given the observed time series (section 4) and find that human annotators outperform language models by a margin of up to thirty percentage points, with otherwise strong language models like gpt-4 barely doing better than random chance. then, we test models on a question answering task by augmenting our dataset to include 230k question-answer pairs (section 5). again, we find that human annotators significantly outperform language models, indicating that language models have limited capacity to interpret the information in time series. finally, we evaluate language models on a context-aided forecasting task (section 6). we find that even with text descriptions of what will happen in future, gpt-4 struggles to incorporate this information, resulting in negligible improvements over models without additional context. taken as a whole these results indicate that despite modest time series forecasting ability, current language models fail to reason about these ubiquitous, critical data despite considerable human performance on the same tasks. ## forms of time series reasoning here we propose a rigorous (though nonexhaustive) definition of time series reasoning. consider a univariate uniformly-sampled time series of n observations, @xmath0, @xmath1. suppose that an autoregressive language model m is able to represent this time series as input and produce time series observations and text as outputs. that is, m estimates the probability p of an output token sequence y given some context tokens c and the time series: p m (y |x, c) = m (y, x, c). definition 2.1 (etiological reasoning). etiological reasoning is the property by which language models are able to hypothesize about the cause of a time series. that is, given a time series x, textual instructions as context c, a correct description d + of how x was generated and an incorrect description d -, a language model should assign higher probability to d + : language models that can reason about time series should also be able to answer questions about the behavior and implications of a time series. definition 2.2 (question answering). we define question answering as a model's ability to use information in the time series x to interpret queries about the time series or the events surrounding the scenario it represents. for the sake of evaluation, the questions should be time-series-dependent-correct answers should be unattainable without interpreting x. for example, given an ecg, a dependent question might be, "does this signal demonstrate atrial fibrillation?" while a trivially non-dependent question would be, "who was the first president of the united states?" formally, given a question q and an answer a + , the model should predict a language model should be able to exploit this information. in a multiple-choice setting, given a correct answer a + and an incorrect answer a -: finally, for an lm to reason about time series it should be able to integrate relevant information from text into forecasts about how the time series will behave in the future. ## dataset evaluating these forms of time series reasoning requires pairs of time series and highly-relevant text descriptions. without a strong relationship between the two, it is impossible to determine if a model's failure to reason about time series is due to poor fundamental capabilities or a poorly-designed evaluation. however, there is no general corpus of time series and natural language descriptions that captures such relationships (section 7.1). to address this challenge, here we contribute a firstof-its-kind dataset of synthetic multi-domain time series and highly relevant text captions. ## etiological reasoning: near random performance by defining time series reasoning (section 2) and creating our first-of-its kind dataset of time series and associated captions (section 3) we can evaluate the capacity of lms to reason about these ubiquitous data. reasoning implies an ability to provide explanations for observed phenomena. in our context if a model can reason about a time series then it should be able to hypothesize about how that series was generated. for example, given a time series with a strong daily seasonality "sunlight intensity" is a more likely description than "nvidia stock price since 1999." we evaluate etiological reasoning by tasking an llm to select the correct time series caption from a set of four, with three incorrect captions (figure 1(a) ). we sampled incorrect descriptions by randomly selecting three captions from the remainder of the dataset. to encourage the models to focus on the time series itself and not on metadata like the series' units or start and end timestamps we only provided the values of the time series. for pure language models, time series were encoded into text using the method from @xcite which uses separate schemes for llama and gpt-4. details on this method are available in section a.1. we also experiment with representing time series as images of plots and passing them to gpt-4-vision, as in as @xcite . a natural question is whether text is the correct way to represent a time series. to answer this, we experimented with five other representations (including time series as audio, spectrograms, precomputed embeddings, and images) by training existing multimodal models on our data and found no difference in performance (section b.1). our results show that all models perform remarkably poorly relative to the human baseline (66.1% accuracy, section 3.2), with some models performing at or near random chance (e.g llama with 27.3% accuracy) ( these results indicate that current language models are poor zero-shot judges of time series etiology. ## question answering: trailing behind human-level proficiency a lm that can reason about time series should be able to answer questions about a time series and the implications of the scenario it describes. to properly evaluate this property it should not be possible to answer the questions without the time series. this avoids misleading performance estimates observed in visual question answering with models performing well even without the associated image @xcite . a good candidate for these questions are counterfactual "what-if"-style queries that ask the lm to interpret how the time series might be different if its related scenario were changed. for example, given a time series of coffee shop sales over the course of a day with a peak at 2pm, a good "what-if" question might be, "if half as many customers visited the shop at noon, would the peak sales change?" we evaluate this ability by presenting lms with multiple choice questions (mcqs) with four options -one correct and three incorrect. we evaluate question answering using the same techniques as etiological reasoning (section 4). additionally, in section b.4 we experiment with representing the time series as plain, unformatted text and show no appreciable difference. human performance was again assessed using a team of ten data scientists who annotated 500 time series plots using the exact same data as the lms (metadata, time series [as a plot], and the short description). ## context-aided forecasting we next evaluate whether lms can leverage relevant textual context when forecasting future time series values. we build on recent works that find lms can non-trivially zero-shot forecast time series @xcite . using the same zero-shot forecasting method as llm-time @xcite , we experiment with prepending different corresponding textual context alongside the time series. we randomly select 2000 time series with their captions, descriptions, and metadata, feed the first 80% as context and forecast the remaining 20% of the timesteps. this textual context contains highly-relevant in-formation, including future information about the series' behavior. to understand how well these methods compare to a simple baseline we include the "predict median" baseline, which simply computes the median of the first 80% of a time series' values then repeats it for the forecasting window. we measure forecasting success using the common metrics mean absolute error (mae) and mean squared error (mse). since the values of the time series in our dataset span several orders of magnitude we min/max and z-score normalize values before computing these metrics so that error on high-magnitude series does not dominate perceived model performance. highly-relevant captions barely change lm forecasts. as shown in figure 5 , adding all textual context only marginally improves mae. of 2,000 zero-shot samples, only 1,040 show improvement in mae when the full context is shown and in the remaining time series mae increases. an example is illustrated in figure 4 , showing that the lm ignores potentially useful information in the context. we also experimented with other combinations of metadata, characteristics, and descriptions and found that adding more information gradually improves performance, but overall performance remains below or comparable to the weak "predict median" baseline (section e). we include an example where the model appears to integrate context in @xcite this lack of improvement is surprising and demonstrates a clear gap in these lm-powered methods' capacities to leverage relevant text when forecasting time series. further, neither forecasting method clearly outperforms the simple "median prediction" baseline. we note that because our series were intentionally designed to contain interruptions from external events (section 3) median prediction is a particularly weak baseline on our dataset. this experiment shows that current lms largely fail to use context to inform forecasting. ## limitations one limitation of this work is its reliance on synthetic data. while we go to great care to manually validate the quality of the data (section 3.2) and provide examples of our scenarios (figure a .2) we nonetheless recognize that questions may arise about our data's realism. it is important to remember that no "real" dataset of diverse time series and highly relevant text exists. by providing our dataset, tasks, and evaluations we provide progress that would not be possible without synthetic data. we leave it to future work to mine and document a similar "real" dataset. a related limitation is that because gpt-4 was used to generate data (and some questions in section 5) it is possible that the performance of this model (and this model only) is an over-estimate of true ability. we provide evidence to this end in section 5.2, where we show that manually perturbed mcqs are harder for gpt-4 but just as difficult for humans. nevertheless, even if we assume that this is an overestimate the substantial gap between human and gpt-4 performance on all tasks indicates significant room for improvement. ## conclusion we identified three forms of time series reasoning and used them to create a first-of-its-kind dataset of time series and highly relevant text. we then used this dataset to assess etiological reasoning, question answering, and context-aided forecasting. given the substantial gap between language model and human performance on the first two tasks, and mediocre performance on the third, we identified opportunities for the nlp community to develop models that can deeply reason about these critical data.