Evaluating Large Language Models on Time Series Feature Understanding: A Comprehensive Taxonomy and Benchmark (2024)

Abstract: Large Language Models (LLMs) offer the potential for automatic time series analysis and reporting, which is a critical task across many domains, spanning healthcare, finance, climate, energy, and many more. In this paper, we propose a framework for rigorously evaluating the capabilities of LLMs on time series understanding, encompassing both univariate and multivariate forms. We introduce a comprehensive taxonomy of time series features, a critical framework that delineates various characteristics inherent in time series data. Leveraging this taxonomy, we have systematically designed and synthesized a diverse dataset of time series, embodying the different outlined features, each accompanied by textual descriptions. This dataset acts as a solid foundation for assessing the proficiency of LLMs in comprehending time series. Our experiments shed light on the strengths and limitations of state-of-the-art LLMs in time series understanding, revealing which features these models readily comprehend effectively and where they falter. In addition, we uncover the sensitivity of LLMs to factors including the formatting of the data, the position of points queried within a series and the overall time series length.

## introduction time series analysis and reporting are crucial in diverse fields like healthcare, finance, and climate @xcite . the recent progress in large language models (llms) opens exciting possibilities for automating these processes. while recent studies have explored adapting llms for specific time series tasks, such as seizure localization in eeg time series @xcite , cardiovascular disease diagnosis in ecg time series @xcite , weather and climate data understanding @xcite , and explainable financial time series forecasting @xcite , a systematic evaluation of general-purpose llms' inherent capabilities in understanding time series data is lacking. one notable example of domain-specific application is the biosignal copilot framework presented by @xcite , which focuses on leveraging llms for clinical report generation from biomedical signals. this paper aims to fill this gap by uncovering the strengths and weaknesses of general-purpose llms in time series understanding, without any domain-specific fine-tuning. our focus is on assessing their potential for a key downstream task: time series annotation and summarization. by understanding the baseline capabilities of llms, practitioners can identify areas where these models are readily applicable and areas where targeted finetuning efforts may be necessary to improve performance. to systematically evaluate the performance of general-purpose llms on generic time series understanding, we propose a taxonomy of time series features for both univariate and multivariate time series. this taxonomy serves as a structured framework for evaluating llm performance and provides a foundation for future research in this domain. based on this taxonomy, we have created a diverse synthetic dataset of time series that covers a wide range of features, each accompanied by qualitative and quantitative textual descriptions. our evaluations focus on tasks directly relevant to time series annotation and summarization, such as feature detection, classification, and data retrieval as well as arithmetic reasoning. additionally, we assess the llms' ability to match textual descriptions to their corresponding time series, leveraging the textual descriptions in our dataset. these findings will be instrumental for developing llm-powered tools for automated time series annotation and summarization, ultimately enhancing data analysis and reporting workflows across diverse domains. our contributions are three-fold: language models for time series recent progress in time series forecasting has capitalized on the versatile and comprehensive abilities of llms, merging their language expertise with time series data analysis. this collaboration marks a significant methodological change, underscoring the capacity of llms to revolutionize conventional predictive methods with their advanced information processing skills. notably, @xcite have set benchmarks for pre-trained llms such as gpt-3 and llama2 by assessing their capabilities for zero-shot forecasting. similarly, (xue and salim, 2023) introduced prompcast, adopting a novel approach by treating forecasting as a question-answering activity, utilizing strategic prompts. further, @xcite delved into the potential of llms for generating explainable forecasts in financial time series, tackling inherent issues like cross-sequence reasoning, integration of multi-modal data, and interpretation of results, which pose challenges in conventional methodologies. additionally, @xcite demonstrated that leveraging frozen pre-trained language models, initially trained on vast corpora, for time series analysis could achieve comparable or even state-of-the-art performance across various principal tasks in time series analysis including imputation, classification and forecasting. recent advancements in the application of llms to biomedical time series data have also shown promise in the automated generation of clinical reports. @xcite introduce biosignal copilot, a system that leverages llms for drafting reports from biomedical signals, such as electrocardiograms (ecgs) and electroencephalograms (eegs). their work highlights the importance of domain-specific feature extraction in facilitating llm understanding of time series data, aligning with our work on developing a comprehensive taxonomy of time series features to enhance llm interpretability and analysis in various applications. notably, their focus on automatic report generation from the processed signals serves as a specific downstream task, further emphasizing the need for a systematic evaluation of llms' ability to understand and extract relevant features from time series data, such as the one presented in this work. llms for arithmetic tasks despite their advanced capabilities, llms face challenges with basic arithmetic tasks, crucial for time series analysis involving quantitative data @xcite . research has identified challenges such as inconsistent tokenization and token frequency as major barriers @xcite . innovative solutions, such as llama2's approach to digit tokenization @xcite , highlight ongoing efforts to refine llms' arithmetic abilities, enhancing their applicability in time series analysis. ## conclusion in conclusion, we provide a critical examination of general-purpose large language models (llms) in the context of time series understanding. through the development of a comprehensive taxonomy of time series features and the synthesis of a diverse dataset that encapsulates these features, including qualitative and quantitative textual descriptions for each time series, we have laid a solid foundation for evaluating the capabilities of llms in understanding and interpreting time series data. our systematic evaluation sheds light on the inherent strengths and limitations of these models, offering valuable insights for practitioners aiming to leverage llms in time series understanding. recognizing the areas of weakness and strength in general-purpose llms' current capa-bilities allows for targeted enhancements, ensuring that these powerful models can be more effectively adapted to specific domains. in the future, we plan to study the performance of llms on real-world time series datasets to assess the generalizability of the proposed framework. this will involve testing llms on diverse datasets from various domains, such as finance, healthcare, and climate science. additionally, future work should expand the analysis challenges llms face with multivariate time series data, including the ability to identify and interpret relationships between multiple series, such as correlation, cross-correlation, and dynamic conditional correlation. understanding these challenges will be crucial for developing more effective llms for complex time series analysis. finally, evaluating llms in few-shot settings is an important area for future work, as it can reveal the models' ability to learn and generalize from limited time series data. this can be particularly valuable in domains where labeled data is scarce or expensive to obtain.