LLM -Mixer: Multiscale Mixing in LLM s for Time Series Forecasting (2025)

Abstract: Time series forecasting is a challenging task, especially when dealing with data that contains both short-term variations and long-term trends. In this study, we introduce LLM-Mixer, a novel framework that combines multiscale time-series decomposition with the power of pre-trained Large Language Models (LLMs). LLM-Mixer breaks down time-series data into multiple temporal resolutions using downsampling and processes these multiscale representations with a frozen LLM, guided by a carefully designed text prompt that encodes information about the dataset’s features and structure. To understand the role of downsampling, we conduct a detailed analysis using Neural Tangent Kernel (NTK) distance, showing that incorporating multiple scales improves the model’s learning dynamics.We evaluate LLM-Mixer across a diverse set of forecasting tasks, including long-term multivariate, short-term multivariate, and long-term univariate scenarios. Experimental results demonstrate that LLM-Mixer achieves competitive performance compared to recent state-of-the-art models across various forecasting horizons.

## introduction & related work time series forecasting is essential in numerous fields, including finance @xcite , energy management @xcite , healthcare @xcite , climate science @xcite , and industrial operations @xcite . traditional forecasting models, such as autoregressive integrated moving average (arima) @xcite and exponential smoothing techniques @xcite , are widely used for straightforward predictive tasks. however, these models assume stationarity and linearity, which limit their effectiveness when applied to complex, nonlinear, and multivariate time series often found in real-world scenarios @xcite . the advent of deep learning has significantly advanced time series forecasting. cnns @xcite @xcite have been utilized for capturing temporal patterns, while rnns @xcite @xcite are adept at modeling temporal state transitions. however, both cnns and rnns have limitations in capturing long-term dependencies @xcite @xcite . recently, transformer architectures @xcite have demonstrated strong capabilities in handling both local and long-range dependencies, making them suitable for time series forecasting @xcite @xcite . in parallel, pre-trained llms such as gpt-3 (brown, 2020), @xcite , and llama @xcite have achieved remarkable generalization in natural language processing tasks @xcite due to capabilities of few-shot or zero-shot transfer learning (brown, 2020), multimodal knowledge @xcite and reasoning @xcite . these models are now being applied across various fields, including computer vision @xcite , healthcare @xcite , and finance @xcite . recently, a few studies have explored using llms for time series forecasting due to their impressive capabilities @xcite @xcite . however, adapting llms to time series data presents challenges because there are significant differences between token-based text data and continuous time series data @xcite . llms are built to handle discrete tokens, which limits their ability to capture the continuous and often irregular patterns found in time series data. additionally, time series data has multiple time scales, from short-term fluctuations to long-term trends, making it difficult for traditional llms to capture all these patterns at once. llms typically process fixed-length sequences, which means they may only capture short-term dependencies if the sequence length (i.e., the window of time steps) is small. however, extending the sequence length to capture long-term trends increases computational costs and may dilute the model's ability to focus on short-term fluctuations within the same sequence. previous studies using llms on time series data have mostly fed the original or a single sequence directly into a frozen llm, making it hard for the model to fully understand these sequences @xcite @xcite . to address this, we introduce llm-mixer, which breaks down the time series data into multiple time scales. by creating various resolutions (figure 1 ), our model can capture both short-term details and long-term patterns more effectively. since the llm remains frozen during training, the multiscale decomposition provides a diverse range of temporal information, helping the model better understand complex time series data. our contributions of this paper are: (1) we propose llm-mixer, a new method that adapts llms for time series forecasting by breaking down the data into different time scales, helping the model capture both short-term and long-term patterns. (2) our method creates multiple versions of the time series at different resolutions which helps the llm to understand complex time series data more effectively. (3) empirical results show that llm-mixer achieves competitive performance, improves forecasting accuracy on both multivariate and univariate data, and works effectively for both short-term and long-term forecasting tasks. ## llm mixer preliminaries: in multivariate time series forecasting, we are given historical data @xmath0, . . . , x t } ∈ r t ×m , where t is the number of time steps and m is the number of features. the goal is to predict the future values for the next k time steps, denoted as @xmath1, . . . , x t +k } ∈ r k×m . for convenience, let x t,: represent the data at time step t, and x :,m represent the full time series for variable @xmath2. now, suppose we have a prompt p, which includes textual information about the time sequence (e.g., source, features, distribution, statistics). we use a pre-trained language model f(•) with frozen parameters θ, then the prediction is made as follows: ŷ = f(x, p; θ, φ) here φ is a small set of trainable parameters to adjust the model for the specific forecasting task. multi-scale view of time data: time series data contains patterns at various levels-small scales capture detailed changes, while larger scales highlight overarching trends @xcite . analyzing data at multiple scales helps to understand these complex patterns @xcite . following @xcite , we apply a multiscale mixing strategy. first, we downsample the time series x into τ scales using average pooling, resulting in a multiscale representation @xmath3, x 1 , . . . , x τ }, where each x @xmath4. here, x 0 contains the finest temporal details, while x τ captures the broadest trends. next, we project these multiscale series into deep features using three types of embeddings: token, temporal, and positional embeddings. token embeddings are obtained via 1d convolutions @xcite , temporal embeddings represent day, week, and month @xcite , and positional embeddings encode sequence positions. we then use stacked past-decomposable-mixing (pdm) blocks by following the framework from @xcite to mix past information across different scales. pdm works by breaking down complex time series data into separate seasonal and trend components at multiple scales, allowing for targeted processing of each component by using the framework from @xcite . for the l-th layer, pdm is defined as prompt embedding: prompting is an effective technique for guiding llms by using task-specific information @xcite . studies like @xcite show promising results by treating time series inputs as prompts for forecasting. @xcite further improved time series predictions by embedding dataset descriptions in the prompts. inspired by this, we embed dataset descriptions (e.g., features, statistics, distribution) as prompts. we use a textual description for all samples in a dataset, as suggested by @xcite , and generate its embedding using the pre-trained llm's word embeddings, denoted by @xmath5, where v is the llm's vocabulary size. this prompt leverages the llm's semantic knowledge to improve the prediction task. multi-scale mixing in llm: after processing through l pdm blocks, we obtain the multiscale past information x l . since different scales focus on different variations, their predictions offer complementary strengths. to fully utilize this, we concatenate all the scales and input them into a frozen pre-trained llm along with the prompt as f(e ⊕ x l ). finally, a trainable decoder (simple linear transformation) with parameters φ is applied to the last hidden layer of the llm to predict the next k future time steps. ## experiments we evaluate our llm-mixer on several datasets commonly used for benchmarking long-term and short-term multivariate forecasting and compared with sota baselines. for long-term forecasting, we use the ett datasets (etth1, etth2, ettm1, ettm2) from @xcite , as well as the weather, electricity, and traffic datasets from @xcite . for short-term forecasting, we use the pems dataset @xcite , which consists of four public traffic network datasets (pems03, pems04, pems07, and pems08) with time series collected at various frequencies. we used roberta-base @xcite as a mediumsized language model and llama2-7b @xcite as a large language model as the backbone of our framework. baselines we compare our model with wellestablished time-series forecasting baselines such as timemixer @xcite , itransformer @xcite , timellm @xcite , rlinear @xcite , scinet @xcite , timesnet @xcite , tide @xcite , dlinear @xcite , patchtst @xcite , fedformer @xcite , stationary @xcite , estformer @xcite , lightts @xcite , and autoformer @xcite . additionally, we include llmbased systems such as timellm @xcite and gpt2ts @xcite . for multivariate time series forecasting, we follow the setup of @xcite . for short-term forecasting, we adopt the settings from @xcite , and for univariate forecasting, we adhere to the approach in @xcite . implementation details all experiments in this work are implemented using pytorch. we utilize the hugging face library for the llm model. experiments were conducted on an nvidia h100 gpu with 80 gb ram. hyperparameters: for long-term experiments, a look-back window of 96 is used to predict the next 96 (future context) and 192 (forecast horizons), while short-term experiments use windows of 24 and 48. all experiments run for 10 epochs with a batch size of 64 for roberta and a batch size of 8 with gradient accumulation of 4 for llama2. the adam optimizer is employed with default settings (β 1 , β 2 ) = (0.9, 0.999) and a learning rate of 0.0001. downsampling levels range from 2 to 5 across all experiments. for the baseline models, we have followed their original works, with differences only in batch size and learning rate to align with our experimental setup. multivariate forecasting results: llm-mixer demonstrates competitive performance in multivariate long forecasting, as shown in mse and mae values across most datasets, particularly excelling on etth1, etth2, and electricity. compared to other models such as time-llm, timemixer, and patchtst, llm-mixer performs favorably, showing that its design effectively captures both short-and long-term dependencies. notably, llm-mixer also exhibits robustness on challenging datasets such as traffic, where it outperforms several baseline models. these results highlight the efficacy of the llm-mixer in handling complex temporal patterns over extended horizons. short-term forecasting results: in ## conclusion this work introduces the llm-mixer, a novel framework that combines multiscale time-series decomposition with pre-trained llms for improved forecasting. by leveraging multiple temporal resolutions, the llm-mixer effectively captures both short-and long-term patterns, enhancing the model's predictive accuracy. our experiments demonstrate that the llm-mixer achieves competitive performance across various datasets, outperforming recent state-of-the-art methods.