LLM s learn governing principles of dynamical systems, revealing an in-context neural scaling law (2024)

Abstract: We study LLMs’ ability to extrapolate the behavior of various dynamical systems, including stochastic, chaotic, continuous, and discrete systems, whose evolution is governed by principles of physical interest. Our results show that LLaMA-2, a language model trained on text, achieves accurate predictions of dynamical system time series without fine-tuning or prompt engineering. Moreover, the accuracy of the learned physical rules increases with the length of the input context window, revealing an in-context version of a neural scaling law. Along the way, we present a flexible and efficient algorithm for extracting probability density functions of multi-digit numbers directly from LLMs.

## introduction since the introduction of the transformer architecture @xcite , large language models (llms) have shown a variety of unexpected emergent properties, such as program execution @xcite and multi-step reasoning @xcite . our work explores llms' ability to model the world by proposing a new perspective and empirical approach. inspired by the recent observations that llms are capable of in-context time series extrapolation without specific prompting or fine-tuning @xcite , we aim to quantify llms' ability to extrapolate stochastic dynamical systems. we find that, as the number of observed time steps increases, an llm's statistical prediction consistently converges to the ground truth transition rules underlying the system; leading to an empirical scaling law, as observed in figure 1 . our main contributions are as follows: • demonstrating llms' zero-shot ability to model the evolution of dynamical systems without instruction prompting; • implementing a computationally efficient framework called hierarchy-pdf to extract statistical information of a dynamical system learned by a transformer-based llm; • discovering a scaling law between the accuracy in the learned transition rules (compared to ground truth) and the context window length. in-context learning refers to llm's emergent ability to learn from examples included in the prompt @xcite . one example of in-context learning is zero-shot time series forecasting @xcite . this work aims to forecast empirical time series and introduces a tokenization procedure to convert a sequence of float-ing point numbers into appropriate textual prompts for llms. this led to several subsequent studies on the application of llms for time series forecasting @xcite @xcite @xcite . unlike these prior studies, our work does not focus on forecasting real-world time series, such as weather data or electricity demand, where the underlying model generating the sequence is unavailable or undefined. instead, we aim to extract the learned transition rules from the probability vector generated by the llm and compare them against the ground truth rules (chaotic, stochastic, discrete, continuous, etc.) governing the input time series. ## methodology our methodology for testing llms' ability to learn physical rules from in-context data consists of three steps: 1. sample a time series {x t } @xmath0. 2. prompt the llm with this time series to extract the learned probability densities for subsequent digits pij . 3. measure the discrepancy, between the ground truth p ij and learned pij , using bhattacharyya distance. foot_1 ## experiments and results this section reports empirical in-context learning results on two example systems: a discrete markov chain and a stochastic logistic map. we defer discussion of other systems, reported in figure 1 , to appendices a.3 and a.4. each experiments is repeated ten times with trajectories initiated by different random seeds. model choice. all numerical experiments reported in this section are performed using the open-algorithm 1 hierarchy-pdf input: unrefined pdf, current depth d c , target depth d t procedure: recursiverefiner(pdf, d c , d t ) if d @xmath1, d c , d t ) else if current branch is unrefined then refine pdf with new logits if d c + 1 < d t then append the last digit to launch 10 recursive branches recursiverefiner(pdf, d c + 1, d t ) end if end if output: refined pdf source llama-13b model. while we observe that larger language models, such as llama-70b, may achieve lower in-context loss on some dynamical systems (appendix a.3.3), they do not display qualitative differences that would affect our conclusions.