Exploring the Learning Capabilities of Language Models using LEVERWORLDS (2024)

Abstract: Learning a model of a stochastic setting often involves learning both general structure rules and specific properties of the instance. This paper investigates the interplay between learning the general and the specific in various learning methods, with emphasis on sample efficiency. We design a framework called LEVERWORLDS, which allows the generation of simple physics-inspired worlds that follow a similar generative process with different distributions, and their instances can be expressed in natural language. These worlds allow for controlled experiments to assess the sample complexity of different learning methods. We experiment with classic learning algorithms as well as Transformer language models, both with fine-tuning and In-Context Learning (ICL). Our general finding is that (1) Transformers generally succeed in the task; but (2) they are considerably less sample efficient than classic methods that make stronger assumptions about the structure, such as Maximum Likelihood Estimation and Logistic Regression. This finding is in tension with the recent tendency to use Transformers as general-purpose estimators. We propose an approach that leverages the ICL capabilities of contemporary language models to apply simple algorithms for this type of data. Our experiments show that models currently struggle with the task but show promising potential.

## introduction many statistical learning settings combine two types of challenges: discovering the underlying persistent structure or representation of the problem, and modeling the context-dependent variability @xcite . these two factors differ in their generality -the structure is shared by many cases that might differ in their variability. for example, assume we want to know how long a typical object will take to reach the ground when dropped from a building in some city. assuming this knowledge has not been directly reported, we must acquire it from the data. we can conduct experiments by dropping different balls from different buildings. learning involves acquiring two types of knowledge -one is the physical rules of free fall (e.g., that the mass does not influence the time of the fall), and the other is the distribution of the heights of the buildings in this city (assuming we cannot directly measure the heights). both types are induced from experiments, but the first is universal, and as such it might already be known based on experiments conducted in a different place. recent large language models (llms) are used as general purpose learners, as almost any task that does not require multiple modalities can be formulated as text-to-text or text completion. therefore, the model must learn both the world and stochastic model for effective learning. as typically models are trained with likelihood-based objectives, the models' output confidence for completions should reflect the underlying distribution of the data. the density estimation task from observations is well-studied in statistics and machine learning. different methods for parameter estimation differ in their assumptions (or parameterizations). generally, a model with fewer assumptions can better fit the true unknown) distribution, compared to a model with many assumptions. this is described as having lower bias. on the other hand, when a model is more flexible it is more sensitive to noise in the specific training set, hurting generalization. this is described as having higher variance. this tradeoff between these two phenomena is known as the bias-variance tradeoff. the tradeoff has implications on the sample complexity of a model, as high variance might require training sets with more samples. we further show that leading off-the-shelf llms, like gpt4o 2 , fail on this task in an in-contextlearning setting. however, inspired by the underlying structure vs. variability distinction, we propose a method for introducing inductive bias into these models by guided construction of a learning pipeline that includes classical models. we find that although challenging for all models, some models show a substantial improvement over others, revealing a promising trend. this further supports the centrality of the distinction. our framework, leverworlds, is also a contribution in its own right. it has many appealing traits: (1) it is based on real physical laws, and is thus not a "toy example"; (2) it includes many variations, all of which are simple to learn when the physical rules are known but complicated oth-2 https://openai.com/index/hello-gpt-4o/ erwise; (3) it follows a generative process, which allows for the generation of arbitrarily large sets of supervised samples, with confidence scores, for training and testing. our contributions are: (1) we present a framework for experiments in a physical setting; (2) we show that lms succeed in learning world models, but they are substantially inferior (in terms of sample complexity) to classical models with stronger assumptions; (3) we show the distinction between learning the world model and learning the latent model; (4) we propose methods for using llms in combination with classical models, with promising initial results. ## the leverworlds environment as empirical validation for our arguments, we experiment with a case in which the world has a simple structure and assess the sample complexity when using language models. understanding the underlying world model can significantly reduce the effective sample complexity of the task, allowing accurate estimations with a small amount of training data. we design a framework for generating worlds that enables efficient sampling and estimation. for each generated world, we design models for estimating the distribution of data based on observed samples. ## results here we report and plot the performance of the various models and methods. we conducted the main experiments on two randomly chosen world settings. the first setting (world-1) has visible variables of mass1, dis-tance1, side1, mass2, and side2. the second setting (world-3) foot_2 has visible variables of density1, vol- (a) total-variation distance (b) structure scores figure 5: results for mle models. in figure 3 we report the tv distances and structure scores for the transformer models. additionally, we report the perplexity of the trained transformers on a portion (first 500 documents) of wikitext-2. 5 these values serve as indicators of the extent to which the textual pretraining is affected by the task-specific finetuning. in figures 4 and 5 we report the tv distances and structure scores for the logistic regression and mle models, respectively. zero-shot scores. results for the pipeline learning experiment are reported in ## discussion as world-3 is clearly more complex than world-1, it is interesting to see how this affects different models. in naïve-mle we see a significant performance gap between the settings, with the method only starting to learn with ≈ 10k samples in world-3. in appendix b we derive simple bounds for the expected squared tv distance of this method and show that the extra variables have a substantial effect. in logistic regression and opt models, there is a consistent gap between the two settings but it is not substantial. in structure-mle the performance is practically the same. this fits a general trend regarding the provided knowledge about the structure. the more the model is provided with structure, the less the effect of additional variables. in figure 6 we plot the tradeoff between the tv distance and the structure score (3.2). since the naïve-mle model makes no explicit assumptions regarding the structure, we can see its curve as representing the structure score that can be achieved without explicit learning. this curve bounds the curves from below. from above, the structure-mle bounds all other methods, as it is provided with full structure knowledge by definition. in between, we see that logistic regression has a higher structure score (per error) compared to opt models. the general trend we see is that transformer models seem to learn the structure to some extent, but it is not as strong as in models that are given stronger assumptions. in most of our experiments, the models achieve low error. the exceptions are logistic regression with polynomial-degree 1 (for both worlds) and naïve-mle for world-3. in the first case, the model makes strong assumptions that simply do not fit the world. in the second case, the lack of assumptions regarding the world leads to extremely high sample complexity. among different settings of the transformers, we find that larger models learn with fewer samples. however, smaller models seem to preserve the textual pretraining for longer learning. additional epochs improve the results, but only up to a certain point. the trend regarding the provided structure is aligned with the number of parameters in the models. naïve-mle has |states| |values| parameters. structure mle has 1 parameter. logistic regression has |states| degree . in this respect, the opt model is an exception, as larger models have better performance. while our zero-shot experiments generally show low results, they do show promising directions. with in-context learning, we find that llms, even strong long-context ones, struggle with this task. it seems then that the models implicitly apply simplistic heuristics instead of rigorous analysis. in the pipeline experiment, we see that despite the poor performance, there is a clear hierarchy in which gpt4 and gpt4o clearly outperform gpt3.5. this shows that, to some extent, llms can be used as components in a pipeline that uses other models. this type of approach was described in @xcite , and we view it as a promising approach for the future. this observation impacts many highly studied tasks that involve components that do not fall under the description of "natural language", such as chess @xcite and arithmetics @xcite . our findings suggest that perhaps tools that were designed for natural language are not optimal for these tasks. we note that although our experiments address finetuning and inference, the findings are relevant for pretraining too. our findings show that the training data can contain information, that can be captured by simple models, but llms may not capture. ## conclusion in this paper, we presented a novel framework for generating experimental worlds from a common physical setting, with easily manipulable distributions. the framework allows sampling from the ground-truth model and enables carefully controlled experiments in learning the distributions. we applied various methods, from classic learning algorithms to various sizes of transformers. the methods range from highly structure-aware to structure-agnostic. our findings show that even in a very simple physical setting, models that make stronger assumptions as to the structure present better sample complexity. specifically, in the given setting, simple structure-based models like logistic regression and full structure mle can be substantially more sample-efficient compared to transformer models. we further propose an approach to leverage llms as part of a pipeline that involves a classic learning algorithm. we show that models still struggle with this task but show a promising trend, as newer models show substantial improvements over older ones.