User Behavior Prediction as a Generic, Robust, Scalable, and Low-Cost Evaluation Strategy for Estimating Generalization in LLM s (2025)

Abstract: Measuring the generalization ability of Large Language Models (LLMs) is challenging due to data contamination. As models grow and computation becomes cheaper, ensuring tasks and test cases are unseen during training phases will become nearly impossible. We argue that knowledge-retrieval and reasoning tasks are not ideal for measuring generalization, as LLMs are not trained for specific tasks. Instead, we propose user behavior prediction, also a key aspect of personalization, as a theoretically sound, scalable, and robust alternative. We introduce a novel framework for this approach and test it on movie and music recommendation datasets for GPT-4o, GPT-4o-mini, and Llama-3.1-8B-Instruct. Results align with our framework’s predictions, showing GPT-4o outperforms GPT-4o-mini and Llama, though all models have much room for improvement, especially Llama.

## generalization in the era of llms in essence, generalization is a model's ability to perform well on unseen test data, given that the test set measures the same task during model training, indicating the efficacy of the model in learning the underlying patterns in the training data without overfitting or underfitting, attaining a trade-off between bias and variance @xcite . llms, conceived as general-purpose models, are expected to follow instructions in natural language and perform well on varied tasks, emulating human-like behavior. however, current task-centric evaluation schemes fail to measure the generalization capacity of models holistically, often leading to conflated results. this deviation is primarily due to the following factors. ## proposed method in the current context of llms, the ability to perform complex reasoning tasks, such as writing code or solving math problems, although difficult for many humans, are shallower measures of intelligence and generalization since they are universal tasks. however, the ability to solve the same task, mimicking a specific person, is much more challenging than solving the task like any person. extending turing's definition, we argue that any machine that can mimic individual users from its training data is the most robust test of intelligence and the strongest measure of generalization. hence, we propose measuring a model's capacity for personalization as a robust test of its generalization. ## measuring generalization via personalization recommendation systems, at their core, are personalization engines. they are predictors of a user's future behavior based on some observed past behavior. we test our proposed statistical framework in section 3.2 and experiment with movie and music recommendations, where the task is to recommend a list of n items based on the user's history. ## conclusion we propose a statistically motivated framework using personalization to assess generalization in llms. since llms are trained on vast humangenerated data, we argue that true generalization lies in predicting human behavior rather than specific tasks. this philosophically aligns with wittgenstein's language games and turing's imitation game, though with a key distinction with the latter: while turing's test requires mimicking any human, our framework challenges models to replicate a specific user at varying complexity levels. this shift has profound philosophical and mathematical implications, only some of which we could explore in this paper. we would like to highlight the lack of datasets where complex user behaviors or preferences are available alongside their demographic proxies, which will enable us to conduct large-scale and more extensive studies of generalization.