Uncovering Scaling Laws for Large Language Models via Inverse Problems (2025)

Abstract: Large Language Models (LLMs) are large-scale pretrained models that have achieved remarkable success across diverse domains. These successes have been driven by unprecedented complexity and scale in both data and computations. However, due to the high costs of training such models, brute-force trial-and-error approaches to improve LLMs are not feasible. Inspired by the success of inverse problems in uncovering fundamental scientific laws, this position paper advocates that inverse problems can also efficiently uncover scaling laws that guide the building of LLMs to achieve the desirable performance with significantly better cost-effectiveness.

## introduction llms represent a paradigm shift in artificial intelligence, embodied by their unprecedented levels of complexity and scale in both data and computations, and their demonstrated generalization capabilities across a wide array of tasks and domains, such as natural language processing, computer vision, coding, gaming, among many others @xcite @xcite @xcite . these remarkable successes result from the amalgamation of several input ingredients, including high-quality and diverse training data, advanced modeling techniques, skillfully designed training procedures, and effective inference schemes @xcite @xcite . the intricate interactions among these ingredients are not fully understood, yet they collectively influence the overall performance of large language models. to advance the development of high-performance and cost-effective models further, it is essential to uncover the underlying scaling laws that govern these interactions. more importantly, designing an llm that achieves desirable performance under resource constraint is an inherently complex challenge, as it requires the careful selection and combination of data, model architecture, training procedures, and inference strategies. as an example, when building an llm specifically for gsm8k @xcite (i.e., grade school math benchmark), several design principles must be considered: (i) the training data should contain ample examples that foster language understanding and reasoning capabilities to ensure that the llm can learn the nuances of math problems presented in natural language; (ii) the model architecture should be complex enough to process sequential inputs (since each question in gsm8k is described in natural language) and generate the required output formats, such as multiple-choice questions or detailed natural language explanations; (iii) the training procedure should be designed to allow the model to effectively acquire task-specific knowledge from the data (e.g., suitably defined loss functions tailored for solving math problems); and (iv) the inference scheme should guide the llm toward generating accurate and desired outputs, as demonstrated by techniques like chain of thought (cot) @xcite , react @xcite , and tree of thoughts @xcite . due to the scale of the required data and modern model architectures, creating an llm instance is an extremely costly process, e.g., gpt-4 costs over @xmath0191 million @xcite . this high expense makes building better llms through brute-force trial and error prohibitively costly. in contrast, deepseek v3 achieved state-of-the-art performance with just $5.6 million by optimizing training protocols and architecture @xcite . thus, it becomes necessary to uncover underlying scaling laws (e.g., the required composition and minimum size of training data or model architecture) that help build llms with the desired performance and significantly better cost-effectiveness. to this end, we advocate examining the class of inverse problems for llms. inverse problems involve determining unknown parameters of an underlying model from observational data, a concept crucial in scientific and engineering domains @xcite @xcite @xcite . tackling the inverse problems is a tried-and-true methodology for inferring and uncovering fundamental scientific laws from observations. for example, kepler's laws of planetary motion were derived from the observed motion of mars and its elliptical orbit; newton's law of universal gravitation was based on the empirical effects of kepler's laws; and in modern quantum mechanics, schrödinger's wave equation was inferred from electron diffraction experiments. inspired by these successes, inverse problems offer a powerful approach for uncovering the underlying scaling laws behind the behavior of llms. a typical approach to tackle an inverse problem involves using a forward process to obtain observation data given some specified input and latent parameter values. however, this forward process is often costly. the inverse problem, which involves identifying the latent parameter values that are consistent with a given set of observation data, is inherently very challenging due to the complexity of the search space and the lack of solution uniqueness. in the llm context, the inverse problem requires finding the optimal combination of input ingredients (i.e., data, model architecture, training procedures, and inference schemes) to build the llms that achieve desirable performance, while forward processes refer to the costly training of llms and running model inference for task execution and evaluation. formally, let t denote the training ingredients, such as the dataset, model architecture, and training procedure, and let i represent the ingredients of the inference scheme (e.g., prompting method). note that t includes both pretraining and fine-tuning, and it affects the llm's model parameters, whereas i typically does not alter these parameters. let f (t ) → llm denote the process of creating an llm by executing the computation following the specified ingredients t (i.e., forward process). let t (llm, i) → c represent the evaluation of the llm on a task using the inference scheme i, resulting in a performance metric c. therefore, we have the following two forward processes: t (f (t ) , i) → c . these two forward processes are illustrated in fig. 1 . to understand how these forward processes function, consider the above example of building an llm for the gsm8k task, which assesses various design principles related to data (i.e., pretraining and fine-tuning datasets), model architecture, and training procedure. these ingredients are included within t and used in the creation process of an llm as f (t ) → llm. subsequently, the trained llm, along with the inference ingredients i, is evaluated on the gsm8k task as t (llm, i) → c. here, t (•) includes both the evaluation metric (e.g., accuracy) and the evaluation dataset (i.e., gsm8k questions), and c is thus representing the accuracy of the trained llm on the gsm8k benchmark. given practical constraints such as limited data and computational resources, tackling the inverse problems to uncover end-to-end scaling laws may be overly ambitious. therefore, as a first step, we consider simplified inverse problems by fixing certain ingredients or focusing on a manageable subset of the problem space. specifically, this position paper frames the following classes of inverse problems in the context of llms: ## data selection the recent successes of llms have been driven by training on massive and heterogeneous datasets @xcite . for example, llama 3 was trained on 15 trillion multilingual tokens @xcite . previous works have established scaling laws that link data quantity to model performance @xcite @xcite @xcite . however, more recent studies @xcite @xcite @xcite demonstrate that strategically selecting data subsets can improve the performance of both llms and multi-modal llms (mllms) in a way even surpassing the conventional scaling laws, particularly in domains like computer vision @xcite . this naturally raises key questions: how does model performance scale with data quantity when data selection methods are used for mllms? furthermore, how do the scaling laws vary across different stages of mllm training, such as pretraining, fine-tuning, and alignment? we formulate data selection as an inverse problem of t (f (t ) , i) → c. the goal is to understand how the quantity of selected training data (in t ) scales with the desired mllm performance (c). for example, we might want to identify the minimal dataset required to train an mllm to achieve specific performance metrics under optimal data selection. therefore, efficient data selection can significantly reduce computational costs by prioritizing informative and representative data, thereby improving training efficiency without sacrificing performance. furthermore, these scaling laws should be general enough so that they are applicable to a family of data selection methods instead of specific implementations (e.g., the family of influence functions @xcite versus its implementation datainf @xcite ). ## inference optimization optimizations carried out at the inference stage significantly affect the performance of llms. for example, given a trained llm, it is common practice to provide a prompt (i.e., a snippet of text) that the llm uses to generate further text conditioned on the snippet. this represents a forward process t (f (t ) , i) → c in eq. ( 1b ), where the prompt is a component of inference ingredient i, and inverting the process to carefully construct prompts that can instruct the llm to perform a specific downstream task, hence achieving a desired performance measured by the metric c, is challenging. thus, inference optimization can be viewed as an inverse problem of t (f (t ) , i) → c in eq. ( 1b ), where the goal is to design inference schemes in i that, when combined with a model trained on t , achieves the desired performance metric c. furthermore, one can also aim to uncover the underlying scaling laws at inference time with respect to optimized data, model architecture, and computational resources. ## unlearning machine unlearning (mu) is the process of removing the influence of a set of training data (i.e., erased data) from a trained model to either comply with data owners' deletion requests @xcite or erase harmful data to improve the model performance @xcite @xcite . we consider two inverse problems. verification of mu is an inverse problem of f (t ) → llm as given any "unlearned" model, aiming to identify if the erased data is present in the training ingredients t . mu techniques can also be viewed as an inverse problem of t (f (t ) , i) → c. given certain performance metrics (e.g., poor knowledge on weapons of mass destruction @xcite , similar performance on the retained data as before unlearning), the goal is to design the inference ingredients (e.g., unlearning prompts) in i, or to adjust the datasets and training procedure (e.g., use of training checkpoints, model architecture that facilitates unlearning without retraining) in t to achieve the desired performance metrics. ## conclusion and future outlook this position paper highlights the importance of understanding of the scaling laws that govern the behavior of llms, such as data requirements and compute scaling laws. to uncover the underlying scaling laws, we advocate for research exploring two classes of inverse problems for llms (i.e., eq. ( 1a ) and eq. ( 1b )): identifying optimal input ingredients and achieving desired performance metrics by adjusting both training and inference ingredients. specifically, we frame data selection, inference optimization, and machine unlearning as inverse problems, each presenting unique challenges to solve. yet, jointly optimizing them (including data, model architecture, training procedures, inference scheme, and unlearning techniques) holds great potential for advancing the development and deployment of llms. instead of iterating over the engineering efforts to further improve the empirical performance, we advocate to uncover the underlying fundamental scaling laws governing the training and inference of llms via inverse problems, which can lay the foundations for building better llms. these scaling laws can improve specific applications by providing better selection methods for training data and model architectures, flexible unlearning techniques, methods with improved inference efficiency, and optimized inference schemes. looking ahead, future research should explore these scaling laws and investigate how the interplay among various components and ingredients impacts overall performance. emerging technologies and methodologies from fields like optimization theory can provide novel tools for tackling inverse problems in llms. additionally, advancements in machine unlearning will be crucial as models become more complex, ensuring they can adapt without compromising functionality or privacy standards. by integrating these approaches, we may uncover innovative solutions that improve the efficiency and cost-effectiveness of llm development.