How Predictable Are Large Language Model Capabilities? A Case Study on BIG -bench (2023)

Abstract: We investigate the predictability of large language model (LLM) capabilities: given records of past experiments using different model families, numbers of parameters, tasks, and numbers of in-context examples, can we accurately predict LLM performance on new experiment configurations? Answering this question has practical implications for LLM users (e.g., deciding which models to try), developers (e.g., prioritizing evaluation on representative tasks), and the research community (e.g., identifying hard-to-predict capabilities that warrant further investigation). We study the performance prediction problem on experiment records from BIG-bench. On a random train-test split, an MLP-based predictor achieves an R^2 score greater than 95%, indicating the presence of learnable patterns within the experiment records. We then formulate the problem of searching for “small-bench,” an informative subset of BIG-bench tasks from which the performance on the full set can be maximally recovered. We find a subset as informative as BIG-bench Hard for evaluating new model families, while being 3\times smaller. Additionally, we find competitive subsets by clustering task representations learned by our MLP-based predictor and selecting tasks close to cluster centroids, highlighting the importance of task diversity in constructing “small-bench.”

## introduction large language models (llms) have revolutionized natural language processing (nlp) research. typically, when researchers introduce a new set of llms, they release them in various sizes and conduct extensive evaluation on different tasks, while also considering different experiment configurations, such as prompting strategies and the number of in-context examples (black et al., 2021; zhang 1 code can be found at https://github.com/ink-usc/ predicting-big-bench . ## creating challenging train-test splits previously, we randomly split d into d train , d dev , and d test , i.e., we randomly sampled 0.6 0.7 0.8 0.9 gradient boosted trees mlp (l, n param , t, n shot ) combinations. this is a relatively easy setting; for example, when the model family l, number of parameters n param , and task t are kept the same, it may be easy for a model to predict performance of n @xmath0. a more challenging data split would ensure that the combinations of (l, n param , t) in the test set are completely unseen in d train , and the model is required to predict for all possible n shot values. taking it a step further, one may want make predictions on an unseen configuration of (l, t) for all possible values of n param and n shot . ## searching for "small-bench" there has been a recent emphasis on assessing the generality of large language models, which involves evaluating these models on numerous tasks and scenarios. however, it will be extremely expensive to conduct all these experiments every time a new model is developed in the future. extending from the holding out l test × t test setting in §4.1, in this section, we formulate and study the problem of searching for "small-bench:" can we find a subset of big-bench tasks, such that when a new model family is evaluated on it, the performance of the remaining tasks can be maximally recovered? in the following we give a formal definition of this problem ( §5.1), construct "small-bench" candidates using different search algorithms and strategies ( §5.2), and present our findings ( §5.3). ## conclusion and future work in this work, we began with the question, "how predictable are large language model capabilities?" and conducted a detailed case study on big-bench. we first formulated the machine learning problem of predicting performance given configurations such as model family, the number of parameters, task, and the number of in-context learning examples. our strongest prediction model achieves an r 2 score greater than 95%, which suggests past llm experiment observations can be used to predict the performance of new experiment configurations. to address the problem of increasing evaluation cost on massively multi-task benchmarks, we introduced the problem of searching for an informative "small-bench." results suggest that popular subsets such as big-bench lite and big-bench hard are not optimal for this purpose. instead, subsets characterized by diversity and high task values offer competitive "small-bench" candidates, highlighting the importance of these two factors. in closing, while our study primarily focused on the predictability of llm capabilities, we hope to initiate discussions on the following broader topics.