Collaborative Performance Prediction for Large Language Models (2024)

Abstract: Comprehensively understanding and accurately predicting the performance of large language models across diverse downstream tasks has emerged as a pivotal challenge in NLP research. The pioneering scaling law on downstream works demonstrated intrinsic similarities within model families and utilized such similarities for performance prediction. However, they tend to overlook the similarities between model families and only consider design factors listed in the original scaling law. To overcome these limitations, we introduce a novel framework, Collaborative Performance Prediction (CPP), which significantly enhances prediction accuracy by leveraging the historical performance of various models on downstream tasks and other design factors for both model and task. We also collect a collaborative data sourced from online platforms containing both historical performance and additional design factors. With the support of the collaborative data, CPP not only surpasses traditional scaling laws in predicting the performance of scaled LLMs but also facilitates a detailed analysis of factor importance, an area previously overlooked.

## introduction large language models (llms) @xcite have emerged as one of the most important ai research powered by largescale parameters, high computational resources, and massive training data. with the substantial increase in model sizes, the evaluation cost of llms' performance becomes even more significant. for example, testing a single llm on certain benchmarks often requires $10k+ and 4k+ gpu hours @xcite . therefore, understanding the behaviors and predicting the capabilities of llms across scales under various tasks becomes a vital question @xcite @xcite for both researchers and engineers. scaling laws @xcite @xcite @xcite have been powerful tools for predicting the capabilities of llms. it indicates a power-law correlation between the model performance and design factors such as computational measure (flops) utilized during training. although the scaling law was originally proposed as a strong intuitive guide for designing llm, researchers @xcite @xcite have extended its utility into predicting model performances on various metrics, such as bleu in machine translation, and different tasks. these works can accurately predict model performances by utilizing the similarity within each model family, e.g., models within each family are usually trained on the same dataset. however, there are several issues rooted in their methods: the performance prediction 1) requires transparent design factors that consume substantial training resources to fit the curve, 2) is only tailored to a certain model family and a specific task metric, and 3) neglects the connections among different models and tasks. the aforementioned limitations motivate us to design more effective methods for predicting the performance of llms on downstream tasks. two observations sparked our attention. firstly, a strong similarity exists between model families, e.g.llama-family and gpt family. models from different families behave similarly in prediction distribution @xcite and emergent phenomenon @xcite . secondly, with the emerging llm models and the increasingly diverse tasks, the cost of enumerating and benchmarking models with tasks increases exponentially. therefore, we aim to utilize the similarities across model families in order to collaboratively predict the model performance over diverse tasks in an accurate yet efficient way. to incorporate the aforementioned intuitions, we propose a new scheme, collaborative performance prediction (cpp), to efficiently predict the performance of llms on evaluation tasks. this scheme learns the latent representations of llms and tasks, which captures the intrinsic similarity among different models and tasks. the interaction (e.g., inner product) between the latent representations of llms and tasks can be utilized to predict the performance of llms on certain tasks. to fulfil the proposed scheme, we collect the llm performance data from academic papers, technical reports, and open leaderboards covering 72 models and 29 tasks. to summarize, our scheme has several advantages: under our scheme, multiple customized prediction methods (e.g., collaborative fiter-ing @xcite ) can be incorporated to predict the performance of llms, further validating the feasibility and generality. our method enables more diverse factors as input, ranging from traditional llm design factors to task design factors, e.g., targeted ability and few-shot setting. upon extensive experimentation within the openreleased core leaderboard of helm @xcite and our collected historical matrix, our predictive performance demonstrated exceptionally well. specifically, even without any input of model factors or task factors: in helm, we use 50% of the scores to predict the other 50%, the predictive ranking (derived from predicted scores) achieves @xmath0, and m ae@2 =39%; in our collected matrix (characterized by a 44% sparsity level) achieves an @xmath1, and the m ae@2 =84%. notably, the accuracy of our prediction from small to large lms significantly exceeded that predicted by scaling laws. using an analysis method similar to shapley-values @xcite , we elucidate the importance of different factors, which surprisingly does not fully align with scaling law @xcite . therefore, our method is undoubtedly more versatile. ## experiments in this section, we evaluate the feasibility of cpp from an overall benchmark perspective and a model perspective in section 5.1 and 5.2, respectively; ## conclusion and discussion advancing beyond traditional scaling laws on downstream tasks, we propose a collaborative performance prediction framework for large language models. it offers significant advantages, including easy deployment, low training costs, and superior predictive accuracy. uniquely, it enables incorporating additional design factors and supports an in-depth analysis of their impact, including factor importance and correlations in models and tasks. for prediction, we collect collaborative data containing many historical performances and factors. our method's predictive accuracy is expected to improve as it benefits from an expanding pool of collaborative data. moreover, this approach highlights the potential to identify neglected but vital factors beyond traditional scaling laws, such as task design factors, thereby enriching our comprehension of llm performance predictability on downstream tasks. limitations "single-source-of-truth". when collecting the collaborative data, we hypothesize that each model's performance on each task is identical. however, in the real world, the detailed testing setting, for instance, the testing prompt writing, can influence llm's performance variance. although we observed this, we only saved one score from different sources. how to incorporate the setting of testing as an additional dimension remains to be solved in future works. susceptibility to data quality. the prediction accuracy of cpp highly depends on the quality of collaborative data. the current version passively collects collaborative data from online resources. should information from either of these data sources be incorrect, the prediction capability of cpp would decrease correspondingly. to overcome such a limitation, jointly considering passive information collected from data sources and active information, such as performances of models tested on some tasks by the user, might be a solution. utilizing techniques such as efficient benchmarking @xcite could alleviate the cost of obtaining active information.