GPT -Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT -4 and Beyond (2024)

Abstract: With the rapid advancement of large language models (LLMs), there is a pressing need for a comprehensive evaluation suite to assess their capabilities and limitations. Existing LLM leaderboards often reference scores reported in other papers without consistent settings and prompts, which may inadvertently encourage cherry-picking favored settings and prompts for better results. In this work, we introduce GPT-Fathom, an open-source and reproducible LLM evaluation suite built on top of OpenAI Evals. We systematically evaluate 10+ leading LLMs as well as OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories, all under aligned settings. Our retrospective study on OpenAI’s earlier models offers valuable insights into the evolutionary path from GPT-3 to GPT-4. Currently, the community is eager to know how GPT-3 progressively improves to GPT-4, including technical details like whether adding code data improves LLM’s reasoning capability, which aspects of LLM capability can be improved by SFT and RLHF, how much is the alignment tax, etc. Our analysis sheds light on many of these questions, aiming to improve the transparency of advanced LLMs.

## introduction recently, the advancement of large language models (llms) is arguably the most remarkable breakthrough in artificial intelligence (ai) in the past few years. based on the transformer @xcite architecture, these llms are trained on massive web-scale text corpora. despite their straightforward method of using a self-supervised objective to predict the next token, leading llms demonstrate exceptional capabilities across a range of challenging tasks @xcite , even showing a potential path towards artificial general intelligence (agi). with the rapid progress of llms, there is a growing demand for better understanding these powerful models, including the distribution of their multi-aspect capabilities, limitations and risks, and directions and priorities of their future improvement. it is critical to establish a carefully curated evaluation suite that measures llms in a systematic, transparent and reproducible manner. although there already exist many llm leaderboards and evaluation suites, some key challenges are yet to be addressed: • inconsistent settings: the evaluation settings, such as the number of in-context example "shots", whether chain-of-thought (cot; @xcite prompting is used, methods of answer parsing and metric computation, etc., often differ across the existing llm works. moreover, most of the released llms do not disclose their prompts used for evaluation, making it difficult to reproduce the reported scores. different settings and prompts may lead to very different evaluation results, which may easily skew the observations. yet, many existing llm leaderboards reference scores from other papers without consistent settings and prompts, which may inadvertently encourage cherry-picking favored settings and prompts for better results. to achieve reliable conclusions, it is crucial to make applesto-apples comparisons with consistent settings and prompts. these challenges hinder a comprehensive understanding of llms. to dispel the mist among llm evaluations, we introduce gpt-fathom, an opensource and reproducible llm evaluation suite developed based on openai evals foot_0 . we evaluate 10+ leading open-source and closed-source llms on 20+ curated benchmarks in 7 capability categories under aligned settings. we also evaluate legacy models from openai to retrospectively measure their progressive improvement in each capability dimension. our retrospective study offers valuable insights into openai's evolutionary path from gpt-3 to gpt-4, aiming to help the community better understand this enigmatic path. our analysis sheds light on many community-concerned questions (e.g., the gap between openai / non-openai models, whether adding code data improves reasoning capability, which aspects of llm capability can be improved by sft and rlhf, how much is the alignment tax, etc.). with reproducible evaluations, gpt-fathom serves as a standard gauge to pinpoint the position of emerging llms, aiming to help the community measure and bridge the gap with leading llms. we also explore the impacts of model sensitivity on evaluation results with extensive experiments of various settings. the key contributions of our work are summarized as follows: ## related work benchmarks constantly play a pivotal role in steering the evolution of ai and, of course, directing the advancement of llms as well. there are many great existing llm evaluation suites. by comparing gpt-fathom with previous works, we summarize the major difference as follows: 1) helm @xcite ) primarily uses answeronly prompting (without cot) and has not included the latest leading models such as gpt-4 (as of the time of writing); 2) open llm leaderboard @xcite focuses on open-source llms, while we jointly consider leading closed-source and open-source llms; 3) opencompass @xcite evaluates latest open-source and closed-source llms (all released after 2023/03), while we cover both leading llms and openai's earlier models to decipher the evolutionary path from gpt-3 to gpt-4; 4) instructeval @xcite is designed for evaluating instruction-tuned llms, while we evaluate both base and sft / rlhf models; 5) alpacaeval @xcite evaluates on simple instructionfollowing tasks as a quick and cheap proxy of human evaluation, while we provide systematic evaluation of various aspects of llm capabilities; 6) chatbot arena @xcite evaluates human user's dialog preference with a elo rating system, while we focus on automatic and reproducible evaluation over popular benchmarks; 7) chain-of-thought hub @xcite focuses on evaluating the reasoning capability of llms with cot prompting, while we support both cot and answer-only prompting settings and evaluate various aspects of llm capabilities. we discuss more related work in appendix g. ## method imagine the ultimate superset of llm evaluations: a holistic collection that evaluates every llm on every benchmark under every possible setting. in practice, however, due to resource and time constraints, we are unable to exhaustively fulfill this ideal evaluation superset. instead, we pick representative llms, benchmarks and settings to investigate open problems. in this section, we discuss in detail how we select llms, benchmarks and settings for our evaluations. ## conclusions we present gpt-fathom, an open-source and reproducible evaluation suite that comprehensively measures the multi-dimensional capabilities of llms under aligned settings. our retrospective study on openai's models helps the community better understand the evolutionary path from gpt-3 to gpt-4, and sheds light on many community-concerned questions. for example, our study reveals that sft and rlhf yields more benefit for weaker models, while for stronger base models, it offers diminishing returns or incurs an alignment tax. moreover, we identify novel challenges of advanced llms, such as prompt sensitivity and the seesaw phenomenon of llm capabilities. gpt-4 with functionalities of advanced data analysis and sandboxed python code interpreter; 5) gpt-4-1106-preview, an early-access api of the upgraded model gpt-4 turbo foot_10 .