AGIE val: A Human-Centric Benchmark for Evaluating Foundation Models (2024)

Abstract: Assessing foundation models’ abilities for human-level tasks is crucial for Artificial General Intelligence (AGI) development.Traditional benchmarks, which rely on artificial datasets, may not accurately represent these capabilities. In this paper, we introduce AGIEval, a novel bilingual benchmark designed to assess foundation models in the context of human-centric standardized exams, such as college entrance exams, law school admission tests, math competitions, and lawyer qualification tests. We evaluate several state-of-the-art foundation models on our benchmark. Impressively, we show that GPT-4 exceeds the average human performance in SAT, LSAT, and math contests, with 95% accuracy on SAT Math and 92.5% on the Chinese college entrance English exam. This demonstrates the exceptional performance of contemporary foundation models. In contrast, we also find that GPT-4 is less proficient in tasks requiring complex reasoning or specific domain knowledge. Our comprehensive analyses of model capabilities (understanding, knowledge, reasoning, and calculation) reveal their strengths and limitations, providing valuable insights into future directions for enhancing general capabilities. By concentrating on tasks pertinent to human cognition and decision-making, our benchmark delivers a meaningful and robust evaluation of foundation models’ performance in real-world scenarios.

## introduction recently, large foundation models, such as the large language models (llms) @xcite and @xcite , exhibited remarkable versatility and adaptability, with plethora of applications spanning various domains as a decision-making assistant, from processing daily events to assisting in specialized fields such as law figure 1 : the performance of llms (text-davinci-003, chatgpt, and gpt-4) was evaluated on several humancentric exams under zero-shot learning with a chainof-thought (cot) prompting setting. human performance (avg.) refers to the average performance of all test takers, while human performance (top) refers to the performance of the top 1% of test takers. compared to the averaged human performance, gpt-4 achieves better scores on the sat, lsat, and math competitions. and finance. with these advancements, ai systems are inching closer to achieving artificial general intelligence (agi). as these ai systems continue to evolve and become more integrated into our daily lives, it is essential to effectively assess their general abilities in handling human-centric tasks, identify potential shortcomings, and ensure that they can handle complex, human-centric tasks effectively. moreover, evaluating their reasoning abilities is also crucial to ensure their reliability and trustworthiness across diverse settings. traditional benchmarks for evaluating foundation models often fall short in providing an accurate assessment of their general abilities in handling human-level tasks. this is primarily due to the use of artificial datasets and a lack of emphasis on realworld tasks that require human-like cognitive capabilities. moreover, these benchmarks often focus on tasks that do not truly represent the complexities and nuances of real-world human cognition and decision-making, leading to a skewed evaluation of models' capabilities and limiting their ability to provide meaningful insights into the models' real-world applicability. consequently, there is a growing need for a more human-centric benchmark that allows for a robust evaluation of foundation model in the context of tasks that are relevant to human reasoning and problem-solving. we introduce a human-centric benchmark, agieval, specifically designed to evaluate the general abilities of foundation models in tasks pertinent to human-level problem-solving. this benchmark is derived from official, public, and high-standard admission and qualification exams intended for general human test-takers, such as general college admission tests (e.g., chinese college entrance exam (gaokao) and american sat), law school admission tests, math competitions, lawyer qualification tests, and national civil service exams. these exams are taken by a diverse range of individuals seeking entry into higher education institutions or new career paths, with millions participating annually (e.g., 12 million for the chinese gaokao and 1.7 million for the american sat). as a result, these exams establish officially recognized standards for assessing human-level capabilities. additionally, the benchmark covers bilingual tasks in both chinese and english, allowing for a more comprehensive evaluation. by concentrating on these tasks, our benchmark provides a more meaningful and comprehensive evaluation of large language model performance in scenarios directly relevant to human decision-making. we employ 20 human-centric tasks across a wide variety of subjects in our benchmark to assess the performance of cutting-edge foundation models, encompassing close-source models, i.e., textdavinci-003, chatgpt and gpt-4, and an opensource model, vicuna @xcite . our experiments explore their performance under various settings, including few-shot learning, zeroshot learning, and chain-of-thought prompting techniques. we compare the performance of these models with human performance, as illustrated in fig. 1 . remarkably, the results reveal that gpt-4 outperforms the average human performance on lsat, sat, and math competitions under the zero-shot chain-of-thought (cot) setting, demonstrating its capability on human-centric tasks. however, there remains a gap between gpt-4 and the top human performance, indicating opportunities for future improvement. we also discover that these models struggle with tasks requiring complex reasoning (e.g., lsat-analytical reasoning and physics) or specific domain knowledge, such as law and chemistry. moreover, our comprehensive qualitative analyses of the four dimensions of model capabilities (i.e., understanding, knowledge, reasoning, and calculation) delve into their respective strengths and limitations, providing valuable insights into their general capabilities. this multifaceted approach enables us to examine the models' single-task behavior and identify general patterns, ultimately contributing to a more robust understanding of these state-of-the-art models and their potential applications in tackling human-level tasks. evaluation of language models: constructing benchmarks is a reliable way to establish evaluation standards and monitor model performance. numerous benchmarks @xcite have been proposed and widely adopted for evaluating single-task performance, such as squad @xcite for assessing answer extraction ability and snli @xcite for evaluating natural language inference capability. the emergence of general language models (lms) like bert @xcite has made it increasingly essential to develop more comprehensive benchmarks to assess the general capabilities of these lms. glue @xcite and superglue @xcite are popular benchmarks that evaluate language model performance across diverse nlp tasks. glue series benchmarks have significantly influenced language model development, encouraging researchers to enhance their models' generalization capabilities. the lambada language modeling task @xcite assesses language models' ability to capture long-range dependencies in text. senteval @xcite and @xcite also set benchmarks for evaluating models' general capabilities. toxi-gen @xcite and bold @xcite evaluate the bias in language models. despite their broad applicability, these benchmarks mainly consist of artificially curated datasets designed to evaluate specific machine skills, rather than real-world problems aimed at assessing human behaviors. consequently, these benchmarks primarily focus on simpler textual understanding rather than complex reasoning abilities aligned with real-world applicability. mmlu @xcite addresses this issue by collecting questions from online sources covering a diverse set of subjects (e.g., history, humanities) that humans learn, pushing towards human-centric evaluation. our work differs from mmlu in two main ways: (1) we derive our benchmark from high-standard human-centric exams like college admissions tests, ensuring a robust, standardized evaluation, unlike mmlu which lacks explicit sourcing details. (2) agieval is bilingual (english and chinese), broadening the assessment scope across languages and cultures, whereas mmlu is solely english-based. the official technical report of gpt-4 (openai, 2023) also underscored the importance of evaluating models' behaviors on human exams and analyzed gpt-4's performance on several such exams. however, the relevant benchmarks in these reports and the corresponding model outputs are not publicly available, and the evaluation metric is also not transparent. these factors limit further research to follow up their evaluation. ## conclusion we introduce agieval, a novel benchmark specifically designed to assess the general capabilities of large foundation models with respect to humanlevel cognition. the benchmark comprises highquality official admission tests, qualification exams, and advanced competitions tailored for human participants, including law school admission tests and college entrance examinations. these assessments establish officially recognized standards for gauging human capabilities, making them well-suited for evaluating foundation models in the context of human-centric tasks. additionally, agieval incorporates bilingual tasks in both chinese and english, offering a more comprehensive assessment of model behavior. we have carried out an extensive evaluation of three cutting-edge large foundation models: text-davinci-003, chatgpt, and gpt-4, using agieval. remarkably, gpt-4 surpasses average human performance on lsat, sat, and math competition, attaining a 95% accuracy rate on the sat math test and a 92.5% accuracy on the gaokao english test, demonstrating the impressive performance of contemporary foundation models. despite their significant achievements, our in-depth manual analyses also reveal the limitations of these large language models in terms of understanding, knowledge utilization, reasoning and calculation. guided by these findings, we explore potential future research avenues in this domain. by assessing these foundation models on human-centric tasks and probing their capabilities more deeply, we strive to foster the development of models that are more closely aligned with human cognition.