Through the Lens of Core Competency: Survey on Evaluation of Large Language Models (2023)

Abstract: “From pre-trained language model (PLM) to large language model (LLM), the field of naturallanguage processing (NLP) has witnessed steep performance gains and wide practical uses. Theevaluation of a research field guides its direction of improvement. However, LLMs are extremelyhard to thoroughly evaluate for two reasons. First of all, traditional NLP tasks become inade-quate due to the excellent performance of LLM. Secondly, existing evaluation tasks are difficultto keep up with the wide range of applications in real-world scenarios. To tackle these problems,existing works proposed various benchmarks to better evaluate LLMs. To clarify the numerousevaluation tasks in both academia and industry, we investigate multiple papers concerning LLMevaluations. We summarize 4 core competencies of LLM, including reasoning, knowledge, relia-bility, and safety. For every competency, we introduce its definition, corresponding benchmarks,and metrics. Under this competency architecture, similar tasks are combined to reflect corre-sponding ability, while new tasks can also be easily added into the system. Finally, we give oursuggestions on the future direction of LLM’s evaluation.”

## introduction large language models(llms) have achieved great progresses in many areas. one representative, chat-gpt 0 , which applies the ability of llms in the form of dialogue, has received much attention due to its incredible versatility such as creative writing, coding, planning, etc. the evaluation of such a model thus becomes necessary to benchmark and build up its ability while preventing potential harmfulness. existing works on the evaluation of llms can be divided into three paradigms. the first line of work is evaluating llms with traditional nlp tasks like dialogue, summarization, etc. since llms are actually pre-trained language models(plms) with huge model parameter size and data size @xcite , benchmarks like glue @xcite , superglue @xcite can be adopted to evaluate its language understanding ability. the problem is that llms work really well on less restrictive tasks like translation, summarization, and natural language understanding tasks. sometimes llms generated outputs' third-party scores are even higher than human generations @xcite , showing the need for higher-quality tasks. secondly, advanced ability evaluations are proposed to completely test language models. the parameter size difference between llms and plms brings an amazing phenomenon, emergence @xcite , which means that scaled models exhibit abilities that are not possessed in small-scaled language models. for instance, in tasks like reasoning, and tool manipulation, the correlation curve between the number of model parameters and the task effect is non-linear. and the effect will rise sharply when the model parameter exceeds a certain parameter scale. they're called "advanced" because they're more closely related to human abilities and harder for models to complete @xcite . thirdly, test language models' intrinsic abilities independent of the specific tasks. it can be tested in parallel with almost every task above. robustness is a classic ability in this paradigm. due to the black-box nature of neural networks @xcite , robustness problems exist for every modality of input data @xcite . current evaluation benchmarks @xcite @xcite @xcite are mostly a mixture of the former three paradigms. they emphasize a complete system of evaluation tasks, in which all tasks are of equal importance. but the significance of marginal increases in model effects on tasks with excellent performance is debatable. thus numerous evaluation tasks and benchmarks are proposed to follow and challenge the ever-evolving llms, while, oddly, seldom being reviewed in a systematic way. how to link numerous tasks and benchmarks, better present the evaluation results, and thus facilitate the research of llms is an urgent problem. an ideal large language model needs to be capable, reliable, and safe @xcite . one surely needs extensive tests on multiple datasets to meet these miscellaneous standards. moreover, to avoid the prevalent training set leakage, test sets also should be updated regularly @xcite . this is similar to the competency @xcite ) tests adopted in corporate recruitment. in competency tests, different task sets are combined to test the corresponding competency. and task sets also need renewal to prevent possible fraud. in this survey, we draw on the concept of the core competency to integrate multiple evaluation research for llms. we investigated 540+ tasks widely used in various papers, aggregating tasks corresponding to a certain competency. during this process, 4 core competencies are summarized, including knowledge, reasoning, reliability, and safety. we will introduce the definition, taxonomy, and metrics for these competencies. through this competency test, superabundant evaluation tasks and benchmarks are combed and clarified for their aiming utility. furthermore, the evaluation results presented with this procedure will be direct, concise, and focused. updated new tasks can also be added comprehensively. to support the community in taking this competency test further, we also create an extensible project, which will show the many-to-many relationship between competencies and tasks precisely. due to the length of the paper, we can only present part of the surveyed results in this paper. a more comprehensive study will be released in a later version. ## core competencies in this section, we introduce the definition and taxonomy of the core competencies we summarized. ## future directions in this section, we outline some other competencies that are important for evaluating llms. ## conclusion this survey provides a comprehensive review of various literature for the evaluation of llms. we aggregate different works with their intended competencies. some of the competencies(reasoning, knowledge) already have holistic evaluation benchmarks, while others(planning, coding) still face disparate challenges. the goal of this paper is to comb the numerous work concerning llms' evaluation through the lens of the core competencies test. lighten the cognitive load for assimilating numerous evaluation 5 planning domain definition language, a formal language used to describe classical planning problems.