F -Eval: Asssessing Fundamental Abilities with Refined Evaluation Methods (2024)

Abstract: Large language models (LLMs) garner significant attention for their unprecedented performance, leading to an increasing number of researches evaluating LLMs. However, these evaluation benchmarks are limited to assessing the instruction-following capabilities, overlooking the fundamental abilities that emerge during the pre-training stage. Previous subjective evaluation methods mainly reply on scoring by API models. However, in the absence of references, large models have shown limited ability to discern subtle differences. To bridge the gap, we propose F-Eval, a bilingual evaluation benchmark to evaluate the fundamental abilities, including expression, commonsense and logic. The tasks in F-Eval include multi-choice objective tasks, open-ended objective tasks, reference-based subjective tasks and reference-free subjective tasks. For reference-free subjective tasks, we devise new evaluation methods, serving as alternatives to scoring by API models. We conduct evaluations on 13 advanced LLMs. Results show that our evaluation methods show higher correlation coefficients and larger distinction than other evaluators. Additionally, we discuss the influence of different model sizes, dimensions, and normalization methods. We anticipate that F-Eval will facilitate the study of LLMs’ fundamental abilities.

## introduction large language models (llms) @xcite @xcite @xcite are gaining increasing popularity in the field of natural language processing (nlp), owing to their commendable performance in various applications. as the development of llms is blooming, there has been a surge of evaluation work to probe the diverse capabilities of llms. ## related work recent advancements in large language models have attracted significant interest, with the depth and breadth of evaluation work consistently expanding. broadly speaking, these evaluations can be classified into two distinct categories: objective evaluations and subjective evaluations. objective evaluations typically adopt formats such as multiple-choice queries and some openended questions with definitive responses. a large proportion of the multiple-choice benchmarks @xcite @xcite are task-oriented, primarily assessing the model's question-answering capabilities. open-ended questions frequently encompass a range of knowledge queries, as exemplified by the naturalquestions @xcite and triviaqa @xcite . apart from knowledge, reasoning capabilities are often a key focus of evaluation, such as gsm8k @xcite and theoremqa @xcite . however, a notable limitation of objective evaluations is their misalignment with human, leading to high scores but do not correlate with users' experience. subjective evaluations, on the other hand, aim to harmonize with human experiences, primarily gauging the ability to adhere to instructions and engage in dialogues. these evaluations typically employ a scoring system for llms apis, such as the gpt4.0 and gpt3.5. there's a wealth of research in this domain, including alignbench @xcite , which offers a comprehensive, multidimensional evaluation benchmark for chinese llm alignment, utilizing a rule-based language model for evaluation. alpacaeval @xcite provides a fully automated evaluation benchmark based on the llm and employing gpt4.0 or claude as automatic evaluators. the benchmark compares the target model's responses with those of gpt3.5 and calculates the win rate. several studies @xcite @xcite have also focused on how to utilize llms for scoring. however, current subjective evaluations heavily rely on references. in scenarios where references are not available, the quality of llms' results is limited, failing to accurately reflect the ability partial order of llms. we propose a new evaluation dataset and evaluation method that can address these issues. it can examine both base models and chat models. at the same time, it allows subjective evaluations to have higher credibility and greater distinction when there is no reference. ## benchmark to assess the fundamental capabilities of llms, we design f-eval to examine the model's fundamental abilities from 3 dimensions and establish corresponding appropriate evaluation methods for each sub-dataset. ## experiments in this section, we conduct experiments to evaluate the performance on various llms on f-eval using opencompass @xcite . then, we pay our attention on two aspects: the evaluation methods' agreement with human judgements, and the distinction of the evaluation scores. settings when designing prompts, we directly provide the base model with texts that need to be continued or questions that need to be answered, without any additional instructions, which ensures that the evaluation of fundamental abilities is not limited by instruction-following abilities. for llms that default to a chat format, we add relevant instructions to the above prompts, such as "please complete the text" or "please answer the following question". in our experimental setup, three sub-datasets are evaluated in a few-shot setting. the icl sub-dataset examine whether llms can induce information from in-context examples, while commonsense triple and coreference use incontext examples to enable the model to learn for continuation, without explicit instructions. apart from them, the remaining sub-datasets are all evaluated in a zero-shot setting. specific prompts and settings for each sub-dataset are described in appendix a. models we evaluate 13 advanced llms from 6 model series in various sizes. for commercial models, we evaluate gpt series @xcite , while for open-source models, we select llama2 @xcite , baichuan2 @xcite , qwen @xcite , chatglm @xcite and @xcite series. notably, we choose the base model of open-source models to better examine the fundamental abilities for the existence of alignment tax @xcite . we introduce a detailed description of each llm in appendix c. ## discussion the impact of the model size on performance across three dimensions. in order to investigate how the performance of llms improves with the increase in the model size, we categorize the selected open-source models into three levels: smallscale models with 7b or less, medium-scale models between 7b and 20b, and large-scale models ranging from 60b to 80b. we depict the score trends of open-source models across three model scales, as well as those of api models in figure 6 . it is clear that the performance in each dimension increases with the enlargement of the model size. specifically, we observe that increasing the model size from small scale to medium scale obviously enhances the ability of logic, while the performance of expression and commonsense only have a tiny improvement. with further enlargement, a substantial improvement is observed in all dimensions. based on the observation above, we speculate that with parameters bigger than 80b, the model should exhibit better fundamental capabilities. however, the figure clearly shows that current open-source llms significantly lag behind api models in every dimension. therefore, there is still a considerable journey ahead in our exploration of llms. the ability imbalance of each llm across three dimensions. as shown in figure 4 , the overall performance of the llms is not completely consistent with its performance in each dimension. to further explore whether each model's abilities are balanced across 3 dimensions, we compare rankings of the overall results and those in each dimension in self-adaptive normalization v.s. rank standard normalization as mentioned in section 3.3, we design a self-adaptive normalization method to substitute the rank standard normalization. to compare these two normalization methods, we also present the correlation coefficient when using rank standard normalization in ## conclusion we introduce f-eval, a bilingual evaluation benchmark that focuses on the fundamental abilities of large language models within 3 dimensions, covering both objective and subjective tasks. for reference-free subjective tasks, we design more distinctive evaluation methods as an alternative to api scoring. additionally, we develope a new self-adaptive normalization method to accurately and effectively combine scores from different subdatasets. experiments have shown that f-eval's correlation coefficients across 13 advanced llms surpass those of other evaluation baselines. we hope our benchmarks can empower researchers to better enhance the fundamental abilities of llms during every stage.