Poor-Supervised Evaluation for S uper LLM via Mutual Consistency (2024)

Abstract: The guidance from capability evaluations has greatly propelled the progress of human society and the development of Artificial Intelligence. However, as LLMs evolve, it becomes challenging to construct evaluation benchmark with accurate labels for SuperLLMs whose capabilities approach or even surpass those of humans. To credibly conduct poor-supervised evaluation without accurate labels, we first prove that the consistency between the model under evaluation and the reference model, when their prediction distributions are independent and the sample size is infinite, can equivalently assess the true capabilities of the model to be evaluated. However, using either humans or LLMs as the reference model cannot sufficiently meet the conditions, for which we propose the PEEM algorithm. By treating all models under evaluation as reference models, PEEM alternately optimizes model weights and filters reference models based on EM algorithm to maximally alleviate the insufficiency of the conditions. Comprehensive experiments across 3 types of tasks with 16 mainstream LLMs validate the efficiency, universality, and effectiveness of PEEM. More generally, PEEM has advanced the evaluation paradigm evolution from human-centric to human&model-centric, alleviating the limitations of human capabilities for evaluating SuperLLMs.

## introduction evaluations, such as iq tests and olympic competitions, effectively identify areas of strength and weakness for individuals or groups, providing valuable guidance for the enhancement of various human abilities. in the field of artificial intelligence, benefiting from high-quality annotated data (deng et @xcite , neural network models have been well evaluated and optimized specifically, leading to significant advancements across various tasks. nowadays, advanced large language models (llms) @xcite , called superllms @xcite , have reached the level of human experts in certain tasks @xcite @xcite . to further guide superllms towards agi, evaluating them on a wider range of challenging tasks is of crucial importance. however, building benchmarks for challenging tasks is especially intelligence-intensive and sometimes even difficult to ensure the accuracy of labels. as shown in figure 1 , annotating hard tasks requires scarce human experts and some tasks are even beyond the capability boundary of experts, such as "is the riemann hypothesis true?". so here comes the question: how to evaluate llms accurately on benchmarks with poor supervision (with inaccurate or even no labels)? some works have explored unsupervised (labelfree) model evaluation, including using llm-based pseudo labels @xcite , evaluating with logits @xcite , and detecting conflicts with designed inputs @xcite . unfortunately, they are not proposed for evaluating superllms, presenting issues such as limited applicability across tasks, pseudo labels may still be inaccurate, logits of some llms are unavailable (e.g., gpt4), and dependence on prior knowledge. overall, superllms are in eager need of an accurate, universal, and efficient poor-supervised evaluation paradigm. consequently, we propose the following theorem and prove it in appendix §a. theorem 1. for benchmark x lacking true labels y , if reference model ṁ is independent of predictions { ŷ i } l @xmath0, and performs better than random guess (condition 2), when the size of x is infinite (condition 3), the following equation holds: theorem 1 allows us to credibly evaluate and compare the capabilities of models (consistency between their predictions and true labels) without true labels y . llms themselves or humans can naturally serve as reference model ṁ, but the conditions are not guaranteed to be sufficiently satisfied in real scenarios. some llms may exhibit mutual affinity and tend to make the same predictions (e.g., llama series), and the size of x is finite. thus, we propose peem algorithm (poor-supervised evaluation with em algorithm) to alleviate these insufficiency. peem takes models predictions (with optional human annotations) as inputs, and calculates the weighted average consistency of certain model with reference models as the measurement of its capability. all the models to be evaluated are treated as reference models and assigned equal weights initially. based on the em algorithm, peem iteratively adjusts the weights of reference models during e-step, and filters reference models based on current weights during m-step to optimize the proxy optimization objective inferred from preliminary experiments. peem mitigates the limitation of sample size (condition 3) by comprehensively utilizing the predictions of all models under evaluation. and it alleviates the dependency among model predictions (condition 1) by optimizing the weights of reference models and filtering out the ones with high affinity tendency. our poor-supervised evaluation paradigm based on peem algorithm holds several advantages. efficient: no additional model inference overhead is required except for model predictions. universal: it can be applied across various tasks by proper definition of cons(•), as we demonstrate in §5. accurate: experimental results show that peem aligns perfectly with true label evaluation results, achieving an average of 0.977 pearson and 0.972 spearman correlation coefficient. furthermore, we validate that peem with human annotations and model predictions, as opposed to the traditional consistency check between models and humans, allows for a more accurate evaluation results. more generally, peem has advanced the evaluation paradigm evolution from humancentric to human&model-centric, addressing the limitations of human evaluation capabilities. our contributions are summarized as follows: ## related work surrounding our study, we discuss the researches on poor-supervised evaluation and model consistency, and briefly introduce the em algorithm. ## methods in this section, we propose multiple mapping algorithms f to obtain b based on the insights above, as shown in figure 3 . for each algorithm, we first calculate the consistency matrix c according to ŷ and task types, thus f : @xmath1. ## experiments since it is impossible to obtain accurate true labels for samples that human capabilities can not handle well, theoretically, we cannot directly measure the effectiveness of the proposed algorithms on such benchmarks. thus we have designed a comparable experimental setup: we select benchmarks (math @xcite §5.1, usr (mehri and eskénazi, 2020) §5.2) that are currently considered fully manageable by human experts, and treat human predictions as the true labels. under this setup, we validate whether f p eem (cons(llms)) can align well with cons(llms, true label). furthermore, multirc @xcite §5.3 uses the predictions of single human annotator as human annotations and approximates the ensemble of predictions from multiple human annotators as true labels. we adopt this setting to verify if f p eem (cons(llms and human)) aligns better with cons(llms, true label) than traditional evaluation paradigm cons(llms, human). ## conclusions in this paper, we propose peem, an algorithm for conducting poor-supervised evaluation without true labels for superllms. comprehensive experiments validate the efficiency, effectiveness, and university of peem across 3 types of tasks with 16 mainstream llms. peem has advanced the evaluation paradigm evolution from human-centric to human&model-centric, alleviating the limitations of human capabilities for evaluating superllms. our work is a preliminary exploration of evaluating superllms by analogizing existing llms and benchmarks, longing for further improvement in the future.