Self-Evaluation of Large Language Model based on Glass-box Features (2024)

Abstract: The proliferation of open-source Large Language Models (LLMs) underscores the pressing need for evaluation methods. Existing works primarily rely on external evaluators, focusing on training and prompting strategies. However, a crucial aspect – model-aware glass-box features – is overlooked. In this study, we explore the utility of glass-box features under the scenario of self-evaluation, namely applying an LLM to evaluate its own output. We investigate various glass-box feature groups and discovered that the softmax distribution serves as a reliable quality indicator for self-evaluation. Experimental results on public benchmarks validate the feasibility of self-evaluation of LLMs using glass-box features.

## introduction recently, as the large language models (llms) brings a storm to the area of artificial intelligence, evaluating the quality of llm outputs also draws a lot of research concentration. as the ability of llms develop beyond limitation, it is essential to evaluate them from a comprehensive and scalable perspective. however, traditional evaluation metrics for generative models, such as bleu and rouge, only capture limited aspects of a model's performance @xcite . some research has proposed language modelbased evaluation @xcite , leveraging proprietary llms such as gpt-3.5 or gpt4 @xcite , to evaluate the llm's outputs. however, relying on external api for evaluation may introduce consideration about privacy leakage, and the opacity of api models also challenges the evaluation reproducibility. other works propose to fine-tune open-source models as specialized evaluators @xcite @xcite . however, constrained by the capability of foundation model, these finetuned evaluators severely underperform gpt4 on generalizability and fairness @xcite . in this work, we take a novel approach to llm evaluation: is llm capable of self-evaluation? to answer this, we delve into the utility of glass-box features, namely the useful information that can be extracted from the model as a by-product of generation. concretely, we explore three groups of glass-box features: 1) softmax distribution, 2) uncertainty estimation and 3) attention distribution. our findings reveal that manipulating the softmax distribution by calculating its entropy and variance exhibits a strong correlation with annotated evaluation results. furthermore, we propose two strategies to enhance the evaluation by incorporating features derived from references. we conduct our experiments on two widely used llm evaluation benchmarks, mt-bench @xcite and vicuna-bench @xcite . experimental results notify that the llm is capable of providing accurate self-evaluation, surpassing proprietary evaluators such as gpt-3.5 and auto-j. the self-evaluation capability of llms holds promise for various applications, ranging from self-reflection to reward modeling. ## glass-box features for self-evaluation we assume an llm architecture based on transformer networks @xcite , which is currently the mainstream llm architecture. in this section, we introduce the three groups of glass-box features for self-evaluation. ## self-evaluation with reference when evaluating the llm's response to an instruction, a reference answer might be available. in this work, we introduce two strategies to effectively utilize the references for self-evaluation. ## conclusion in this work, we investigate the utility of glass-box features for llm self-evaluation. our findings verify that the confidence quantified by softmax distribution can be a reliable quality indicator. the selfevaluation ability of llm is a promising pathway, for example, the llm can rely on self-evaluation results to decide whether to perform self-reflection @xcite , or to select preferred data for reward modeling @xcite , and we leave these as the future work.