Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluation (2024)

Abstract: In recent years, substantial advancements have been made in the development of large language models, achieving remarkable performance across diverse tasks.To evaluate the knowledge ability of language models, previous studies have proposed lots of benchmarks based on question-answering pairs.We argue that it is not reliable and comprehensive to evaluate language models with a fixed question or limited paraphrases as the query, since language models are sensitive to prompt.Therefore, we introduce a novel concept named knowledge boundary to encompass both prompt-agnostic and prompt-sensitive knowledge within language models.Knowledge boundary avoids prompt sensitivity in language model evaluations, rendering them more dependable and robust.To explore the knowledge boundary for a given model, we propose projected gradient descent method with semantic constraints, a new algorithm designed to identify the optimal prompt for each piece of knowledge.Experiments demonstrate a superior performance of our algorithm in computing the knowledge boundary compared to existing methods.Furthermore, we evaluate the ability of multiple language models in several domains with knowledge boundary.

## introduction recently, large language models (llms) have made significant advancements in a variety of tasks @xcite @xcite . in order to gain deeper insights into the knowledge capabilities of different llms to help select appropriate llm in practice, numerous studies have proposed various benchmarks for llm evaluation @xcite . the majority of previous research on model evaluation constructs a test dataset sourced from standardized examinations, such as college * these authors contributed equally to this work. ## method to obtain a better knowledge boundary for llm, our effort is directed toward identifying the optimal prompt within the semantic space. as illustrated in figure 2 , pgdc optimizes prompt in the neighbor semantic space of the original question. the prompt in text form is first mapped to prompt in embedding form as continuous text embedding. during pgdc optimization, the text embedding is updated through gradient descent with direction of the loss function φ(•). after each update, if the text embedding is close to a discrete prompt, it is projected to the discrete prompt through embedding projection. to avoid the text embedding from entering unprojectable region where there are no close discrete prompts to project, we introduce a regularization to force the embedding not to enter these regions. after multiple iterations of the update, we get the final optimized prompt. ## experiments in this section, we perform comparisons between our method and baseline methods which are commonly used in model evaluation on common knowledge benchmarks and unanswerable knowledge benchmarks. we also conduct a manual evaluation to check whether the semantics of the prompts we obtained are consistent with the original question. ## assessments of llms the above experiments have demonstrated the effectiveness of pgdc in detecting knowledge and the reasonableness of the obtained optimal prompt. in this section, we apply pgdc on mmlu @xcite to evaluate llms. ## discussion of randomness in model evaluation the inherent randomness in model evaluation presents significant challenges to the reliability of evaluation results and the process of model selection. this randomness can be seen in several key areas: benchmark question selection randomness is inherent in the selection of benchmark questions for testing, as the objective is to choose representative questions that accurately demonstrate the model's abilities in specific areas or domains @xcite . prompt expression formulating the selected questions into prompts introduces additional randomness, including aspects such as the phrasing of questions, system instructions, the order of options, and other details. numerous studies have demonstrated that models are sensitive to variations in prompts @xcite @xcite . decoding settings the choice of decoding parameters (e.g., temperature) during evaluation can introduce randomness into the model's output, potentially raising concerns about fairness. ## related work model evaluation several benchmarks have been proposed to evaluate large language models (llms) on human exams like college entrance and law school admission tests @xcite @xcite . in terms of knowledge assessment, lama @xcite evaluates whether models can correctly predict masked object entities in a cloze-style prompt. some studies @xcite @xcite focus on measuring llms' understanding and mastery of world knowledge. these benchmarks do not take into account that llms are sensitive to different prompts. some works @xcite focus on estimating and measuring the consistency of llms given diverse prompts. all previous studies on model evaluation use fixed prompts, and our work pioneers prompt optimization for evaluating llms' knowledge boundaries. prompt optimization due to the sensitivity of language models to prompts, better prompts can help achieve higher performance in specific tasks @xcite ; @xcite ). prompt engineering like in-context learning greatly improves the performance of prompt methods @xcite . another related line of work attempts to formalize prompt searching as a discrete optimization task to achieve better performance in specific tasks @xcite . some studies adopt hotflip-based algorithms @xcite to automatically construct prompts @xcite al. (2019); shin et al. (2020); jones et al. (2023) ). in addition, several work tries to optimize prompts in continuous embedding space with gumbel-softmax trick @xcite and projection ( cheng et al. (2020); wen et al. (2023) ). ## conclusion the sensitivity of llms to prompt leads to the unreliability of the results obtained from traditional model evaluation works that use fixed queries to evaluate the model. to address this problem, we propose semantics-preserving prompt optimization methods, pgdc, to find the knowledge boundaries of models for model evaluation. our experiments demonstrate shortcomings of previous model evaluation methods and the fact that the prompt we find is superior to the fixed prompt. at the same time, the prompt found by our method maintains the original semantics and does not induce knowledge that is not captured by the model, which outperforms previous prompt optimization efforts. moreover, we conduct experiments exploring the boundaries of the model's different domain knowledge and compare and analyze the llm's capabilities.