Toward the Evaluation of Large Language Models Considering Score Variance across Instruction Templates (2024)

Abstract: The natural language understanding (NLU) performance of large language models (LLMs) has been evaluated across various tasks and datasets. The existing evaluation methods, however, do not take into account the variance in scores due to differences in prompts, which leads to unfair evaluation and comparison of NLU performance. Moreover, evaluation designed for specific prompts is inappropriate for instruction tuning, which aims to perform well with any prompt. It is therefore necessary to find a way to measure NLU performance in a fair manner, considering score variance between different instruction templates. In this study, we provide English and Japanese cross-lingual datasets for evaluating the NLU performance of LLMs, which include multiple instruction templates for fair evaluation of each task, along with regular expressions to constrain the output format. Furthermore, we propose the Sharpe score as an evaluation metric that takes into account the variance in scores between templates. Comprehensive analysis of English and Japanese LLMs reveals that the high variance among templates has a significant impact on the fair evaluation of LLMs.

## introduction decoder-based large language models (llms) have become foundational resources in the field of natural language processing, demonstrating superior natural language understanding (nlu) abilities and high pre-trained knowledge capacity in a wide variety of downstream tasks. recently, llms can produce more human-like responses through instruction tuning @xcite , which involves training the llms to respond appropriately to user instructions for various tasks. although llm performance has been evaluated across various nlu tasks, the evaluation processes lack standardization in terms of prompts and output formats. this lack of standardization leads to differences in evaluation outcomes that cannot be attributed solely to the differences among llms. moreover, the differences in prompts used for evaluation affect the evaluation results in nlu tasks @xcite @xcite @xcite @xcite @xcite . in the specific case of instruction tuning, the goal is a prompt-independent generalization, though it is questionable to measure such generalization performance using prompts designed for specific targets. for fair evaluation and comparison of the nlu performance of llms, we created benchmark datasets comprising multiple evaluation instruction templates for each nlu task based on the flan templates @xcite , using five english nlu tasks and their corresponding japanese tasks based on jglue @xcite . additionally, we proposed a new evaluation metric, the sharpe score, which accounts for the variance in llm outputs due to template differences, inspired by the sharpe ratio @xcite used in finance to assess investment efficiency. we demonstrated its effectiveness for the evaluation of template-based nlu capability, as well as for analysis of the nlu performance of multiple llms in various experimental scenarios, such as zero-shot versus fine-tuning settings and english versus japanese settings. we examined how factors such as continuous training, instruction tuning, and language-specific knowledge affect knowledgetransfer capability. in order to enforce output generation in line with the expected response format, we accompanied each instruction template with a regular expression of the expected output for each task. the regular expressions are employed in constrained decoding methods as implemented in outlines @xcite . we experimented with both constrained decoding and greedy decoding, demonstrating that constrained decoding with regular expressions is effective for zero-shot evaluation. our datasets and evaluation scripts are available at https://github.com/naist-nlp/vite . ## background and related work the evaluation of the nlu capability of llms has mostly been based on benchmark datasets that combine several nlu tasks, such as glue @xcite and superglue @xcite . furthermore, nlu datasets that include domainspecific knowledge such as medical, economic, and mathematical knowledge @xcite @xcite @xcite @xcite @xcite @xcite have been proposed for testing domain specific knowledge in llms. these benchmark datasets generally use automatic evaluation metrics, such as accuracy and f1 score. these datasets are typically constructed in a concise format relevant to the particular task, providing only minimal information, such as questions and their answers. therefore, the standard practices in evaluating llms employ instruction templates to make the datasets easy for llms to understand the instructions. the data instances are instantiated with instruction templates to yield natural language sentences, from which llms infer answers in an autoregressive manner. benchmark datasets for several languages other than english are available as well. datasets for japanese, a language we focus on in this study, include llm-jp-eval foot_0 , jp language model evaluation harness foot_1 , and nejumi foot_2 , all of which employ japanese nlu datasets centered around jglue @xcite . the jp language model evaluation harness uses llms as classifiers by combining each question with corresponding answer choices and selecting the one with the lowest perplexity when all choices are ranked. llm-jp-eval and nejumi perform automatic evaluation by postprocessing the generated text. in the evaluation used by nejumi, if an answer cannot be obtained from the generated text, it assigns an arbitrary label, whereas the llm-jp-eval treats it as incorrect 4 . ## evaluation method in our evaluation, we focus on the variance in results caused by differences in templates. to this end, we propose datasets and methods for evaluating the nlu performance of llms using multiple instruction templates. we evaluate performance in zero-shot and fine-tuning settings, but omit incontext learning, i.e., few-shot learning, settings. the prior studies @xcite have shown that the few-shot setting merely represents the exploration for optimal input prompts, capped by the performance of fine-tuning under the same number of examples. ## experimental results and discussions results on the japanese benchmark dataset are shown in tables 3 and 4 for the zero-shot and fine-tuning setting, respectively. similarly, results on the english benchmark dataset are shown in tables 5 and 6 for the zero-shot and fine-tuning setting, respectively. note that the results for the english benchmark dataset exclude the japanese llms listed in ## conclusion in this paper, we focused on the variance in the evaluation results of llms caused by the variations in instruction templates. we proposed a cross-lingual benchmark dataset based on multiple instruction templates, and reported the evaluation results of models trained on varied data. we also proposed the sharpe score, which considers the variance in evaluation scores among templates, and demonstrated that it is necessary to consider variance when evaluating llm performance. based on a comparison of diverse llms using our dataset and an analysis of the results, we focused on the tasks where cross-lingual knowledge is effective and the effectiveness of llms created for specific languages such as japanese. an issue closely related to what we touched upon in section 4.1, i.e., the catastrophic forgetting due to continuous training and instruction tuning, is already being studied @xcite @xcite , and our dataset may help in analyzing the knowledge and cross-lingual capability of llms in more detail. future llm development would also benefit from a study verifying the extent of knowledge acquisition and the effects of instruction tuning after different sequences of pre-training, instruction tuning, and continuous training. as a future work, we intend to conduct further analyses and to create a comprehensive evaluation framework for analyzing the nlu capabilities of llms by expanding the proposed dataset.