I nstruct E val: Towards Holistic Evaluation of Instruction-Tuned Large Language Models (2024)

Abstract: Instruction-tuned large language models have revolutionized natural language processing and have shown great potential in applications such as conversational agents. These models, such as GPT-4, can not only master language but also solve complex tasks in areas like mathematics, coding, medicine, and law. However, there is still a lack of comprehensive understanding regarding their full potential, primarily due to the black-box nature of many models and lack of holistic evaluation. To address these challenges, we present InstructEval, a more comprehensive evaluation suite designed specifically for instruction-tuned large language models. Unlike previous works, our evaluation involves a rigorous assessment of models based on problem-solving, writing ability, and alignment to human values. We take a holistic approach to analyze various factors affecting model performance, including the pretraining foundation, instruction-tuning data, and training methods. Our findings reveal that the quality of instruction data is a crucial factor in scaling model performance. While open-source models demonstrate impressive writing abilities, there is substantial room for improvement in problem-solving and alignment.

## introduction the advent of instruction-tuned large language models has marked a significant turning point in the field of natural language processing (nlp). their transformative capabilities are evident in numerous applications, from conversational assistants such as chatgpt foot_0 to complex problem-solving. examples of such models include @xcite , which has shown proficiency not only in language understanding but also in areas as diverse as mathematics, coding, medicine, and law. however, despite their remarkable proficiency and adaptability, the full extent of their potential remains to be comprehensively understood. this situation arises primarily due to the black-box nature of many models and the current absence of in-depth and holistic evaluation studies. to address these challenges and gain a deeper understanding of the capabilities of these models, we introduce a novel evaluation suite named in-structeval. this suite is designed explicitly for the comprehensive assessment of instructiontuned large language models, pushing beyond the confines of earlier evaluation approaches. our evaluation strategy diverges from prior studies in its systematic and holistic approach. it not only scrutinizes the models' problem-solving abilities and writing proficiency but also critically examines their alignment with human values. at the heart of our evaluation methodology, we consider various factors affecting the performance of the models. these include the pretrained foundation upon which the models are developed, the nature and quality of instruction-tuning data used to refine them, and the specific training methods adopted. through a rigorous exploration of these factors, we seek to shed light on the vital elements that determine model performance, facilitating an understanding of how these models can be better harnessed to meet our needs. our research findings underscore the critical influence of the quality of instruction data on the scaling of model performance. open-source models have shown impressive writing abilities, signifying their potential to contribute meaningfully to various domains. however, our study reveals considerable room for improvement, particularly in the models' problem-solving abilities and alignment with human values. this observation accentuates ## instructeval benchmark suite to address the challenges of assessing instructed llms discussed in section 3, we introduce a more holistic evaluation suite known as instructeval. to cover a wide range of general abilities, we test the models in terms of problem-solving, writing, and alignment to human values, as shown in figure 1 . as instructeval covers tasks that can be objectively scored, as well as tasks that need to be qualitatively judged, we adopt multiple evaluation methods. we also include the full evaluation data statistics and implementation in the appendix. ## conclusion instruction-tuned large language models have transformed natural language processing and demonstrated significant potential in various applications. however, more comprehensive evaluation is needed due to limited understanding caused by the black-box nature of many models and the lack of holistic evaluation studies. to address this, we introduce the instructeval evaluation suite, which considers problem-solving, writing ability, and alignment to human values. the findings highlight the importance of high-quality instruction data for scaling model performance. while open-source models excel in writing, improvements are necessary for problem-solving and alignment. we hope that instructeval inspires more rigorous evaluation and understanding of instructed llms.