Instruction-following Evaluation through Verbalizer Manipulation (2024)

Abstract: While instruction-tuned models have shown remarkable success in various natural language processing tasks, accurately evaluating their ability to follow instructions remains challenging. Existing benchmarks primarily focus on common instructions that align well with what the model learned during training. However, proficiency in responding to these instructions does not necessarily imply strong ability in instruction following. In this paper, we propose a novel instruction-following evaluation protocol called verbalizer manipulation. It instructs the model to verbalize the task label with words aligning with model priors to different extents, adopting verbalizers from highly aligned (e.g., outputting “positive” for positive sentiment), to minimally aligned (e.g., outputting “negative” for positive sentiment). Verbalizer manipulation can be seamlessly integrated with any classification benchmark to examine the model’s reliance on priors and its ability to override them to accurately follow the instructions. We conduct a comprehensive evaluation of four major model families across nine datasets, employing twelve sets of verbalizers for each of them. We observe that the instruction-following abilities of models, across different families and scales, are significantly distinguished by their performance on less natural verbalizers. Even the strongest GPT-4 model struggles to perform better than random guessing on the most challenging verbalizer, emphasizing the need for continued advancements to improve their instruction-following abilities.

## introduction large language models have achieved remarkable success in zero-shot generalization for various natural language processing (nlp) tasks via instruction tuning @xcite @xcite . one representative model is chatgpt foot_0 , which has shown promising results in text summarization @xcite , coding (surameery and shakor, 2023), healthcare @xcite , education @xcite , finance @xcite and law @xcite . existing benchmark datasets @xcite @xcite primarily focus on common instructions that align well with what models learned during pre-training or instruction-tuning. however, proficiency in responding to these instructions does not necessarily imply strong ability in instruction following as models may rely on memorization of favorable responses rather than genuine generalization due to the vast volume of data they see during training @xcite . nonetheless, instruction following capability plays an important role in task generalization for real-world applications. for example, a user may want models to output answers only when they are certain to reduce hallucinations or control model response length or assign models with specific roles (e.g. tax expert). a natural question arises: how can we systematically and automatically evaluate instruction-tuned models in terms of instruction-following capability? in this paper, we propose to evaluate the instruction-following ability from the aspect of how well models can follow instructions that may not align with their priors and design a novel framework to synthesize them. specifically, we propose verbalizer manipulation foot_1 that can be used to construct instructions aligning with model priors to different extents, from natural, to neutral, to unnatural, as shown in figure 1 . in natural instructions, we choose multiple verbalizers that align with prior knowledge for each dataset. in neutral if a movie review is positive, you need to output "positive". if a movie review is negative, you need to output "negative". movie review: lovely and poignant. ## conclusion in this paper, we design a framework to evaluate the instruction-following capabilities of instructiontuned language models via verbalizer manipulations. we design three instruction-following evaluation sets, namely natural, neural and unnatural instructions, which align with prior knowledge to different extents. we evaluate four different model families on nine datasets across scales. our results show that although larger instruction-tuned models generally perform better on both natural and neutral instructions, their performance diverges significantly in unnatural instructions. we further examine verbalizers one by one in unnatural instructions, and find that the same model family performs significantly different on instructions with different verbalizers, even with more advanced zero shot cot prompting. these results show there still exist fundamental limitations within state-of-the-art instruction-tuned large language models in following human instructions. we hope that our work can inspire future research to focus more on instructionfollowing capabilities.