TOWER : Tree Organized Weighting for Evaluating Complex Instructions (2024)

Abstract: Evaluating the ability of large language models (LLMs) to follow complex human-written instructions is essential for their deployment in real-world applications. While benchmarks like Chatbot Arena use human judges to assess model performance, they are resource-intensive and time-consuming. Alternative methods using LLMs as judges, such as AlpacaEval, MT Bench, WildBench, and InFoBench offer improvements but still do not capture that certain complex instruction aspects are more important than others to follow.To address this gap, we propose a novel evaluation metric, TOWER, that incorporates human-judged importance into the assessment of complex instruction following. We show that human annotators agree with tree-based representations of these complex instructions nearly as much as they agree with other human annotators. We release tree-based annotations of the InFoBench dataset and the corresponding evaluation code to facilitate future research.

## introduction the ability of large language models (llms) to follow human-written instructions is crucial for real world applications. despite this, evaluating and fairly comparing the ability of these llms to follow instructions remains a significant challenge, with many proposed approaches. for instance, chatbot arena employs pairwise comparison with crowd sourced human judges from a diverse user base to score the performance of language models @xcite . ideally, a model is judged purely on human feedback as in the case of chatbot arena, but this is generally expensive, takes lots of time, and is difficult to reproduce. further, results can differ based on the selection of the annotators and the distribution of instructions used for evaluation. some approaches such as alpacaeval and mt bench use llms instead of humans as judges, which significantly reduces cost, improves reproducibility, and is much faster @xcite @xcite . however, in practice many real-world use cases involve instructions which are much more complex than the ones found in many instruction following benchmarks. some recent promising work has been done creating evaluation benchmarks for these complex instructions, which contain many atomic aspects. one such benchmark, infobench, breaks 500 instructions into decomposed requirements, and propose decomposed requirements following ratio (drfr) as a metric for measure the percentage of requirements in the benchmark followed by a language model @xcite . despite these improvements it remains unclear how to best measure performance on complex instruction following, as some aspect of a complex instruction may be more important than others. in this work, we aim to remedy this issue by proposing a new complex instruction following evaluation metric, tower, which weights com-plex instruction requirements based on human judged importance. to summarize, our main contributions are as follows: ## related work the development of language models capable of following human-written instructions has seen significant advancement in recent years. a large variety of training approaches have been proposed for training these models @xcite @xcite @xcite @xcite significant research attention has been recently paid to evaluating the ability of llms to follow instructions. some approaches rely on crowd sourcing pairwise human feedback to allow humans to directly judge which model's generation is preferred @xcite . many works have aimed to automate this process by using llm's instead of humans to judge whether an instruction has been properly followed @xcite @xcite . some work has been done to evaluate the ability of language models to follow complex instructions which are made of many atomic aspects @xcite . however, relatively little work has been done to evaluate the importance of each aspect question within the context of the entire instruction and how these should be measured or evaluated. ## benchmark existing approaches for complex instruction following evaluation such as infobench rely on prompting a language model with a complex instruction. unlike many prior approaches for evaluation of instruction following, a judge is then asked a series of questions about the generated text, each addressing one aspect of the complex instruction. for instance, if the instruction asks for a letter from a parent to their child that is warm and supportive, two aspect questions may be "is the generated text suitable from a parent to their child?" and "is the letter warm and supportive?". these aspect questions are created by human annotators prior to evaluation and are held constant for each model being evaluated. existing metrics for complex instruction following such as decomposed requirements following ratio (drfr) from infobench weight all instruction aspects equally @xcite . however, this is not ideal. in many cases it may be important to weight adherence to higher level aspects such as system-level instructions stronger than adherence to lower level aspects. to that end, we propose a new instruction aspect weighting scheme which aims to weight the individual aspects of a complex instruction by their relative importance. as manually annotating the importance of each aspect is a lengthy and prohibitively costly process, we instead opt to use automated annotation and measure agreement with human evaluators. this also gives us freedom to explore a variety of different weighting approaches with minimal extra cost and to ensure reproducibility. we propose three different candidate approaches for automated labeling the relative importance of complex instruction aspects: direct scoring, ranking, and tree-based weighting. in the end we find tree-based importance weighting, which we refer to as tower, agrees best with human evaluators. details on our human annotation process are found in section 4.4. ## experiments here we discuss our experiments showing that our proposed tree-based weighting metric, which we refer from here on as tower, strongly aligns with human preferences and provides insight into the complex instruction following capabilities of various models. ## conclusion and future work in this paper we explore a new metric for evaluating large language models' ability to follow complex instructions. surprisingly, we find tree-based representations have nearly the same spearman correlation with human annotators (0.72) that human annotators have with themselves (0.74), indicating that tree-based methods are a strong candidate for representing and evaluating complex instructions. based on this observation our new proposed metric, tower, provides strong insights into which models are better at addressing the most important aspects of complex instructions. there are numerous exciting directions for future work. while progress is being made on designing new metrics for evaluating complex instruction following, more benchmark datasets focused on complex instructions are needed as few currently exist in this domain. additionally, finding ways to reduce evaluation costs is crucial, as evaluating a single model in infobench currently costs approximately $15.