Label-Efficient Model Selection for Text Generation (2024)

Abstract: Model selection for a given target task can be costly, as it may entail extensive annotation of the quality of outputs of different models. We introduce DiffUse, an efficient method to make an informed decision between candidate text generation models based on preference annotations. DiffUse reduces the required amount of annotations, thus saving valuable time and resources in performing evaluation.DiffUse intelligently selects instances by clustering embeddings that represent the semantic differences between model outputs. Thus, it is able to identify a subset of examples that are more informative for preference decisions. Our method is model-agnostic, and can be applied to any text generation model for selecting between models, prompts and configurations. Moreover, we propose a practical iterative approach for dynamically determining how many instances to annotate. In a series of experiments over hundreds of model pairs, we demonstrate that DiffUse can dramatically reduce the required number of annotations – by up to 75% – while maintaining high evaluation reliability.

## introduction model evaluation is a prerequisite for informed decisions -predominantly, choosing the right model for the task. as such, an essential requirement is the ability to compare models based on how well they perform. moreover, since a given model can be configured in many ways, there is a need for an even wider range of comparisons. for instance, comparing different prompts a model is provided with can affect task performance significantly. comparing model performance generally requires some sort of oracle -a human annotator or llm-based evaluator -that can judge model outputs and prefer one output over another. however, * these authors contributed equally to this work. depending on the nature of the oracle, such judgments can incur significant costs, particularly in terms of annotation budgets @xcite and computational requirements @xcite @xcite . specifically for text generation tasks, the oracle is burdened with making nuanced judgments of the quality of generated texts @xcite ; often, this can only be done by expert human annotators (van der @xcite , or possibly by powerful llms (e.g., @xcite , both of which are costly to apply at scale. moreover, as the number of models and tasks increases, conducting these evaluations becomes prohibitively expensive @xcite . our goal is to address the costs associated with evaluating model outputs in text generation, by reducing the burden on the oracle. to our knowledge, this goal has not been addressed in the literature. specifically, we focus on the use case of directly comparing two candidate models, where the oracle is asked to make preference judgements between the outputs generated by the two models. our focus is on comparative judgments and not absolute scores, as these are considered more reliable for evaluating text generation @xcite @xcite . in this work, we propose a method that substantially reduces the number of examples that must be annotated by the oracle, while yielding a more reliable estimate of the preferred model for the task. our approach -diffuse -selects pairs of model outputs that on the one hand are representative of the space of differences between model behaviors on a given task, and on the other hand are more informative, showing clearer preference. specifically, we calculate embedding vectors that represent the semantic difference between the outputs of the two models; then, by partitioning these embeddings into clusters, we can intelligently select a diverse informative subset of instances for annotation. diffuse is inherently generic and does not assume anything about the models, tasks, unlabeled test data, prompts, or model hyper-parameters. our results ( §6) demonstrate its stability and effectiveness for different text generation tasks, across hundreds of pairs of generative models, and across a broad range of annotation budgets. one representative example of this can be seen in figure 1 . we also propose an iterative real-world solution for practitioners ( §6.2), which enables making reliable and cost-efficient choices between candidate models. we find this method to be better in all of our experiments, achieving a reduction in annotations of up to 75% compared to random sampling. furthermore, we conduct a comprehensive analysis ( §7) of the components of our method. our findings suggest that our method tends to select examples from regions in the output-difference space that are dominated by the preferred model. ## definitions and problem formulation in this work, we focus on comparative evaluation of models. given two text generation models, we wish to evaluate which one is stronger with respect to a given generation task, based on preference labels of an oracle over the model outputs. for an input instance x and a pair of models m a ,m b with corresponding outputs y a ,y b , a preference label y @xmath0, m b , t } indicates whether y a is better than y b (m a ), worse than y b (m b ) or similar to y b (t ). test winning model the model for which the outputs over the test set d test are more frequently preferred by the oracle. formally: ## method our algorithm, diffuse, is simple and effective, and relies solely on the outputs generated by the models. the full flow is described in figure 2 . we aim to represent examples in a manner that captures the distribution of model mismatching behaviors, i.e., various types of differences between model outputs. to this end, we first embed model outputs into a semantic vector space (using off-theshelf methods). subsequently, we generate difference vectors by subtracting the embeddings of one model from the embeddings of the other, for each example in the test set. then, we cluster these difference vectors and select one representative from each cluster to be labeled by the oracle. given the construction of the difference vectors, we expect this vector space to largely carry information about semantic differences, i.e., the nature of disagreements between models. choosing an example from each cluster ensures that the set of selected examples is representative of this space; hence, these examples are expected to be informative for estimating which is the preferred model. ## results we start by comparing the success rate of diffuse to that of the random selection baseline. figure 4 illustrates two such comparisons, each for a specific pair of models. as can be seen, success rates can vary greatly between cases where there is a relatively large performance difference between the generative models (left panel) and those with a small performance difference (right panel). as for the latter, estimating the preferred model is harder and requires more annotated instances. naturally, the model preference estimation becomes more accurate as the budget n increases and the preference decision relies on a larger set of examples annotated with oracle preference. in the two cases presented in fig. 4 , diffuse achieves higher success rates at identifying the better generative model, in comparison to random sampling. these results showcase that with diffuse one can reach the correct decision with a smaller number of examples to be annotated by the oracle. to give a broader and quantitative picture, figure 1 depicts the aggregated results for the cnn/dailymail summarization data, averaged across all 666 model pairs. the plot demonstrates a clear advantage of our approach over random selection, arriving at the correct decision more often and using fewer examples. thus, using diffuse there is a much lower risk of choosing the wrong model. this pattern is quite consistent across the different datasets tested, as can be seen in appendix a.1. note that while diffuse demonstrates a clear advantage, its effect does vary across datasets, and across "oracles" (in our case, different referencebased metrics). ## related work in light of the soaring costs of language model evaluation, even when using automatic metrics, some recent works @xcite have studied the effects of reducing the size of evaluation sets -via random sampling -on the reliable ranking of models. other prior works have examined methods of intelligently selecting subsets of examples for evaluation, aiming to find sets of examples that are more informative than randomly sampled instances. @xcite ; @xcite look at selecting examples for evaluating new models, given fully-annotated question answering data for an existing set of models. they show that some selection strategies, based on item response theory @xcite , outperform the random selection baseline for ranking new models on a question answering task. several works have addressed label-efficient assessment in the context of classifier performance. @xcite propose a label-efficient algorithm to gain better accuracy estimates of classifiers, by selecting examples to label based on stratified sampling. @xcite suggest an active bayesian approach that uses inferred uncertainty to guide selection of instances. inspired by works on active learning, @xcite unlabeled test set, and labels examples that lead to maximal uncertainty reduction of the metric estimation. with a similar spirit to our work, @xcite find anchor examples in classification datasets that represent how confident different models are over those input examples. our work differs from these prior efforts in that we tailor our approach to the nature of text generation. existing methods for example selection are not easily adapted from classification to generation tasks -the concepts of uncertainty, confidence and errors are inherently different for natural language generation, necessitating a different approach. in addition, unlike e.g., @xcite ; @xcite , our method does not require any annotations and assumes only a set of model outputs. ## discussion we have demonstrated that our method, diffuse, provides significant cost savings in model selection. we tested the approach for choosing the better underlying model as well as the most effective incontext prompt. given the generality of the method, this likely means diffuse is applicable to a vast range of model configuration and hyper-parameter choices. moreover, using a dynamic algorithm such as the one proposed here ( §6.2), practitioners can reduce the number of oracle judgements while maintaining high evaluation reliability. here we examined the problem of selecting between a pair of candidate models. we leave to future work the scenario of picking from a larger set of candidates. this may entail adapting our method to a multi-model scenario, or combining our pairwise approach with an efficient method for limiting the number of pairwise comparisons (e.g., @xcite . while the current work deals with model selection, our approach of modeling differences between outputs can potentially be applicable for other purposes as well. this can include qualitative assessment of model behaviours, collection of preference data for training reward models, and more. moreover, the proposed approach can be easily adapted to domains beyond nlp such as vision and speech.