Annotation-Efficient Language Model Alignment via Diverse and Representative Response Texts (2025)

Abstract: Preference optimization is a standard approach to fine-tuning large language models to align with human preferences. The quantity, diversity, and representativeness of the preference dataset are critical to the effectiveness of preference optimization. However, obtaining a large amount of preference annotations is difficult in many applications. This raises the question of how to use the limited annotation budget to create an effective preference dataset. To this end, we propose Annotation-Efficient Preference Optimization (AEPO). Instead of exhaustively annotating preference over all available response texts, AEPO selects a subset of responses that maximizes diversity and representativeness from the available responses and then annotates preference over the selected ones. In this way, AEPO focuses the annotation budget on labeling preferences over a smaller but informative subset of responses. We evaluate the performance of preference learning using AEPO on three datasets and show that it outperforms the baselines with the same annotation budget.

## introduction language model alignment aims to address these issues by guiding large language models (llms) to generate responses that aligns with human preferences, steering them to generate responses that are informative, harmless, and helpful @xcite @xcite @xcite . the performance of the alignemnt algorithms is highly dependent on the choice of the preference dataset. however, building a human preference dataset requires expensive human annotations, which is the major bottleneck for constructing a large and high quality preference dataset. a large number of works have investigated the synthesis of preference data using a powerful llm (e.g., gpt-4) to distill the knowledge of human preferences @xcite @xcite @xcite @xcite . however, human preferences are known to be diverse and pluralistic, and they are unlikely to be represented by the opinion of a single model @xcite @xcite @xcite @xcite @xcite @xcite @xcite . several papers have pointed out that llms may exhibit bias toward aligning with people from a particular background @xcite @xcite . for example, @xcite reports that chatgpt has a strong alignment with american culture, but adapts less effectively to other cultural contexts. in addition to cultural biases, previous work suggests that even a highly capable model (e.g., gpt-4) still has biases such as length bias @xcite , style bias @xcite , and positional bias @xcite . thus, human annotation is desirable to align and personalize an llm with diverse and unbiased human preferences @xcite @xcite . the efficiency of annotation is critical to making llms accessible and useful to people from diverse backgrounds, who may have only a small amount of preference feedback data to work with. to this end, we investigate how to generate an effective preference dataset with a limited human annotation budget. previous work has shown that the following three features are desirable for a preference dataset to be effective @xcite : greater quantity and diversity are desirable for the instruction set @xcite @xcite @xcite @xcite @xcite . ## annotation-efficient preference optimization (aepo) we propose annotation-efficient preference optimization (aepo), a method for efficiently learning preferences from a large number of responses with a limited budget on preference annotations ## experiments we first conduct simulated experiments using reward models instead of human annotation to evaluate the performance of aepo in depth with ablation studies. then, we evaluate aepo on a more realistic setting of learning cultural commonsense morality @xcite where human annotations are difficult to obtain. ## conclusions we propose annotation-efficient preference optimization (aepo), an annotation-efficient dataset subsampling strategy for language model alignment. aepo selects response texts to annotate from candidate responses, maximizing the representativeness and diversity. by focusing the annotation effort on the selected responses, aepo achieves efficient preference optimization under a limited annotation budget. the experimental results and ablation studies (appendix c) show that aepo consistently outperforms the baseline strategies on a wide range of benchmarks. we believe that aepo is a critical contribution to promoting preference optimization in low resource settings by addressing the severe obstacle, the cost of creating better preference data.