ZEBRA : Leveraging Model-Behavioral Knowledge for Zero-Annotation Preference Dataset Construction (2025)

Abstract: Recent efforts in LLM alignment have focused on constructing large-scale preference datasets via human or Artificial Intelligence(AI) annotators. However, such approaches rely on instance-wise supervision, incurring substantial annotation cost and limited interpretability. In this paper, we propose **ZEBRA**—a model behavior-wise zero-annotation framework that constructs preference data by leveraging model behavior knowledge derived from benchmark performances.ZEBRA binarizes response pairs by evaluating the quality and similarity of their origin models, entirely bypassing instance-level annotation. This allows scalable, controllable, and cost-effective alignment data generation. Empirical results show that ZEBRA achieves alignment performance comparable to instance-supervised methods, despite requiring no manual or model-based labeling.

## introduction aligning large language models (llms) with human preferences is an essential step toward making them both useful and safe. a common way to achieve this is through instance-wise labeling, where pairs of model responses are compared one by one to see which is better. well-known methods like reinforcement learning from human feedback ( rlhfouyang et al. (2022a) however, instance-wise labeling faces two major challenges. first, it is very costly, whether it involves human annotators or additional computational resources for llm-based labeling @xcite . second, it lacks a global view of the model's behavior @xcite . since each response pair is judged in isolation, it is difficult to consider broader factors. for example, whether fluency should outweigh factual accuracy. or whether a model's outputs are consistently aligned with certain policies @xcite . this can lead to labeling noise, mistakes, and limited interpretability. to address these limitations, we propose a new preference binarization approach called zeroannotation behavior-based response alignment (zebra) (figure 1 ). the main idea is: (1) extract each model's behavioral patterns from its past performance trajectories, (2) measure and compare these behaviors in terms of model strength or similarity, and (3) assign preferences at the model level rather than for each individual response pair. we implement this idea through three key components. first, we define model behavior knowledge (mbk) for llms (discuss in section 4.1). second, we propose a way to quantify and collect mbk from objective data sources such as bench-mark performance. third, we introduce three strategies-based on superiority, similarity, and a hybrid-to construct a binarization dataset in a zeroannotation, cost-free manner. a major advantage of our approach is that it creates response pairs based on model superiority without any additional human or llm labeling. by classifying models with higher benchmark scores as "positive" and those with lower scores as "negative," we can systematically label preferences throughout the dataset. this significantly reduces the cost of annotation and, because mbk visualizes each model's behavior pattern, increases the interpretability of the preference decisions. through extensive experiments, we show that zebra achieves performance comparable to existing instance-wise labeling methods in the ultrafeedback @xcite dataset-without any extra labeling cost. in summary, our contributions are as follows: ## experimental setup in this paper, we propose the zebra framework. to validate the functionality of this framework and the characteristics of alignment tuning, we conducted experiments to address the following research questions (rqs): ## practical insights and recommendations a key motivation for zebra is its practical usability when researchers or practitioners wish to expand their alignment datasets without additional annotation cost. conventional rlhf or rlaif pipelines require costly human or llm-based supervision, but zebra enables an alternative: leveraging existing benchmark performance tables to generate preference signals at zero cost. practical use case. the procedure for applying zebra in practice can be summarized as follows: ## conclusion we introduced zebra, a zero-annotation framework that replaces costly instance-level preference labeling with model-behavior knowledge from benchmark tables. by leveraging superiority, similarity, or hybrid strategies, zebra constructs largescale preference datasets entirely without annotation cost. experiments across six standardized benchmarks show that zebra achieves performance comparable to, and in some cases exceeding, rlaif baselines, confirming benchmark-derived behavior as an effective proxy for preference supervision. future work will extend zebra to additional behavioral axes (e.g., safety, toxicity, multilingual ability) and further examine its correlation with explicit human judgments. future work will expand to additional behavioral axes (e.g., safety, toxicity, multilinguality), explore adaptive pairing schedules, and assess correlation with human judgments.