Diverse AI Feedback For Large Language Model Alignment (2025)

Abstract: Recent advances in large language models (LLMs) focus on aligning models with human values to minimize harmful content. However, existing methods often rely on a single type of feedback, such as preferences, annotated labels, or critiques, which can lead to overfitting and suboptimal performance. In this paper, we propose Diverse AIFeedback (DAIF), a novel approach that integrates three types of feedback—critique, refinement, and preference—tailored to tasks of varying uncertainty levels. Through an analysis of information gain, we show that critique feedback is most effective for low-uncertainty tasks, refinement feedback for medium-uncertainty tasks, and preference feedback for high-uncertainty tasks. Training with this diversified feedback reduces overfitting and improves alignment. Experimental results across three tasks—question answering, dialog generation, and text summarization–demonstrate that DAIF outperforms traditional methods relying on a single feedback type.1

## introduction in recent years, large language models (llms) have demonstrated significant capabilities in addressing a wide range of information needs @xcite @xcite @xcite . a growing area of research in this field focuses on aligning llms with human values to reduce the risk of generating combining different types of feedback to improve agent policies. however, a notable gap in the literature exists regarding the application of these findings to the alignment of llms. moreover, existing studies have yet to investigate the most suitable feedback type for each training sample, which could maximize training data efficiency. in response to these gaps, we introduce diverse ai feedback (daif), a novel approach that optimizes llm alignment by integrating three types of feedback: critique (critic), refinement (refine), and preference (prefer). daif tailors these feedback types to tasks of varying uncertainty to enhance alignment with human values. starting with an unaligned base model, we generate outputs for a predefined set of problems. using the concept of information gain from active learning @xcite , we evaluate the information gain for each feedback type across different uncertainty levels of problems, with perplexity, a property that has been widely recognized in studies addressing uncertainty in language generation tasks @xcite , serves as a task-agnostic metric for assessing uncertainty. based on this analysis, we classify problems into three uncertainty categories: ''low'', ''medium'', and ''high''. for ''low'' tasks, critique feedback, consisting of natural language critiques, is utilized; for ''medium'' tasks, refinement feedback, involving improvements to the model outputs, is employed; and for the most challenging ''high'' tasks, preference feedback, drawn from annotated preferences across multiple outputs generated by the same model, is sought. the primary contributions of our work are as follows: ## preliminary in this section, we first introduce the dataset used in our experiments, along with the various feedback types employed. we then present the information gain methodology utilized to determine the optimal feedback type for each problem. subsequently, we conduct a preliminary study to evaluate the information gain associated with critique, refinement, and preference feedback types. based on the results of this analysis, we propose an effective strategy for combining these diverse feedback mechanisms. ## method in this section, we provide a detailed explanation of our proposed daif method, which is depicted in figure 1 . ## analysis 6.1 over-optimization in rm-ppo training previous studies have acknowledged overoptimization as a common issue in rm-ppo training @xcite . we postulate that daif's superior performance, compared to the methods relying on singular feedback types, primarily stems from its capacity to alleviate over-optimization. to evaluate this hypothesis, we conduct additional experiments specifically targeting the aspects of the rm-ppo training process, namely, the reward model and rm-ppo training. analysis of the reward model. for the reward model, we assess the precision of models trained with daif and individual feedback types on a separate test set. this test dataset, comprising 2,700 comparison answer pairs derived from 300 uniquely sampled problems across each task, was structured to assess feedback variations-critique, refinement, preference-on problem answers. as illustrated in ## conclusion in this paper, we introduced daif, an innovative data collection methodology designed to improve the alignment of llms with human values. we proposed a differentiated feedback approach, where various types of feedback are utilized based on the varying uncertainty levels of the problems. we presented experimental results across three downstream tasks, demonstrating that daif achieved superior performance even with a smaller dataset. further experiments and analyses provided additional evidence of the effectiveness of daif. we believe that our method opens up new possibilities for the effective utilization of feedback in aligning llms.