Paper: Everyone Deserves A Reward: Learning Customized Human Preferences (ICLR-2024)

Reviewer comment (Novelty):
Limited Novelty of Baseline Approach: While the introduction of the benchmark dataset is innovative, the baseline approach proposed for addressing this dataset is somewhat derivative, primarily building upon the work by Askell et al. (2021). This is not in itself problematic if we consider the dataset to be the main contribution of the paper, but combined with the synthetic nature of the dataset it limits the significance of the contribution.

Author response:
1. Novelty of Approach: we agree that the imitation learning method we used in GRFT and CRFT steps is similar to the PMP method in Askell et al.(2021). However, we use this imitation learning loss for a different purpose: PMP uses imitation learning to improve the performance of a general RM; we use imitation learning to preserve the general RM performance when learning a customized RM. Our main contribution to the methodology part is we find this imitation learning loss can enhance a new task CRFT, which is practically valued for many LLM customization scenarios.

Reviewer comment (Experiments):
Synthetic Nature of Dataset: The use of a synthetic dataset, generated by a language model, raises concerns about its practical utility. The dataset might not adequately represent real-world complexities of individual preferences, as it is model-generated. In particular, the fact that a language model trained on general human preferences was able to generate these personas calls the need for customized reward models into question for this particular dataset. Addressing this limitation by incorporating or comparing with real human-generated data would significantly strengthen the paper.

Author response:
2. Synthetic Nature of Dataset: We agree that the dataset we release is somehow a synthetic dataset, at least not annotated by humans. We will extend this dataset with more complicated preferences and more large size. However, the main point of the DSP dataset is to quickly build scenarios for customized RM training and to quickly verify the effectiveness of DSP baseline approaches.

Reviewer comment (Experiments):
Lack of Baselines and Alternatives: Since (as mentioned in the previous point) prompted language models already show significant capability of customizing their outputs to align with the preferences described in their prompt, comparison of fine-tuned reward models to a prompted language model would further strengthen the contribution.

Author response:
3. Lack of Baselines: The reviewer provides a good suggestion to use the prompted LLM for evaluation as a baseline. We will add this baseline in the future revision. However, the performance of prompted LLM can be with high variance for the variety in prompt selection.

Reviewer comment (Evaluation):
Lack of Policy-Level Evaluation: The paper focuses exclusively on reward-model level evaluation. Including policy-level evaluations, such as example outputs of language models fine-tuned with the customized reward models, would provide a more comprehensive understanding of the practical applications and effectiveness of the approach.

Author response:
4. Lack of Policy-Level Evaluation: Although we pay most of our attention to the RM learning task, we acknowledge that adding policy-level evaluation can further verify the effectiveness of the three-stage RM training scheme. However, evaluating domain-specific LLM can be far more challenging: we need annotators to act as a person from a particular domain, which requires additional annotation resources.

Reviewer comment (Novelty):
Insufficient Discussion on Personalization: While the background on language-model specific alignment techniques is quite solid, the paper does not sufficiently explore prior work in personalization within the realms of general preference learning and information retrieval. Expanding the discussion to include these fields could provide a richer context and potentially lead to new directions to explore.

Author response:
5. Discussion on Personalization: We will add more personalization-related introductions in the future review. Thanks for the suggestion.

Reviewer comment (Presentation):
Structural Improvements: The current structure of the paper could be optimized for better readability and impact. The paper currently puts large emphasis on the evaluation of the imitation learning generalization, although this is not the main contribution in my view. It could be improved by presenting the most promising results (Figure 11) first and only then discussing the ablations.

Author response:
1. Structural Improvements: we will reorder the experiments and highlight our discovery of imitation learning in the next revision.

Reviewer comment (Writing):
Clarity in Abstract and Terminology: The abstract could be a bit clearer. On first read, I did not understand what you mean by "data strategies", and "training strategies" is similarly vague. I suggest to be more explicit, i.e., explain that you test your method with varying combinations of datasets in the intermediate fine-tuning stage and that you evaluate the impact of "imitation regularization". The abstract could additionally explain the relation between individuals and domains better. As it is, it is slightly confusing that you first call for the need for individualized reward models but then propose a domain-specific (rather than individual-specific) dataset.

Author response:
2. Clarity in Abstract and Terminology: We will make the abstract more clear. Our initial logic is: both domain-specific and individual-specific preferences are customized preferences. However individual-specific preferences are difficult to collect so we collect domain-specific preferences instead.