Paper: Group Preference Optimization: Few-Shot Alignment of Large Language Models (ICLR-2024)

Reviewer comment (Presentation):
Lack of clarity in the presentation of the method. For instance, it remains ambiguous which specific parameters are subject to training. Incorporating a detailed algorithm diagram or flowchart would greatly enhance the comprehensibility. Besides, more training details would be better.

Author response:
A1: Thanks for pointing out the ambiguity. We added an algorithm box in Appendix A of the revision to illustrate how GPO is trained. GPO trains a transformer module for in-context predicting unseen group preferences based on the few-shot examples provided. And we do not train the base LLM weights. Moreover, the training details and hyperparameters are given in the appendix section G. We hope this clarifies your question. If you have more questions, please let us know.

Reviewer comment (Evaluation):
The absence of evaluation results concerning the generalization capability of the GPO method. As GPO optimizes the parameters, it's crucial to show whether the model retains performance on general benchmarks after tuning.

Author response:
A2: GPO does not alter the base LLM's parameters. Instead, it augments the LLM with an additional module trained to predict group preferences based on the few-shot examples provided. This approach allows the LLM to retain its broad language capabilities while aligning its outputs to specific group preferences​​ in a few-shot manner.

Reviewer comment (Reproducibility):
As mentioned, it is ambiguous which part parameters are tunable. Could the authors further clarify the training details?

Author response:
A1: Thanks for pointing out the ambiguity. We added an algorithm box in Appendix A of the revision to illustrate how GPO is trained. GPO trains a transformer module for in-context predicting unseen group preferences based on the few-shot examples provided. And we do not train the base LLM weights. Moreover, the training details and hyperparameters are given in the appendix section G. We hope this clarifies your question. If you have more questions, please let us know.

Reviewer comment (Experiments):
Could the authors provide examples of generation tasks (QA) to better showcase the model's preference? Demonstrating the model's performance on responses could provide a clearer insight into its practical applications.

Author response:
A3: We agree such datasets could provide valuable insights into GPO's capabilities. However,
* We want to highlight that GPO is not inherently restricted to multiple-choice datasets. For open-ended long-form QA, GPO could predict raw preference scores and guide the LLM to align with group distributions, without requiring the scores to sum to 1. 

* To the best of our knowledge, we are unaware of suitable labeled long-form QA group preference datasets for real world settings which are diverse, nuanced, and seperated for diverse demographic groups that would allow benchmarking. 

* Collecting these dataset would be time consuming and is outside of this work’s scope. Additionally, evaluating the long-form generations would require human judgement which is expensive in our academic research setting. For example, gathering diverse demographic groups labellers to evaluate generated responses and aggregate their reponses as group preference distributions would be impractical for academic research.
* That said, we have made our best efforts to utilize the most comprehensive opinion datasets available for our current evaluations on multiple-choice QA. We chose OpinionQA and GlobalOpinionQA, which is of high community interest given their growing adoption, with 60+ GitHub stars recently for the former and 300+ monthly downloads of the latter. Additionally, their survey QA format enables efficient, accurate analysis without costly human evaluation.

If you have suggestions on such long-form QA datasets, please let us know. We would not hesitate to test GPO on them.

Reviewer comment (Experiments):
I am not sure about the significance and broadness of the problem this work tries to solve. Alignment of LLMs to group preferences sounds important, however, the evaluation datasets used by this work look quite specific and narrow and I don't think it is of interest to a broad range of research community.

Author response:
A1:Thanks for acknowledging the importance of steering LLMs to group preferences. We think it is an important task as current popular alignment methods such as prompting, in-context steering, and SFT do not work well for it. 

Regarding the evaluation datasets:
* We have exerted our best effort to utilize the most comprehensive and relevant opinion datasets currently available for assessing LLM opinion distributions. Additionally, there are currently no long-form QA datasets with labeled ratings collected for diverse demographics groups. We chose OpinionQA and Anthropic's GlobalOpinionQA for their growing use and recognition in the field. For instance, OpinionQA has garnered over 60 stars on github in about half a year. Similarly, GlobalOpinionQA has seen over 300 downloads in the past month, indicating high community interest in LLM opinion distribution studies.

* Our selection was also larged guided by the practical constraints of academic research, where datasets that allow for easy verification and no costly human evaluation are preferred. This motivation aligns with OpinionQA (as stated in [1] section 2.1) and GlobalOpinionQA’s motivations too, which states that the use of survey QA data enables capturing diverse and nuanced public interest topics and also enables efficient and accurate evaluation.

* Additionally, in contrast to prior work that treats human alignment as a single quantity such as safety or helpfulness, our work conceptualizes human alignment as an inherently subjective measure that depends on who it is evaluated against; to the best of our knowledge, **we are among the first to align LLMs to a group preference distribution rather than a single preference or group consensus.**

[1] Whose Opinions Do Language Models Reflect?  https://arxiv.org/abs/2303.17548

Reviewer comment (Novelty):
The proposed method look quite trivial and standard, which is a in-context fine-tuning method. There are some changes in the method details but those details are a bit hard to understand, which I will pose some questions on next.

Author response:
A2: 
* We would like to clarify that GPO is not an in-context fine-tuning method, it does not alter the base LLM's parameters. Instead, it augments the LLM with an additional transformer module trained to predict unseen group preferences based on the few-shot examples provided. Although GPO does not modifying the weights of the LLM itself, it can guide the LLM to produce sample outputs that are attuned to the desired group preferences. This allows the LLM to retain its broad language capabilities while aligning its outputs to specific group preferences​​ in a few-shot manner. 
* In section 3 (Page 6), the in-context finetuning method is actually one of our baselines that performs worse than GPO.

Reviewer comment (Experiments):
In the second last paragraph of page 4, it is said that "In particular, we discard the positional encodings commonly found in standard transformer architectures". Wouldn't the removal of positional encodings significantly deteriorate the performance of those pre-trained and fine-tuned LLMs?

Author response:
A3: We discard positional encoding for GPO but not the base LLM. We do this to achieve the property of context invariant, which means the prediction of preference for new questions should not depend on the ordering of the context questions. In the appendix Table 3 we submitted, we compared GPO with a standard autoregressive transformer that employs standard causal mask and positional encodings. And GPO’s inherent inductive biases yield superior alignment performance compared to a traditional transformer.

Reviewer comment (Reproducibility):
Still in the second last paragraph of page 4, it is said that "we employ a masking strategy where the context pairs can self-attend to each other". Could you elaborate such a masking strategy?

Author response:
A4: Thank you for pointing out the potential confusion in our writing. We will provide more details on this in the revision. The masking strategy is as follows: 

Our input to the transformer is a sequence of context pairs followed by padded target pairs as in figure 2. The mask allows:
1) The context pairs attend to each other, rather than using a causal mask that restricts each token to only attend to previous tokens. We use this because the order of the context pairs should not influence predictions for target points - it should be permutation invariant.
2) The padded target pairs to only attend to the context pairs, and not to other targets. This restriction aligns with the conditional independence assumption in Equation 2, ensuring each target pair is predicted solely based on the context.

Reviewer comment (Experiments):
For the baseline, the authors seem to fail to include one direct method — a simple Transformer model that purely learns from the (q, y) pair and uses them in a semi-supervised fashion as a sequence of inputs. As is pointed out by the author, the Reward Model baseline is underperforming a lot of the other baselines, and it would really make sense to add a comparable-sized baseline as in the author’s proposed method to rule out the possibility that overfitting is the only cause of inferior baselines without really relying on LLM.

Author response:
A2: Thank you for highlighting the potential oversight in our baselines. We'd like to address this in two parts based on our understanding of your suggestion:
* Based on our problem setup, we think the q you referred to is actually x (the concatenation of q and r), because the preference y is associated with x rather than q alone. Then the concept of inputting the (x, y) pairs directly to a transformer model closely aligns with our 'In-Context Finetune' baseline. In this baseline, n context pairs (x,y) are indeed fed sequentially as inputs to the LLM to predict the preference y, and this method performs worse than GPO in our experiments. However, we are not using GPO nor the baseline in a semi-supervised fashion but rather train them on labeled (x, y) pairs.
* Secondly, we have also included a baseline that aligns with your suggestion of a simple transformer model in appendix Table 3 we submitted. We compared GPO with an equally-sized standard autoregressive transformer that employs standard causal mask and positional encodings and also takes in (x,y) pairs as sequence inputs too. In contrast, GPO has a different masking strategy and removed positional encoding to adopt inductive biases that lead to two properties. One is context permutation invariance, where the prediction of queries does not depend on the ordering of the context pairs; and the other property is target equivariance, which requires that whenever we permute the target inputs, the predictions are permuted accordingly. These inductive biases are important for our problem setup. And GPO’s inherent inductive biases yield superior alignment performance compared to a traditional transformer as shown in the table below. 

If our interpretations do not fully address your baseline suggestion, please give us further clarification and we will provide additional baseline experiments.

Reviewer comment (Reproducibility):
The PeftConfig configuration seems not consistent with the description of the Reward Model baseline in the paper. The authors argue to use a linear MLP head for the Reward Model baseline. However, in the code, the authors used a LoraConfig which should by default fine-tune every layer of LLM, and this might be the major reason for the underperformed score of this baseline. Also, it makes more sense to use cross-entropy for the loss function instead of MSE as the final output is a distribution.

Author response:
A4: Our reward models are initialized using the HuggingFace AutoModelForSequenceClassification which adds an additional linear layer to the LLM. When this model is converted to Peft model, the Peft library ensures that only the LoRA adapters and the additional linear layer are trained since our task_type is set to SEQ_CLS in our PeftConfig and we are using an AutoModelForSequenceClassification model. In the description of reward model baseline in section 3, when we state “we train a per-group reward model by adding a linear MLP head on a base LLM and train it on m context samples”, “it” refers to all of the trainable parameters of the entire Peft model. We have updated the reward model baseline description to more clearly reflect the reward model training setup and apologize for the confusion.

Related works including InstructGPT[1] typically train a reward model by fine-tuning all parameters of a reward model(including the base LLM). Considering that we are training a very small subset of the base LLM’s parameters with LoRA (not the full 7B), compared to prior works which train all 6B+ parameters for reward model baselines, it is unlikely that our reward models are overfitting majorly due to large model size. The number of trainable parameters for the reward linear layer, our original LoRA training and GPO’s transformer module are shown in the table below. 

| Model    | Trainable parameter count |
|---------------------|---------------------------|
| GPO                 | 1188354                   |
| Linear reward layer | 4096                      |
| LLM + Linear layer, with LoRA           | 6295552                   |

Nevernethess, we conducted an experiment to rule out the overfitting possibility due to trainable parameters, and the results are shown below in the table, which shows no improvement for training only the top linear classification layer. We believe the reward model’s inability still lies in the difficulty of learning a high-quality reward function from a small number of training examples for every group.
|            Reward model setup              | OpinionQA         | GlobalOpinionQA  |
|--------------------------|-------------------|------------------|
| Train only top linear layer | 0.824 ± 0.014      | 0.675 ± 0.028    |
| Orig setup: train both linear and LLM layers with LoRA | 0.831 ± 0.019      | 0.683 ± 0.033    |


[1] InstructGPT: https://arxiv.org/pdf/2203.02155.pdf


A5: We chose to use MSE as the loss function because the reward model takes a single input viewpoint (x) and outputs a scalar target value (y) rather than a probability distribution. However, you raise a good point that cross-entropy loss may better match the probabilistic nature of our tasks. Given your suggestion, we experimented with CE loss on the two datasets as shown in the results table. Empirically, the two losses perform similarly, but CE has the advantage that the loss converges faster than MSE loss.
|          | OpinionQA      | GlobalOpinionQA |
|----------|----------------|-----------------|
| CE loss  | 0.820 ± 0.032  | 0.680 ± 0.029   |
| MSE loss | 0.831 ± 0.019  | 0.683 ± 0.033   |

Reviewer comment (Writing):
The terminology of alignment seems a bit too abused and distracting in this setting. In this work, the authors only tried to learn a separate Transformer architecture that operates upon the output of LLM while the LLM itself does not enjoy any new abilities in its parameters based on the modifications.

Author response:
A3: We use "alignment" to mean aligning the output distributions of the generative model - a combination of LLM and GPO - to group preferences. Although GPO does not modify the weights of the LLM itself, it can guide the LLM to produce sample outputs that are attuned to the desired preferences. 

This terminology is also in line with prior works for alignment such as [1] and [2], where alignment also refers to tuning sample outputs rather than altering LLM parameters. In these works, alignment is achieved using techniques like reward models and Best-of-N sampling over the pretrained LLM, without updating the base model itself. 
Additionally, in contrast to prior work that treats human alignment as a single quantity such as safety or helpfulness, our work conceptualizes human alignment as an inherently subjective measure that depends on who it is evaluated against; to the best of our knowledge, we are among **the first to align LLMs to a group preference distribution rather than a single preference or group consensus.**

[1] Let’s verify step by step. https://arxiv.org/abs/2305.20050 

[2] Scaling Laws for Reward Model Overoptimization  https://arxiv.org/pdf/2210.10760

Reviewer comment (Experiments):
It would be an interesting point in this work to justify the reason for the output of (r) as input in this work (through ablation studies). The reviewer’s current understanding is that the sampled response would be used as an anchor point for the training of the proposed method, however, it would lead to a natural question: without r, will the model underperform a lot, meaning that LLM would also be the important ingredient? Also, in this case, it seems that the qualities of the response from the model would also be quite important, and it would make more sense to replace it with random strings etc. to further make the study more coherent.

Author response:
A6: We think there may be some confusion about the role of the response in our setup. To clarify, the response r is essential in our problem formulation. Our goal is to model preferences y over query-response pairs (q, r), where r provides the crucial context to ground each opinion. Without r, the score y would be assigned to the question q alone. Thus, r is critical for accurately reflecting the nuanced "opinion" we aim to infer group preferences over.

However, we appreciate your suggestion of an interesting ablation and have conducted this experiment by giving a random string response to every query. As shown in the table below, the performance degraded substantially without the response. 

| GPO setup  | OpinionQA          |
|---------------------------|--------------------|
| Random string response    | 0.7909 ± 0.009  |
| Actual response           | 0.9201 ± 0.0026    |