Paper: Training Socially Aligned Language Models on Simulated Social Interactions (ICLR-2024)

Reviewer comment (Experiments):
I don’t think the authors have enough evidence to claim that “their approach is considerably more scalable” (from the abstract). I’m a bit concerned that the method does not scale with model size (page 4) and I didn’t see experiments showing scaling wrt society size. If it doesn’t scale, Stable Alignment is significantly less promising as an alignment method. (It’s still interesting as an LLM capabilities analysis though.)

Author response:
As explained on page 6, the scalability of Stable Alignment stems from its ability to decouple data collection from the training process. Unlike RLHF, which requires sequential pipeline operation with at least two large models in memory (see Figure 2 of the Instruct-GPT paper [1], which clearly shows sequential stages), Stable Alignment allows offline, asynchronous data production.

This enables deploying hundreds or thousands of agents in the SandBox to interact in parallel. Our scaling analysis (Figure 3) demonstrates that societies with more capable LMs require fewer interactions to reach Pareto Optimality. Furthermore, we conducted additional experiments with larger LMs like Google PaLM 2 M/L and LLaMA 2, and the results demonstrate Stable Alignment can offer consistent improvement for LMs in different sizes (see results in Table A1).

We hope all above can serve as evidence that Stable Alignment can perform well in scaling-up scenarios (data, model, number of interactions, etc.). Thanks again for your question!

Reviewer comment (Novelty):
I’m not entirely convinced how important is the social aspect of the data-generating process. It seems to be just an extension of [language model cascades](https://arxiv.org/abs/2207.10342) line of work, including [imitational learning from language feedback](https://arxiv.org/abs/2303.16755) (ILF) or [constitutional AI](https://www.anthropic.com/index/constitutional-ai-harmlessness-from-ai-feedback) (CAI), where LMs give feedback on LMs’ initial responses which are then refined. The agents are weakly individuated in the proposed framework, i.e. (as far as I understand) they start the same and only differ in their conversation histories. I wonder how important is treating multiple calls to the same LM as simulating different agents. Does it improve alignment compared with a single-agent society, more similar to ILF or CAI? Is there a scaling plot? (I might’ve missed this, I’m happy to be corrected by the authors.)

Author response:
We explored the necessity of the multi-agent setup by comparing it with a "single-teacher" approach ((i.e., not using the multi-agent setup, but distilling data from only one teacher model). The results show that a single-teacher setup leads to inferior performance (Table A2), lacking the self-improvement achievable through extensive agent interactions in a multi-agent environment. Training with data from a single teacher also resulted in slower and less effective alignment learning compared to our multi-agent approach (Figure 5b).

Reviewer comment (Novelty):
Relatedly, I think [language model cascades](https://arxiv.org/abs/2207.10342) line of work (including [imitational learning from language feedback](https://arxiv.org/abs/2303.16755) (ILF), [constitutional AI](https://www.anthropic.com/index/constitutional-ai-harmlessness-from-ai-feedback) (CAI), [critiques](https://arxiv.org/abs/2206.05802), [Reflexion](https://arxiv.org/abs/2303.11366), [STaR](https://arxiv.org/abs/2203.14465)) could be discussed briefly in Related work. I’m also happy to be corrected if the authors think I’m wrong seeing their paper as a multi-agent extension of this line of work.

Author response:
We acknowledge your recommendation to discuss related works in language model cascades, such as imitational learning from language feedback (ILF), constitutional AI (CAI), critiques, Reflexion, and STaR. We have cited these works in the revised version of our paper and briefly discussed their relevance within the constraints of our page limit.

Reviewer comment (Evaluation):
How is perplexity in Figure 5 measured, i.e. w.r.t. what data?

Author response:
For perplexity calculation, we used the LLaMA 70B model on generated samples (with the [script](https://huggingface.co/docs/transformers/perplexity)).

Reviewer comment (Experiments):
The proposed methodology is complicated compared to existing alternatives, but ablation shows that each component is critical in the effectiveness. I would assume the proposed method generalizes to other (larger) LMs, but additional experimental results would be helpful.

Author response:
**Weakness 1: Experimentation on Larger Language Models**

We appreciate your suggestion regarding the need for experiments on larger LMs. In response, we conducted additional experiments with LLaMA-27B, Google’s PaLM 2 M (Bison), and PaLM 2 L (Unicorn) to examine the scalability of Stable Alignment. The results, which we'll add to the main paper, are in the new Appendix Section A.6 and Table A1. They indicate that Stable Alignment is more beneficial for smaller models, with a greater gain observed in the 7B models compared to the PaLM 2 M/L models. Furthermore, we noted improved performance with more powerful base models (LLaMA 2 outperforming LLaMA 1).

Reviewer comment (Evaluation):
I might overlook this but an analysis of the results on chatgpt will be helpful to identify future improvements.

Author response:
**Weakness 2: Comparative Analysis of ChatGPT and Stable Alignment**

Thank you for suggesting a detailed analysis of ChatGPT versus our Stable Alignment-trained LLaMA model. We revised the analysis section, adding a paragraph focusing on their differences. Despite their size disparity, this comparison has yielded actionable insights for future improvements in LM training methodologies.

Reviewer comment (Experiments):
Experiments can be extended to more human value relevant datasets such as mmlu and civil comments.

Author response:
**Weakness 3: More general-purpose evaluation, such as MMLU?**

We took your suggestion to extend our evaluation to more datasets seriously. In the new Appendix Table A2, we included results from MMLU for our default setting and some ablation settings with the recent LLaMA 2 model. The results show moderate improvement in zero-shot MMLU performance, likely due to the inclusion of societal topic discussions during training, which enriched the knowledge base.

Reviewer comment (Writing):
A few clarifications on Back-Scatter would be helpful.

Author response:
**Question 1: Prompts/Ratings details in SandBox Simulation**

Yes, we have included the exact prompts we used throughout the simulation in the revised Appendix Section A.4.

About ratings, we actually generate both alignment ratings and engagement ratings and filter the training data for imitation by using both ratings (e.g., we only choose to imitate responses whose alignment > 5 and engagement > 5). For self-critic, we teach the model to rate responses only in terms of alignment (so the model needs to generate the alignment rating + rationale), because the revisions used in the third step Realignment are only based on improvement in alignment.

The points of ratings in Figure 3 are actually the average of each round (each round means you finish traversing all 100 agents). The observer agents in these scenarios are memory-less LMs like GPT-4. If you are interested in more details, please refer to our submitted code and data files. Thanks for the great question!

Reviewer comment (Reproducibility):
It’s not clear how the observer LLMs generate the ratings using the Likert scale; the relevant prompts would be helpful given these ratings play a key part in the training component.

Author response:
**Question 1: Prompts/Ratings details in SandBox Simulation**

Yes, we have included the exact prompts we used throughout the simulation in the revised Appendix Section A.4.

About ratings, we actually generate both alignment ratings and engagement ratings and filter the training data for imitation by using both ratings (e.g., we only choose to imitate responses whose alignment > 5 and engagement > 5). For self-critic, we teach the model to rate responses only in terms of alignment (so the model needs to generate the alignment rating + rationale), because the revisions used in the third step Realignment are only based on improvement in alignment.

The points of ratings in Figure 3 are actually the average of each round (each round means you finish traversing all 100 agents). The observer agents in these scenarios are memory-less LMs like GPT-4. If you are interested in more details, please refer to our submitted code and data files. Thanks for the great question!

Reviewer comment (Evaluation):
Is the rating on imitation and self-critic data only alignment ratings.

Author response:
**Question 1: Prompts/Ratings details in SandBox Simulation**

Yes, we have included the exact prompts we used throughout the simulation in the revised Appendix Section A.4.

About ratings, we actually generate both alignment ratings and engagement ratings and filter the training data for imitation by using both ratings (e.g., we only choose to imitate responses whose alignment > 5 and engagement > 5). For self-critic, we teach the model to rate responses only in terms of alignment (so the model needs to generate the alignment rating + rationale), because the revisions used in the third step Realignment are only based on improvement in alignment.

The points of ratings in Figure 3 are actually the average of each round (each round means you finish traversing all 100 agents). The observer agents in these scenarios are memory-less LMs like GPT-4. If you are interested in more details, please refer to our submitted code and data files. Thanks for the great question!

Reviewer comment (Presentation):
In Fig 3, are these the central agents? What is serving as the observer agent here?

Author response:
**Question 1: Prompts/Ratings details in SandBox Simulation**

Yes, we have included the exact prompts we used throughout the simulation in the revised Appendix Section A.4.

About ratings, we actually generate both alignment ratings and engagement ratings and filter the training data for imitation by using both ratings (e.g., we only choose to imitate responses whose alignment > 5 and engagement > 5). For self-critic, we teach the model to rate responses only in terms of alignment (so the model needs to generate the alignment rating + rationale), because the revisions used in the third step Realignment are only based on improvement in alignment.

The points of ratings in Figure 3 are actually the average of each round (each round means you finish traversing all 100 agents). The observer agents in these scenarios are memory-less LMs like GPT-4. If you are interested in more details, please refer to our submitted code and data files. Thanks for the great question!

Reviewer comment (Experiments):
Is instruction-tuning a prerequisite for alignment training with Stable Alignment?

Author response:
**Question 2: Is instruction tuning a prerequisite for Stable Alignment?**

Addressing your question about instruction-tuning as a prerequisite, we performed additional ablations on models with and without instruction tuning. Results in the newly added Table A1 indicate that training on base LLaMA models yields comparable results to those trained on instruction-tuned models. This suggests that the interaction data itself is a rich source of instruction-focused training, especially in alignment contexts. Our experiments with various other LMs also support the generalizability of Stable Alignment.

And we believe our new experiments on other LMs (e.g., LLaMA-2, and PaLM 2 M/L) can already answer your questions about LM generalizations of Stable Alignment.

Reviewer comment (Experiments):
Could a non-instruction tuned model be trained on the alignment data and achieve comparable results (i.e. base LLaMA instead of Alpaca)?

Author response:
**Question 2: Is instruction tuning a prerequisite for Stable Alignment?**

Addressing your question about instruction-tuning as a prerequisite, we performed additional ablations on models with and without instruction tuning. Results in the newly added Table A1 indicate that training on base LLaMA models yields comparable results to those trained on instruction-tuned models. This suggests that the interaction data itself is a rich source of instruction-focused training, especially in alignment contexts. Our experiments with various other LMs also support the generalizability of Stable Alignment.

And we believe our new experiments on other LMs (e.g., LLaMA-2, and PaLM 2 M/L) can already answer your questions about LM generalizations of Stable Alignment.

Reviewer comment (Experiments):
Do these experimental results on alignment tasks generalize to other (larger) base models?

Author response:
**Weakness 1: Experimentation on Larger Language Models**

We appreciate your suggestion regarding the need for experiments on larger LMs. In response, we conducted additional experiments with LLaMA-27B, Google’s PaLM 2 M (Bison), and PaLM 2 L (Unicorn) to examine the scalability of Stable Alignment. The results, which we'll add to the main paper, are in the new Appendix Section A.6 and Table A1. They indicate that Stable Alignment is more beneficial for smaller models, with a greater gain observed in the 7B models compared to the PaLM 2 M/L models. Furthermore, we noted improved performance with more powerful base models (LLaMA 2 outperforming LLaMA 1).

Reviewer comment (Writing):
Naming: ChatGPT->GPT 3.5

Author response:
Finally, we have corrected the typos you highlighted. We hope these revisions satisfactorily address your questions and concerns, and please let us know if anything else we can revise to make the work stronger!

Reviewer comment (Writing):
Related Work (Social Simulation) sentence 5: SandBox -> SANDBOX

Author response:
Finally, we have corrected the typos you highlighted. We hope these revisions satisfactorily address your questions and concerns, and please let us know if anything else we can revise to make the work stronger!

Reviewer comment (Presentation):
Fig 2 (upper left): Back Scatter -> Back-Scatter

Author response:
Finally, we have corrected the typos you highlighted. We hope these revisions satisfactorily address your questions and concerns, and please let us know if anything else we can revise to make the work stronger!

Reviewer comment (Evaluation):
The authors claim the proposed Stable Alignment addresses limitations from (e.g.) RL from a reward model, which "may be inherently imperfect and not fully capture nuances of human judgment". However, since the data collection process (i.e., the sandbox) mainly prompts LLMs to gather feedback and ratings, how are these not imperfect or can be guaranteed to fully capture nuances of human judgement? There is a lack of analysis or discussion on the noises that could come from gathering data from LLMs, which is arguably critical for the proposed method to succeed and can be seen as limitation of how far this approach can go.

Author response:
**Weakness 1: Issues of reward-based optimization for alignment?**

Thanks for the question. This sentence is a summation of the findings from recent studies (e.g., Chain-of-Hindsight, DPO, etc. all replace reward modeling with “verbal reward” or “directly learning from text”), which claim the reward-from might not be fine-grained enough to capture diverse and complicated human preferences during social interactions (i.e., our words “fully capture nuances of human judgement”). Also, there are some discussions on the intrinsic limitations of using a proxy model to delegate human judgment, such as reward gaming/hacking, and vulnerability under adversarial attacks (see the citations in the Introduction Section). These observations motivate us to propose doing rehearsal (i.e., thorough simulation of social interactions) in a separate space (i.e., the SandBox) to expose the issues, and let the LM learn from the corrections (i.e., the three stages of training in Stable Alignment).

In the newly added Appendix Section A.6 and A.7, we conducted additional scaling analysis and ablation studies to understand how much we can obtain and how far we can go with Stable Alignment (results are mainly shown in the Table A1 and A2). In general, we find that Stable Alignment can bring more gains to smaller base LMs, and the gain is not sensitive to the type of the agent LM (RLHF’ed or not). In the revised Ethics Section, we properly warn the readers of some limitations and applicable conditions of our method. We have also included data, code, and exact prompts we used during data collection and training, to help follow-up works reproduce our results.

Reviewer comment (Experiments):
Why is TRLX not included in Figure 4?

Author response:
**Question 2: TRLX not included in Figure 4?**

Thanks for the question! We have now included TRLX results in Figure 4.

Reviewer comment (Experiments):
The motivation of sandbox being a "10 x 10" grid is unclear. Since the interaction mechanism is getting feedback and ratings from some other LLM agents, just sampling those LLMs from some unordered pool should suffice? It is unclear how the additional structure of a grid world is necessary in the proposed sandbox environment.

Author response:
**Question 3: Why grid world?**

Thanks for the question! The motivation for the grid-world design is two-fold: 1) Grid world is actually a classic design in alignment research (see an example in [AI Safety Gridworlds](https://arxiv.org/pdf/1711.09883.pdf), published in 2017; and a blog ["Specifying AI safety problems in simple environments"](https://deepmind.google/discover/blog/specifying-ai-safety-problems-in-simple-environments/) from Google DeepMind), and its simplicity can avoid incorporating potential bias from geographic design; 2) It offers ease for us to calculate the distance between agents so that we control the number of activated agents around one central agent to be roughly a constant. In our code, we actually use a radius = 3 (≈ a 3x3 grid) to scan the nearby agents and activate some of them with a preset dropout rate. Please take a look at our code if you are interested in the actual implementation.

In general, the design of the grid world is to obtain more generalization, which follows a protocol in the prior arts, and it also provides some practical benefits in calculation. Thanks again for your great question!

Reviewer comment (Experiments):
The data collection process of using LLM to gather feedback and asking the central model to generate a revision may be error-prone as indicated by [2] and [4], as a smaller LMs such as LLaMA-7b or Alpaca was not trained initially to follow the feedback and revise its response. Has this been an issue in the proposed approach as well? If so how is this resolved?

Author response:
**Question 4: Simulation with small models is error-prone?**

Not sure whether I understand this correctly but [2] and [4] are both using LLaMA-7B as the base model agent/multi-agent, while our setting is using much larger models (GPT-2/3.5/4) as the professional experience provider. In our preliminary experiments we find initializing a grid world with 10x10 LLaMA-7B models cannot generate proper response, as these pre-trained models cannot follow instructions well and the relatively small size limits the diversity of their generation severely. That's why we switched to our current setting later (which is learning from the interaction between many more capable models).

Reviewer comment (Theory):
However, this method often yields models susceptible to adversarial attacks, like “jailbreaking prompting” (Subhash, 2023; Xu et al., 2021), due to limited exposure to misaligned data during training (Amodei et al., 2016). To address this, a more advanced technique, “reward modeling” has been proposed (Leike et al., 2018; Christiano et al., 2017). Subhash, 2023 uses ChatGPT, a model that has likely been trained with a reward model, as an example of a vulnerable model and appears to contradict the second sentence stating that reward modeling addresses the vulnerabilities of SFT-trained models.

Author response:
**Weakness 1: More accurate citations and better logic**

Thanks for carefully reading our paper and for the great suggestion! We have fixed the citation and added some new related works in the second paragraph of the Introduction, with some edits on the text to make sure the logic flow is consistent and accurate. Please take a look at the revised version. Thanks again for your suggestion!

Reviewer comment (Novelty):
The claim of simulating human society with no supporting references from the social science literature is way too grandiose. The authors should improve the accuracy of their claim.

Author response:
**Weakness 2: Social science support for simulating human society with LMs**

Thanks for the suggestion! Actually, in our original Related Work section, we have already included several milestone works from social science, HCI, and AI, that can serve as support for LMs simulated human society (e.g., the PNAS work [1], using LM simulation to replicate social science studies [2], confirming the fidelity of LMs simulated human society [3], etc.). Following your suggestion, we now have revised the first paragraph of Related Work to better summarize these works so that the readers can better notice and understand the social science background of our work.

Reviewer comment (Experiments):
It is somewhat unclear how the OpenAI models fit into the picture; are 99 of the 100 models all simply one of the 3 OpenAI models and one of them the Stable Alignment model? If so, this seems like a form of distillation of the preferences/values from the OpenAI model, which was purportedly trained with SFT and RLHF. Given that this method claims to do away with the need for RLHF/RMs, experiments showing its efficacy should not rely on models trained with such methods. In other words, the current experiments do not convince me that Stable Alignment can replace RLHF on its own. For example, I would be much more convinced if all the agents were initialized from non-RM based models. Relatedly, while convenient, using the OpenAI API models for critical parts of the experiments makes them essentially non-replicable and rather non-reproducible, given the constantly changing model in the backend.

Author response:
**Weakness 3: “I will be more convinced if all the agents were initialized from non-RM based models”**

Thanks for the question! Yes, as described in Section 4, the LMs used for simulation not only include RLHF’ed (or RM-based) models (e.g., text-davinci-003 and GPT-4), but also include non-RM-based models (e.g., text-davinci-002). More information about these models was detailed in [the statement from OpenAI](https://platform.openai.com/docs/model-index-for-researchers). 

In the newly added Appendix A.6 and Table A1, we have included results when SandBox is initialized with non-RM-based models (i.e., we only use the text-davinci-002 to run the simulation to obtain interaction data). In general, the improvement over the baseline is still significant, partially because the data from text-davinci-002 actually takes up a main portion of the final mixture of the interaction data (nearly 50%, and 72% in the realignment data). As text-davinci-002 is not a RM optimized model, we believe these results can serve as evidence that the effectiveness of Stable Alignment is relatively model agnostic (as the improved alignment presented in the interaction data can only be attributed to the multi-agent simulation).

Reviewer comment (Experiments):
An additional nitpick is that the method claims to deviate from SFT-like approaches but uses an SFT-trained model as the starting point. I would like to see this addressed as well; what is the performance of Stable Alignment without using an SFT-trained model as the initialization?

Author response:
**Weakness 4: More ablations about SFT**

Thanks for the question! We have compared the performance with/without an SFT stage before Stable Alignment, and the results are in the newly added Appendix Section A.7 and Table A.2. In general, we find there is no significant difference between SFT/no SFT Stable Alignment. The reason could be the learning objective of Stable Alignment already includes the SFT term, so the convergence on Stable Alignment can guarantee the trained model is at least as good as an SFT-only model.

Reviewer comment (Experiments):
Regarding CPO, since it is a new technique that claims to improve the efficacy of the overall method, there should be an ablation where it is replaced with the regular SFT loss or other alignment algorithms, such as perhaps DPO.

Author response:
**Weakness 5: Comparison with DPO?**

Thanks for the question! In the original manuscript, we have already compared Stable Alignment with DPO. Please take a look at Figure 4, Table 2, and the corresponding analysis in the paper.

Reviewer comment (Experiments):
More details on the procedure and exactly which models and types of social agents were used to generate data for each learning stage are needed; this is still unclear to me after reading through the details and appendices multiple times. Are the numbers of examples for each stage in Fig 2 the total over all iterations, or per iteration? How many iterations were needed to arrive at the final model? Was the model trained on data from all three societies at each stage or were 3 separate models trained? If the latter, which model was used in the final evaluation?

Author response:
**Weakness 6: More details on SandBox and data**

Thanks for the question! About the model for the social agents, as we described in Section 4, we used text-davinci-002, text-davinci-003, and GPT-4 to run the simulation. For the number of iterations, it took 5, 2, and 1 round(s) for the above three models to reach Pareto Optimality in terms of both alignment and engagement. The number of data shown in Figure 2 presents the total number of data samples we collected from the simulation with the three models. For the three training stages, the major difference is in how we construct the data from the interaction trajectories (as shown in Figure 2), and we use a mixture of the data from the three models to construct those samples (though for ablation purposes we can easily select any one of them alone as the only data source). The supplementary material contains the data, code, and handy scripts to reproduce our work.

Reviewer comment (Evaluation):
Use of HH-A as an estimate of adversarial prompt robustness: I went through the examples in the HHH dataset (there are typos where the last H is missing throughout the paper) and disagree with the use of the appended “misaligned” response as a dataset good enough for adversarial robustness evaluation. A better evaluation of adversarial prompt robustness would be to use a library such as [garak](https://github.com/leondz/garak) with known adversarial prompts used by the community.

Author response:
**Weakness 7: HH-A and other adversarial tasks**

This is a great suggestion! We have run additional evaluations with the garak framework, and we picked the RealToxicPrompt as the task that seems to be most related to our goal. The results are shown in the newly added Appendix Section A.7 and Table A.2, which we find are mostly aligned with our other alignment evaluations.

Reviewer comment (Experiments):
Given that the motivation was alignment to social values, how was diversity across demographics ensured in the user study? The user study is meant to demonstrate that models trained with this method are better aligned, but better aligned to whom? This is important to call out since regular practitioners will simply use the popular models of the day with the released toolkit without thinking critically about whose values they capture.

Author response:
**Question 1: Diversity, and demographics of the human evaluators**

We acknowledge the importance of demographic diversity in our user studies. The revised version now includes detailed demographic information (i.e., gender, race, ideology) of the participants in Section A.5 and Figure A.4. We also included additional discussions and warnings in the revised Ethics Section (sorry we don’t have enough space in the main body of the paper), to make potential readers aware of the limitation of our work (e.g., the limited coverage all possible value judgment systems, which seems to be inevitable for any current LM based alignment research).

Reviewer comment (Evaluation):
Additionally, given the use of simulation with LMs is touted as a benefit of this approach, what are the drawbacks of using this instead of collecting representative feedback (and training a reward model), whose values are the models implicitly aligning to with such simulation? These should be explicitly called out and elaborated in the limitations section but are currently briefly glossed over in the ethics section.

Author response:
**Question 2: Why not collect representative feedback to train reward models?**

Thanks for the question! We believe the setting described by the reviewer is actually the typical reward model based method, which seems to have reward gaming/hacking problems (as reported by many related works we cited in the Introduction and Related Work Section). Another challenge is reward model based methods typically require at least two models pinned in the memory to interact with each other, which might be a practical deployment challenge in low-resource scenarios. Additionally, sometimes directly collecting enough representative feedback from a social group is time-consuming and costly, while simulation with proper conditions seems to be able to serve as a quick and low-cost alternative in such cases.

Reviewer comment (Reproducibility):
For the embedding model and social agents, is it necessary to use an API model? Why not use an open-source model, which makes the experiments more reproducible and replicable?

Author response:
**Question 4: Why not use an open-source model running simulation locally?**

The main concern of using a locally serving model is because of the hardware bottleneck --- since we start 100 agents and all of them need to interact (draft + feedback + revise) with each other in several rounds of simulation, the estimated number of calls would prohibitively large for locally serving settings. We also use multithread parallelization to speed up the call --- a vanilla local setting will need a large number of model checkpoints pinned in memory to handle this high throughput. In other words, it's more like practical hardware limitation, and we'd love to switch to open-source models that can be run and served locally at scale (100+ checkpoints?) as it will greatly save money cost of our experiments.