Paper: SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents (ICLR-2024)

Reviewer comment (Evaluation):
The paper may not clearly state the result of the study of GPT-4's ability to evaluate interactions. The results seem to indicate that GPT-4 is rather good at evaluating the goal dimensions and two others when it comes to the behavior of other models, but the paper then states: “Putting these observations together, we conclude that, with some caution, GPT-4 can be used as a proxy to human judgments for evaluating model performance on most dimensions and for human performance on the GOAL dimension.” This raises the question: is 3/7 'most'?

Author response:
### GPT4's ability to evaluate social interactions
The results are somewhat mixed, GPT-4 is good in some dimensions (esp. The goal). And apologies, yes, “most” is not accurate; we changed it to “some” in our updated version. Also, note that the correlation (Table 1) between GPT-4 and human judgment belongs to instance-level correlation while Figure 2 shows the system-level correlation which indicates GPT-4 still provides an evaluation that can be cautiously applied and offers insights into models’ overall social intelligence (this is further demonstrated in the appendix Table F.1, where human evaluation shows the same trend as GPT-4 when comparing models’ performance.)

Reviewer comment (Reproducibility):
Since this is a benchmark paper, it should explicitly state what will be released publicly. Ideally, the release should include: 1) the codebase to run these experiments, 2) an easy-to-use way to plug in new language models, define other models they could interact with, and obtain the final automatic evaluation score, 3) interaction data across models and with humans, 4) human judgments, and 5) GPT-4 judgments — basically all code and data. The paper should be explicit about what will or will not be released, as the quality of this work is directly tied to how much is released for others to use. Benchmark papers that do not release anything are largely useless.

Author response:
### Comprehensive Release Plan
Yes, we have all of these ready to be released upon acceptance:
1) the codebase to run these experiments.
2) an easy-to-use way to plug new language models, and define other models it could interact with to get the final (automatic) evaluation score. Our API supports both locally hosted and 3rd party models. 
3) interaction data across models and with humans.
4) human judgments.
5) GPT4 judgments.

Reviewer comment (Presentation):
The discussion of limitations should be included in the main paper and not relegated to the appendix.

Author response:
### Bringing Limitations to Main Paper
We agree that it would be best to move the discussion of limitations from the appendix to the main paper. We will try to fit it into the space constraints.

Reviewer comment (Writing):
There are a couple of typos throughout the paper.

Author response:
### Clarifications on Typos and Cost Estimates
Thanks for pointing out the typos, we will fix them. Running evaluation over one episode would on average have 2k input tokens and 500 output tokens. As of the current GPT-4's price (https://openai.com/pricing), it would cost around 0.09 to evaluate one episode, therefore, it would cost around 0.09*900=\$81 to use GPT-4 to evaluate a given agent.

Reviewer comment (Experiments):
While the paper mentions the limitations of existing benchmarks for social intelligence, it does not provide a thorough comparison or analysis of how SOTOPIA addresses these limitations. A more detailed discussion and comparison with existing benchmarks would strengthen the paper's argument. Especially, this work does not implement studies about evaluation benchmarks related to other existing tasks such as SocialIQA, Personna Chat, Multi-dialog, social relation QA, e.t.c, showing few connections between our social interactive tasks.

Author response:
Thanks for pointing out that SOTOPIA offers a wide range of realistic social tasks and adds a valuable perspective and allows for a comparison between AI models and human performance. We appreciate your questions that indeed highlight crucial aspects of our work, and we are happy to address them. 

### How SOTOPIA addresses the limitations of the previous benchmarks for social intelligence and how they relate:

1. The current benchmarks, such as SocialIQA, primarily consist of descriptive interactions paired with question-and-answer (QA) sets. In these cases, the model doesn't engage in real-time interactions. In contrast, Sotopia introduces a dynamic approach. It evaluates Large Language Models (LLMs) within a first-person interactive framework, allowing for direct engagement. This marks a significant shift from the static nature of existing benchmarks to a more dynamic and interactive evaluation environment in Sotopia.
2. With the rapid development of LLMs, some of the benchmarks gradually become saturated. While there are new adversarial benchmarks to evaluate social intelligence, which are harder than their predecessors, they still lack the dynamic nature of social interactions and the rich social context, which is deemed insufficient for evaluating social intelligence in AI systems [1].
3. Our work is indeed inspired by several benchmarks and datasets in social intelligence, such as SocialIQA, Persona Chat, and others. For instance, many of our scenarios come from SocialIQA, and our character design is inspired by Persona Chat (see more details in Appendix B.2 Scenarios)
4. While our current focus is on evaluation rather than the improvement of agents’ social intelligence, we acknowledge the importance of the latter. We have ongoing work using SOTOPIA to enhance social intelligence in AI agents and there is likely a correlation between doing better at SocialIQA and being a better interaction agent.

[1] Lee, Mina, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard-Ursin, et al. 2023. “Evaluating Human-Language Model Interaction.” Transactions on Machine Learning Research. https://openreview.net/pdf?id=hjDYJUn9l1.

Reviewer comment (Novelty):
What is the relationship between SOTOPIA and other social interactive tasks? How would solving SOTOPIA benefit the overall social intelligence of an agent? For example, does fine-tuning an agent on SOTOPIA events benefit its ability to understand social morality?

Author response:
Thanks for pointing out that SOTOPIA offers a wide range of realistic social tasks and adds a valuable perspective and allows for a comparison between AI models and human performance. We appreciate your questions that indeed highlight crucial aspects of our work, and we are happy to address them. 

### How SOTOPIA addresses the limitations of the previous benchmarks for social intelligence and how they relate:

1. The current benchmarks, such as SocialIQA, primarily consist of descriptive interactions paired with question-and-answer (QA) sets. In these cases, the model doesn't engage in real-time interactions. In contrast, Sotopia introduces a dynamic approach. It evaluates Large Language Models (LLMs) within a first-person interactive framework, allowing for direct engagement. This marks a significant shift from the static nature of existing benchmarks to a more dynamic and interactive evaluation environment in Sotopia.
2. With the rapid development of LLMs, some of the benchmarks gradually become saturated. While there are new adversarial benchmarks to evaluate social intelligence, which are harder than their predecessors, they still lack the dynamic nature of social interactions and the rich social context, which is deemed insufficient for evaluating social intelligence in AI systems [1].
3. Our work is indeed inspired by several benchmarks and datasets in social intelligence, such as SocialIQA, Persona Chat, and others. For instance, many of our scenarios come from SocialIQA, and our character design is inspired by Persona Chat (see more details in Appendix B.2 Scenarios)
4. While our current focus is on evaluation rather than the improvement of agents’ social intelligence, we acknowledge the importance of the latter. We have ongoing work using SOTOPIA to enhance social intelligence in AI agents and there is likely a correlation between doing better at SocialIQA and being a better interaction agent.

[1] Lee, Mina, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard-Ursin, et al. 2023. “Evaluating Human-Language Model Interaction.” Transactions on Machine Learning Research. https://openreview.net/pdf?id=hjDYJUn9l1.

Reviewer comment (Evaluation):
How to justify the metrics proposed by SOTOPIA including GOAL, BEL, REL, KNO, SEC, SOC, & FIN. For example, how are those metrics connected to sociological or psychological studies?

Author response:
### Justification of SOTOPIA Metrics

It's a great point that metrics should draw on the existing sociological or psychological literature. We'd like to emphasize that our metrics do draw on multiple pieces of literature from these domains, as well as economics. Specifically: Goal Completion (Weber, 1978), Believability (Joseph, 1994; Park et al., 2023a), Knowledge (Reiss, 2004; Maslow, 1943), Secret (Reiss, 2004), Relationship  (Maslow, 1943; Benabou & Tirole, 2006), Social Rules (Maslow, 1943), and Financial and Material Benefits (Gilpin & Sandholm, 2006; Burns et al., 2017). We believe that these were already described in some detail in Section 3 in the submitted draft, but we would be happy for any suggestions about how to make this more clear. Since these metrics are based on multiple disciplines, similar concepts are often defined under different names. We unify similar concepts from different domains by giving them a new definition that is compatible with most of the existing definitions and SOTOPIA scenarios.

Reviewer comment (Evaluation):
The context, goals, and constraints in SOTOPIA are generated by prompting GPT-4. However, in scenarios involving agent-to-agent or agent-to-human interactions, the related aspects, e.g. completion of goals and financial benefits, are also evaluated by GPT-4 or other LLMs. This raises a significant concern as the content of the generated scenarios is already present in the training data. So it is unclear whether the evaluation using GPT-4 is biased or not.

Author response:
Thank you for taking the time to review our paper and provide thoughtful feedback. 

*"The context, goals, and constraints in SOTOPIA are generated by prompting GPT-4. However, in scenarios involving agent-to-agent or agent-to-human interactions, the related aspects, e.g. completion of goals and financial benefits, are also evaluated by GPT-4 or other LLMs. This raises a significant concern as the content of the generated scenarios is already present in the training data. So it is unclear whether the evaluation using GPT-4 is biased or not."*

We acknowledge this is an important issue when evaluating large language models. We have carefully considered the issues of potential bias throughout our framework and experiment design.

###  *If you are asking whether GPT-4 would significantly prefer its own output during evaluation*

This may be a factor, but it does not change our main conclusions as those are corroborated by human evaluations. In addition to GPT-4 based evaluation Table 2 and 3, we have human-annotated evaluation results in Table F.1 and F.2, and they show similar trends. Putting these together with results in Table 1, we can conclude that GPT-4 based evaluation is a reasonable indication of models’ performance on SOTOPIA tasks. In addition, for future evaluations using SOTOPIA, it is also possible to reproduce our human evaluation protocol, so we can re-examine our assumptions about the correlation between human and machine evaluation as more competitive non-GPT-4 models are released. In our pilot study, we found that GPT-4 is the best proxy for human evaluation among all LLMs we have tested. We are conducting more rigorous experiments to evaluate other models’ correlations with human judgments. We will update our revision as soon as we get the results. 

### *If you are asking about biases related to scenarios being more "in-domain" for GPT-4 than other models*

We not only use templates to steer GPT-4’s output, but also manually inspect and correct the generated scenarios and goals. As such, the language distribution of our diverse scenarios significantly deviates from GPT-4's most probable language distribution [1]. Furthermore, the complexity of the task also depends on the partner agent during the interactions (Section 2.1). We will clarify this in the paper.

We are not sure what the reviewer is referring to with "the generated content being present in the training data." For the reviewer's point about the generated content possibly being present in the training data: we would clarify that we do not train any models in our work. If the reviewer is pointing out that generated scenarios might reflect GPT-4's training data: we verified that our scenarios are novel generations and are not present in the public web corpora that we could browse (we searched for our scenarios through n-gram matching with https://wimbd.apps.allenai.org/  in OpenWebText, C4, OSCAR, The Pile, LAION-2B-en and found 0% of them in there, while we don't know what GPT-4 is trained on, these datasets are representative of data used to train LLMs), so we have very little reason to believe that they are in the training data of GPT4 or any other language model. We will discuss this in our paper.

[1] Kim, Hyunwoo, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, et al. 2022. “SODA: Million-Scale Dialogue Distillation with Social Commonsense Contextualization.” EMNLP 2023

Reviewer comment (Evaluation):
SOTOPIA-EVAL introduces a multi-dimensional framework that utilizes different scores with hand-crafted ranges. However, there is no quantitive explanation/definition for those metrics. According to the prompts, only the positive/negative and higher scores are described, so the scores are not presented to the LLMs like the Likert scale presented to human evaluations and appear to be heuristic in nature.

Author response:
*"However, there is no quantitive explanation/definition for those metrics. According to the prompts, only the positive/negative and higher scores are described, so the scores are not presented to the LLMs like the Likert scale presented to human evaluations and appear to be heuristic in nature."*

We provided the same rating instructions for our GPT-4 evaluator and human annotators. And we would like to point out that previous work has demonstrated that LLMs can be effectively used for annotating subjective social tasks. [1].
While there are inherent challenges in rating social interactions, our human eval shows that GPT-4 provides an evaluation for models that aligns with human annotations at the system level (Table F.1). 
Thank you very much for the suggestion of making the prompts more fine-grained. We agree that this is a good idea, and so we ran some experiments where the evaluator is additionally provided with the descriptions of quantitive definitions for each range of the scale (e.g., *Relationship Deteriorates (-5 to -3): Scores from -5 to -3 indicate that the relationship is deteriorating. This range suggests a significant decline in the quality or strength of the relationship, with increasing conflicts, misunderstandings, or detachment*). However, this unfortunately did not result in a significant difference and if anything the correlation with humans became slightly worse (see the table below). We also encourage future work to further improve the evaluation based on our human annotation, which will be publicly available upon acceptance.

|                   | SEC  | KNO  | SOC  | BEL  | REL  | FIN  | GOAL |
|-------------------|------|------|------|------|------|------|------|
| Original Pearson R| 0.22 | 0.33 | 0.33 | 0.45 | 0.56 | 0.62 | 0.71 |
| + Fine-grained Description Pearson R | 0.03 | 0.33 | 0.33 | 0.35 | 0.57 | 0.57 | 0.71 |


[1] Can Large Language Models Transform Computational Social Science?
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang and Diyi Yang
Computational Linguistics, 2023

Reviewer comment (Evaluation):
Some of the evaluation metrics are not really related to social intelligence. For example, believability can be just the ability to create fluent dialogue, it doesn’t need to be meaningful social interactions, and social rules can be just memorizing the patterns, it doesn’t capture the agent’s ability to understand the social scenarios.

Author response:
*"Some of the evaluation metrics are not really related to social intelligence. Fodr example, believability can be just the ability to create fluent dialogue, it doesn’t need to be meaningful social interactions, and social rules can be just memorizing the patterns, it doesn’t capture the agent’s ability to understand the social scenarios."*

We want to emphasize that our dimensions are specifically designed to evaluate social intelligence in AI agents: some of our metrics measure metrics specific to **AI agents** (e.g. believability) while others reflect human social intelligence (See section 3). The former, for example, led us to include, "believability" as a dimension needed specifically to measure AI social intelligence: it captures not only conversation naturalness but also the agent's character consistency (i.e., whether they remain "in-character" and consistent with their prompted persona); this is a critical shortcoming in many social AI systems  [1] that needs to be measured. This contrasts with **humans**, where persona consistency is less of a concern because most humans naturally have their own consistent persona. As for social rules, this dimension is not about just recalling rules, but about measuring whether AI agents can successfully follow or apply the *relevant* rules during the interactions, which often tends to be challenging in many social settings [2].


[1] Shuster, Kurt, Jack Urbanek, Arthur Szlam, and Jason Weston. 2022. “Am I Me or You? State-of-the-Art Dialogue Models Cannot Maintain an Identity.” In Findings of the Association for Computational Linguistics: NAACL 2022.

[2] Jin, Zhijing, Sydney Levine, Fernando Gonzalez, Ojasv Kamal, Maarten Sap, Mrinmaya Sachan, Rada Mihalcea, Josh Tenenbaum, and Bernhard Schölkopf. 2023. “When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment.” Neurips, 2023

Reviewer comment (Experiments):
What are the distributions of social goals and relationships? For an evaluation framework, it will be important to understand what interactions are included. For example, in addition to collaboration and competition, what are the types and number of scenarios included?

Author response:
*"What are the distributions of social goals and relationships? For an evaluation framework, it will be important to understand what interactions are included. For example, in addition to collaboration and competition, what are the types and number of scenarios included?"*

There are 28 negotiation scenarios, 62 exchange scenarios, 7 competition scenarios, 17 collaboration scenarios, 12 accommodation scenarios, and 14 persuasion scenarios. Note that these categories overlap. We will include this distribution in the paper.

Reviewer comment (Experiments):
The framework and the evaluation mainly focus on two agents. It is not trivial to extend the interaction and evaluation to multi-agent coordination. How can we scale the interaction and evaluation to more complex social scenarios?

Author response:
*"The framework and the evaluation mainly focus on two agents. It is not trivial to extend the interaction and evaluation to multi-agent coordination. How can we scale the interaction and evaluation to more complex social scenarios?"*

We agree with the reviewer that extending Sotopia to multi-agent (>2 agent) scenarios is an interesting topic that future work should explore. From a practical perspective, SOTOPIA, built on the PettingZoo framework, could support interactions among multiple agents. The only additional feature required would be managing turn-taking, which could be done via a number of methods, such as bidding [1]. Evaluating multi-dimensional social intelligence for each character in multi-agent scenarios can be trivially conducted in SOTOPIA.

[1] Selfridge, Ethan O., and Peter A. Heeman. 2010. “Importance-Driven Turn-Bidding for Spoken Dialogue Systems.” In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, 177–85. ACL ’10. USA: Association for Computational Linguistics.