Paper: Lyfe Agents: generative agents for low-cost real-time social interactions (ICLR-2024)

Reviewer comment (Novelty):
The paper uses mainly people in the field already practice, e.g., reflecting on top of reflections in generative agents is similar to the self-monitoring in lyfe agents, and time as a inverse importance metric in generative agents is similar to summary-then-forget.

Author response:
**Q:** The paper uses mainly [techniques that] people in the field already practice, e.g., reflecting on top of reflections in generative agents is similar to the self-monitoring in lyfe agents, and time as a inverse importance metric in generative agents is similar to summary-then-forget.

**A:** We believe there are **important differences** between the techniques mentioned and our proposed techniques. We are currently **conducting new experiments** to directly compare our summarize-and-forget technique with time-based importance metric. We expect to update the reviewer with our results in 3-4 days.

Meanwhile, we will edit the text to clarify the difference and innovation of our techniques. We provide some of that clarification here for the reviewer’s convenience. It is important to note that Park et al.’s reflection does not correspond to our self-monitoring module and we argue that this is a *novel* contribution:

*Self-Monitoring Summary*: The function of reflect in Park et al. 2023 is to derive higher-order reasoning on items that are passed to the memory stream. The output of reflection are additional items which are added to the memory stream. Our agents have a reflect action as well which performs this operation — it is different from Park et al. in that it requires a single language model call and is triggered at the end of conversations, but nonetheless serves a similar function.
The self-monitoring summary is responsible for providing coherent context to the agent for decision making. Certainly, context *must* be provided to agents for sensible action selection, and such a mechanism is present in Park et al. However, it’s our method for providing coherent and situationally persistent context here that is novel. The method is cheap in terms of LLM calls required and conditions goal-adherence. It follows a rolling update rule, where goal-relevant recent observations and related memories modify the previous summary, in a single language model call.
	In contrast, the agents from Park et al. generate a context summary from their memory stream through three LLM calls. This was generated through two prompts that retrieve memories and view the queries “What is [observer]’s relationship with the [observed entity]?” and “[Observed entity] is [action status of the observed entity]?”, and their answers summarized together. Another call is used to then decide based on the context summary whether to react and what that reaction would be.
Bringing this all together, Park et al’s method for providing a context summary requires three language model calls for every action decision step. Our approach of using a rolling update has not only the benefit of using a single LLM call for updating, but also the benefit of reusing these contexts multiple times — we don’t need to update the summary frequently since the situational context isn’t expected to change rapidly.

*Summarize-then-Forget Memory*: For the comparison between time-based importance and our summarize-then-forget mechanism, as mentioned above, we will run comparisons between these architectures. However, we want to emphasize that the motivation behind summarize-then-forget was to reduce the storage of redundant memories. This is important when under the constraint of retrieving fewer memories for action selection — if you can only retrieve 4 memories, it is much less useful if these memories are repetitive.

Reviewer comment (Reproducibility):
The paper is missing a lot of key implementation details in order to reproduce, see questions section for a list of them.

Author response:
Reviewer correctly pointed out the lack of implementation details. We are adding **substantially more details** to the main text right now. We can provide the answers to the reviewer’s specific questions for the reviewer’s convenience once these changes are made.

We are in the process of enhancing our paper with additional technical details and a more logical structure for explanations. To date, we have expanded the main text with essential technical information and supplemented this with detailed data in the Appendix. Our efforts to further clarify these technical aspects are ongoing.

Reviewer comment (Experiments):
The environment seems like a navigatable chatroom (only walk/talk actions), which seems a bit simplified to incentivize more complex social behaviors.

Author response:
**Q:** “A navigable chatroom (only walk/talk actions)...[too] simplified to incentivize more complex social behaviors”

**A:** We agree that for this paper, the environment is simplified and does not support complex agent-object interactions or physical movements. We will emphasize this limitation in the Conclusion & Discussion section. 

We do think that a wide range of complex social behaviors can still be simulated in a navigable chatroom. A house-warming party, a presidential debate, and a diplomatic negotiation are all largely conducted through verbal communications.

Reviewer comment (Experiments):
While the proposed agent is performant, it's unclear if this performance is consistent across various base LLMs. The experiment appears to use GPT-3.5. What would happen if we use different LLMs such as Alpaca and GPT4ALL, or LLMs with various model capacities? Without this information, it remains unclear whether using GPT-3.5 is a crucial requirement or not. Also, from the perspectives of reproducibility and transparency, using open-sourced LLMs would be more desirable.

Author response:
### **Alternative LLMs**
**Q:** …unclear if this performance is consistent across various base LLMs

**A:** We agree. We are currently **re-running our simulations** where the LLM is Llama-2-70B, Mistral-7B, or GPT4. The first two are open-sourced.

## Rebuttal 2 (ID: Oo9WjXL7uH)
**1. Running simulations for other language models.**

We ran our simulations using GPT4, Mistral-7B, and Llama-70B and present the results of our preliminary exploration. To normalize these comparisons, we modified the simulation length for each one so that they had approximately the same number of action selection decisions per simulation. We preface that these results are preliminary, we mention some issues below. The results of each trial:

| Model   | Mean ± Standard Error |
|---------|-----------------------|
| gpt3.5-Turbo   | 0.58 ± 0.08            |
| gpt4   | 0.44 ± 0.09            |
| llama2-70B   | 0.13 ± 0.06            |
| mistral-7B   | 0.23 ± 0.09            |


The GPT3.5 runs include the simulations from the original submission along with some additional runs we made during the rebuttal period. We note that there is a caveat for Mistral and Llama, the percentage of incorrectly formatted outputs was higher. This led to requiring longer runs (since effective action selection takes longer to get right) and there may be other confounding factors contributing to the performance.

Reviewer comment (Evaluation):
The cost analysis was not very convincing. Currently, "cost per agent per human hour" is reported. However, for tasks with specific goals such as murder mystery, isn't it more essential to show the cost required until the task is successfully completed? Otherwise, agents with very low throughput (due to, say, insufficient compute resources) will be evaluated as low cost, which I don't believe this paper intends to argue. Furthermore, reporting costs in USD seems unhelpful. This would result in running open-sourced models in a region with cheap electricity as an optimal choice. What happens if the service provider changes API usage prices in the future? Why not just report the number of interactions or number of tokens consumed until task completions?

Author response:
### **Better cost analysis**
**Q:** The cost analysis was not very convincing. Currently, "cost per agent per human hour" is reported…isn't it more essential to show the cost required until the task is successfully completed? 

**A:** Because the task can’t always be completed (e.g. with component ablations), we can’t calculate tokens needed for task completion. We will, however, show how the performance changes as a function of **both tokens and monetary cost**. 

**Q:** reporting costs in USD seems unhelpful… Why not just report the number of interactions or number of tokens consumed until task completions?

**A:** We will report both token consumption and monetary cost. We explain the reasoning below and will add it to the main text:
“We report both token consumption and monetary costs. When using the same base LLM, reporting token consumption gives a fair comparison and is robust to API price change. However, for different base LLMs that may vary dramatically in model size (e.g. GPT4 vs 7B open-source models), the relative monetary cost is a (flawed) approximation of the compute resource used.”

**Q:** Can the contribution of each technical component be evaluated in terms of cost-effectiveness, such as the number of interactions or the number of tokens consumed?

**A:** We are rewriting our main text to better explain how our proposed modules reduce the cost while keeping the performance high. We are planning to look into adapting Stanford GenAgent’s memory and retrieval architecture into our agents as a better point of comparison, and relying on token-based cost for comparison rather than purely USD. We note that the GenAgent architecture is interwoven with the environment, so this adaption may face difficulties, but we will regardless add a **new section that explains why it’s difficult to directly compare them**. The low-cost nature of our work remains true, but we will not emphasize it by comparing it with the Stanford paper. (Stanford paper did not use GPT-4 though, they used text-davinci-3, which is still more expensive)

## Rebuttal 2 (ID: Oo9WjXL7uH)
**2. Comparison of cost with Stanford**

To make a better comparison of our cost with that of the architecture from Park et al., we took their memory and implemented it in our agent for comparison. We kept our self-monitoring summary and action selection modules, but replaced our memory. Our runs indicate that our memory utilizes approximately 30 times fewer tokens than that of the Park et al. memory. Due to differences in latency, we normalize the comparison by looking at the rate of token cost for the memory per action-selection. For our base architecture, the cost was 22.00 tokens per action-selection whereas for Park et al. we found 612.53 tokens per action-selection. We note that the performance using Park et al.’s architecture was also not great, though we anticipate this may be due to our manner of defining memory keywords for which there wasn’t a straightforward adaptation from Stanford to our setting. 

**With your great suggestion on extending“success weighted by path length”, we are doing experiments and plan to further investigate this.**

Reviewer comment (Theory):
The logic of this paper is not rigorous.

Author response:
**Q:** The overall logic of this paper is unclear. What motivates the proposal of the three modules for Lyfe Agents, and how do these modules differ from those in other LLM-based agents? The improvements offered by these modules are difficult to discern.

**A:** We are including substantially more technical details about our agents’ implementations, and are contextualizing our contributions by more directly comparing the modules with relevant modules in the literature.

Reviewer comment (Novelty):
The overall logic of this paper is unclear. What motivates the proposal of the three modules for Lyfe Agents, and how do these modules differ from those in other LLM-based agents? The improvements offered by these modules are difficult to discern.

Author response:
We have incorporated several changes to the paper, though this is ongoing. The introduction better emphasizes the aim of this work as developing agents for social simulation and the relation between the modules and cost effectiveness, with details in Section 3 --- more on this in the next paragraph. We moved the literature review to follow the introduction, better contextualizing our work amidst them. In particular, we discuss language model powered agents, existing benchmarks and problems therein, long-term memory via vector database, and coherent goal-directed behavior with reflection. More details will be provided on model architecture.

We are still in progress of making these changes, but want to summarize the argument for the cost-effectiveness of each module. For action-selection our argument remains the same, to save calls on choosing higher level actions through the hierarchical design. We plan to incorporate our investigations from item (2) above to demonstrate our consideration for cost in memory. We also emphasize the cost-effectiveness and novelty of the self-monitoring summary as a way to provide context for action-selection. In particular, we point out that this provides context for action-selection with a slow update that allows for reuse across multiple action steps — as the contextual narrative changes slowly relative to action steps.

Reviewer comment (Experiments):
The authors do not fairly compare Lyfe Agents with existing LLM-based agents. The paper's most significant claim pertains to the 'low-cost' agents depicted in Figure 6. However, the authors should attempt to keep all variables consistent when comparing the queries of LLMs. Comparing generative agents with Lyfe Agents in different scenarios is unjust. Also, the Lyfe Agents should be compared with other general social simulation agents such as [1,2].

Author response:
**Q:** The authors do not fairly compare Lyfe Agents with existing LLM-based agents. The paper's most significant claim pertains to the 'low-cost' agents depicted in Figure 6. However, the authors should attempt to keep all variables consistent when comparing the queries of LLMs. Comparing generative agents with Lyfe Agents in different scenarios is unjust! Also, the lyfe agents should compare with other general social simulation agents such as [1,2].

**A:** We thank the reviewer for suggesting comparisons with other agents. In the current stage of LLM-based agent research, fair comparison between agents is generally difficult because the application domains can be vastly different. We agree with this assessment. Therefore we are **adding a section** detailing other agents, describing this challenge. We are also **removing direct comparison** of the cost between Stanford Generative Agent from our main text. 

For the same reason, it is difficult to compare Lyfe Agents with [1, 2]. The AgentSims paper [1] proposes a Sandbox like the Stanford paper [3], but neither [1, 3] provides a standardized benchmark to evaluate agents. The Wang paper [2] is focused on a recommender framework that is different from our goal of simulating social behavior in virtual environments.

That said, we are working on implementing some components of the Stanford paper [3] into our agents and evaluating them, as mentioned in our response to an earlier question. We will provide an update as soon as possible.

Reviewer comment (Evaluation):
There are no objective evaluation metrics for Lyfe Agents other than cost per hour (which, as stated in point 3, is an unfair comparison). The police success rate shown in Figure 3b is absurd. It is based on a self-designed script with predetermined and clear answers, which is not convincing and should not serve as an evaluation metric in a scientific paper.

Author response:
**Q:** There are no objective evaluation metrics for Lyfe Agents other than cost per hour (which, as stated in point 3, is an unfair comparison). The police success rate shown in Figure 3b is absurd. It is based on a self-designed script with predetermined and clear answers, which is not convincing and should not serve as an evaluation metric in a scientific paper.

**A:** As mentioned above, we are evaluating other agents in this environment now. We hope these new experiments show that the scenarios we designed are not trivial for other agents.

We want to clarify what the reviewer means here. The agent backstories are provided in the appendix where only a single agent has the memory available to infer the murderer. The task is then an evaluation of information spreading to the right agent, namely the detective Lizhi. While the backstory is preset, the agents are not scripted. Their action selection is naturally stochastic, due to the stochasticity of GPT. If the reviewer is concerned that the task is trivial, we want to emphasize that this was the objective of our ablation analysis. For additional comparisons, as mentioned earlier, we are investigating the inclusion of the Park et al. memory and retrieval mechanism. More updates on that are to come.

Reviewer comment (Experiments):
The proposed framework in Figure 2 should be a general LLM-based agents framework. The Lyfe Agent should be evaluated in other well-known scenarios such as games and generative agents [3,4].

Author response:
**Q:** The proposed framework in figure 2 should be a general LLM-based agents framework. The lyfe agent should be evaluated in other well-known scenarios such as games and generative agents [3,4].

**A:** Our work focuses on social interactions, so we intentionally avoided games and evaluations such as Minecraft [4]. It is also difficult to evaluate our agents fairly with agents such as Voyager [4] that execute actions by writing code. 

The Stanford paper [3] does not provide a reproducible benchmark to evaluate on. The only quantitative benchmark provided requires human participants to judge whether the agents’ answers to interview questions are reasonable. This evaluation is too expensive and time-consuming for us and most practitioners in the field.

In fact, for these reasons, the more agent papers referenced by the authors [1, 2] as well as other recent agent papers (e.g. ChatDev from Tsinghua) typically don’t evaluate their agents on other scenarios.

Reviewer comment (Novelty):
Generative agents [3] propose reflection to summarize the high-level memory. What is the difference between the proposed memory mechanism and the one in generative agents [3]?

Author response:
**Q:** Generative agents [3] propose reflection to summarize the high-level memory. What is the difference between the proposed memory mechanism and the one in generative agents [3]?

**A:** We are editing the text to better clarify how the reflection from Park et al. and our self-monitoring summary differ. Likewise for time-weighted decay and the forgetting algorithm. This textual edit and new simulations should explain the difference.

We are currently **conducting new experiments** to directly compare our summarize-and-forget technique with time-based importance metric.
Meanwhile, we will edit the text to clarify the difference and innovation of our techniques. We provide some of that clarification here for the reviewer’s convenience. It is important to note that Park et al.’s reflection does not correspond to our self-monitoring module and we argue that this is a novel contribution:
	The function of reflect in Park et al. 2023 is to derive higher order reasoning on items that are passed to the memory stream. The output of reflection are additional items which are added to the memory stream. Our agents have a reflect action as well which performs this operation — it is different from Park et al. in that it requires a single language model call and is triggered at the end of conversations, but nonetheless serves the same function.
	The self-monitoring summary is responsible for providing coherent context to the agent for decision making. Certainly, context must be provided to agents for sensible action selection, and such a mechanism is present in Park et al. However, it’s our method for providing coherent and situationally persistent context here that is novel. The method is cheap in terms of LLM calls required and conditions goal-adherence. It follows a rolling update rule, where goal-relevant recent observations and related memories modify the previous summary, in a single language model call.
	In contrast, the agents from Park et al. generate a context summary from their memory stream through three LLM calls. This was generated through two prompts that retrieve memories and view the queries “What is [observer]’s relationship with the [observed entity]?” and “[Observed entity] is [action status of the observed entity]?”, and their answers summarized together. Another call is used to then decide based on the context summary whether to react and what that reaction would be.

Reviewer comment (Reproducibility):
Whilst the intentions and role of all of the modules are well explained, the details of their implementations are not particularly well explained within the main body of the text. For example, take the option-action selection module: what are the implementation details for the LLM call that is used to output an option along with a subgoal? What are the available options, is this defined by the LLM or a discrete list provided beforehand? What are the details of the exit conditions that lead to the option-action selection module being called? Whilst there are some more details included in the appendix, I believe some more important implementation points should be included in the main body of the text.

Author response:
**Q:** Whilst the intentions and role of all of the modules are well explained, the details of their implementations are not particularly well explained within the main body of the text.

**A:** We agree. We are adding substantially more technical details of the key components into the **main text** and Appendix, and will better contextualize these components with the literature.

Reviewer comment (Evaluation):
I am not currently convinced by the cost analysis that is provided by the authors. This is particularly important as one of the key-selling points of the framework, according to the authors, is the cost effectiveness of the framework. I am not disputing the project cost of the Stanford GenAgent work. However, I just have a few queries about the cost provided for the Lyfe Agent. Firstly, is the difference in cost primarily coming from the fact that GPT3.5 is used for LyfeAgent vs. GPT-4 for Stanford GenAgent? In my opinion, this isn't necessarily a fair comparison — instead would it not be fairer to evaluate more in terms of tokens inputted / outputted when solving the scenarios? For example, how many tokens need to be inputted / outputted by LyfeAgent vs. Stanford GenAgent in order to arrive at a Murder mystery solution? In my opinion this is a fairer comparison.

Author response:
**Q:** I am not currently convinced by the cost analysis that is provided by the authors…isn't necessarily a fair comparison…fairer to evaluate more in terms of tokens inputted / outputted when solving the scenarios

**A:** We will add a detailed account of token usage for different components.

Thank you for pointing this out. We agree that comparing across different LLM-based agents can often be apples and oranges, because the environments/tasks are very different. We are planning to look into adapting Stanford GenAgent’s memory and retrieval architecture into our agents as a better point of comparison, and relying on token-based cost for comparison rather than purely USD. We note that the Stanford GenAgent architecture is interwoven with the environment, so this adaptation may face difficulties, but we will regardless add a **new discussion section that explains why it’s difficult to directly compare them fairly**. The low-cost nature of our work remains true, but we will not emphasize it by comparing it with the Stanford paper. (Stanford paper did not use GPT-4 though, they used text-davinci-3, which is still more expensive)

Reviewer comment (Reproducibility):
Please provide more implementation details for the modules, especially in the main body of the paper. Currently, for example, I would not personally be able to reproduce this framework based on all the information provided in the paper and appendix.

Author response:
**Adding implementation details in the main text**

To enhance the clarity of our manuscript, we are actively working on better contextualizing our key components, incorporating additional technical details and relevant literature.

We have incorporated several changes to the paper, though this is ongoing. The introduction better emphasizes the aim of this work as developing agents for social simulation and the relation between the modules and cost effectiveness, with details in Section 3 --- more on this in the next paragraph. We moved the literature review to follow the introduction, better contextualizing our work amidst them. In particular, we discuss language model powered agents, existing benchmarks and problems therein, long-term memory via vector database, and coherent goal-directed behavior with reflection. More details will be provided on model architecture.

Reviewer comment (Novelty):
Implementation of the virtual world: I see no benefit in highlighting the 3D environment work this work contribution. The interaction is purely text-based, and the agents do not perceive their visuals as image inputs. Thus, the work could be reduced to a text-based role-play scenario.

Author response:
**A:** We agree. The 3D environment makes it more appealing and intuitive for human users, and supports the use of image-based or ray-cast visual inputs, which is motivation for future work. You are right that 3D shouldn’t be seen as the selling point for this work (not our intention either). We are adding this to the main text: 
“The 3D nature of this environment is not pertinent to the scientific results of this work. The same results can in principle be obtained in any environment which supports navigation and group chat dynamics.

Reviewer comment (Presentation):
Appendix Structure: I found relevant information about the agent architecture missing in the content but given inside the appendix. Thus, moving relevant information into the body of the paper would emphasize the main contribution.

Author response:
**A:** We are working on adding substantially more technical details of the key components to the main text, and will better contextualize these components with the literature. We will also make space by reducing the size of Figure 1 (environment).

Reviewer comment (Experiments):
Reliance on GPT3.5: Like previous agent-based work (Park et al.), the observations and consequences are limited to a singular foundation model. As this work employs complex memory and decision mechanisms, the behavior of competitive models like Llama-2 70B or Falcon 180B, when prompted with this approach, may strengthen impact.

Author response:
**A:** We are currently **running more simulations** replacing GPT3.5-turbo with Llama-2-70B, Mistral-7B, and GPT4.

We ran our simulations using **GPT3.5, GPT4, Mistral-7B, and Llama-70B** and presented the results of our preliminary exploration. To normalize these comparisons, we modified the simulation length for each one so that they had approximately the same number of action selection decisions per simulation. We preface that these results are preliminary, we mention some issues below. The results of each trial:

| Model   | Mean ± Standard Error |
|---------|-----------------------|
| gpt3.5-Turbo   | 0.58 ± 0.08            |
| gpt4   | 0.44 ± 0.09            |
| llama2-70B   | 0.13 ± 0.06            |
| mistral-7B   | 0.23 ± 0.09            |

The GPT3.5 runs include the simulations from the original submission along with some additional runs we made during the rebuttal period. We note that there is a caveat for Mistral and Llama, the percentage of incorrectly formatted outputs was higher. This led to requiring longer runs (since effective action selection takes longer to get right) and there may be other confounding factors contributing to the performance.

Reviewer comment (Novelty):
The quality and relevance of the paper would improve with more focus on the contribution regarding the learning representation itself: memory architecture, discerning storage, and retrieval.

Author response:
**A:** We are working on adding substantially more technical details of the key components to the main text, and will better contextualize these components with the literature. We will also make space by reducing the size of Figure 1 (environment).

To enhance the clarity of our manuscript, we are actively working on better contextualizing our key components, incorporating additional technical details and relevant literature.