Paper: CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization (ICLR-2024)

Reviewer comment (Evaluation):
The accuracy of the generated memory is not evaluated, and it is unclear whether the performance improves because the LLM generates the correct memory or just provides more details to the task which may not be causal.

Author response:
**> How accurate are the generated memories?**

Thank you for asking this question! While the final performance with memory is indicative of their effectiveness, we performed additional human evaluation of generated memory insights for correctness. For both the Gen-Env and Gen-Task setups, we randomly 10 task-environment combinations to evaluate the correctness of memories used in them, notably the meta-memory used for trial 0 (GEN) and memory adapted for the best trial (G+A). Two annotators rated all the insights (cohen's kappa 0.72) for correctness with reference to the respective gold trajectories, and the results are as follows:

| | GEN-Env (Trial 0) | G+A (best trial) |
|-----|-----|-----|
| No. of insights | 100 | 105 |
| Correct insights | 72.0% | **91.4%** |
| Final score (on sampled tasks) | 39.1 | **55.9** |


| | GEN-Task (Trial 0) | G+A (best trial) |
|-----|-----|-----|
| No. of insights | 98 | 107 |
| Correct insights | 73.9% | **91.1%** |
| Final score (on sampled tasks) | 43.7 | **58.1** |

So, it is possible to have some meta-memories (here ~27%) that are not applicable for a new task or environments at Trial 0; however, with adaptation, in later trials, the correctness of the memory insights significantly improves (17-19% absolute), leading to a direct increase in task performance. We included this analysis in the Appendix. 

**> Does the performance improve because the LLM generates the correct memory, or just provides non-causal details of the task?**

The memory generator is constrained (via prompting) to *always* produce causal statements, so there are no non-causal details in memory. From sample analysis, we observe that the generated memories are almost always correct with respect to the trials that have been seen, and empirically, performance improves using them (same task). However, when the environment or task changes, those learned memories may be over-general or incorrect (e.g., "activating the stove is necessary for heating water" is no longer correct in a new environment where the stove is broken). In such cases, we see an initial performance drop, but then further iterations cause those errors to be corrected, and performance improves even further. This can be seen in Figures 5a and 5b: In the same environment/task, CLIN improves (blue line), but then if the environment/task is changed, its performance drops as some learnings in memory are too specific (start of red line vs. end of blue line), but it then later recovers and improves even further (end of red line).

Reviewer comment (Experiments):
A causal abstraction doesn’t necessarily need to be structured. The drop in ablation when using the free-form advice suggests the importance of the structured sentence format rather than the causal abstraction. This ablation still doesn’t show the usefulness of causal abstraction.

Author response:
**> The ablation doesn't show the usefulness of the causal abstraction**

We apologize for being unclear here: When we drop the constraint that the memory should have a causal structure ("is necessary" / "does not contribute"), the memory generator can generate anything - and, in fact, stops generating causal abstractions and instead generates generic and often unhelpful advice (as others doing unconstrained memory generation have also found). Some example unconstrained generations are:

1. Avoid repeating actions: The agent performed the "look around" action twice at the beginning of the task. This is unnecessary and wastes time. Once you have a clear understanding of your surroundings, move on to the next step. 
2. Be mindful of the order of actions: The agent moved to the workshop and read the instructions after looking around twice. It's better to first read the instructions to understand the task requirements.
3. Be clear with your actions: The agent made an ambiguous request when trying to mix the cup containing red paint. Be specific with your actions to avoid confusion.
4. Avoid unnecessary movements: The agent teleported to the art studio after picking up the paper. Plan your actions carefully to avoid unnecessary movements.

The ablation (Figure 5d) thus shows the value of having a memory focused on **understanding the causal effects of actions** (rather than just any old reflection), following our rationale that this will help an agent mentally simulate their choices before acting. This is a key contribution of our work, and distinguishes it from other reflective approaches. For better readability, we included this example in the Appendix.

Reviewer comment (Evaluation):
The experiment results show that the longer tasks such as growing fruit in Fig 4 are harder for CLIN to adapt. But Fig 4c shows memory improves more episodes for the longer tasks; what are the improvements for long vs. short tasks? What kind of information is still missing for the long tasks?

Author response:
**> What kind of information is still missing for the long tasks?**

Longer tasks often require actions whose effects are not observed immediately but observed later in the time. Many longer tasks require frequent "waiting," allowing the environment to have the necessary changes (waiting for the bees to cross-pollinate). While our agent could not perform well at the initial trials later added a new memory: "Repeatedly waiting after opening the bee hive may be necessary to the growth of the cherry plant.". However, such a memory, while helpful, is often not precise about how long the agent needs to wait for the cross-pollination to happen, a stochastic process. Due to a lack of dense reward and sparse signal from the observations, our agent sometimes cannot complete longer tasks, which is indicated by Figure 4a. 

Figure 4c shows the % of episodes where there is a positive change in scores as compared to the Base agent (first trial). For shorter tasks, the room for improvement is less, as the average absolute score is higher as compared to longer tasks. However, we show higher relative improvements for longer tasks, indicating CLIN is able to bridge the performance gap by rapid adaptation.

Reviewer comment (Novelty):
The novelty is somewhat limited. Specifically, the authors discussed the difference between their method and Reflexion. But it is not clear if the causal abstractions can be generalized in distinct environments with different goals. Even if they can, I assume that the summaries kept in memory are pretty high-level and abstract. Can these summaries still benefit the performance of individual tasks? The experiments in the ScienceWorld benchmark make sense as different tasks share common rules. More experiments would be good to support the method, such as the benchmarks used in Reflexion (e.g., ALFWorld that the authors used as examples), where common rules may be hard to find useful for individual tasks.

Author response:
**> Can you clarify the novelty?** 

There are two key novelties:

1. While the idea of having a memory is not new, our novel contribution is to make this a **dynamic, evolving memory over time**, in contrast to the short-term "reflect, use, then discard" approach used in Reflexion and other memory-based agents. This allows CLIN to progressively build insights about the world over multiple experiences in different environments. It also allows CLIN to recover from bad or not-useful learnings in memory, as these will get dropped when they are found not to help as the memory is updated.

2. The **role** of the memory is itself novel, namely to help the agent mentally simulate the (learned) causal effects of actions, thus making more informed action choices in the future. This was missing in prior work, where memories were either task-specific factual instructions (e.g., "I should go to desk 1") or general hints ("Think about an object's usage when searching for it."). The idea of learning the causal effects of actions ("action models"), to help simulation/planning, has been previously explored in formal planning. Our work is the first to take this idea into the modern context of LLMs, and show its benefits.

**> Can causal abstractions really be generated from different environments or tasks, and if so, do they still provide benefit?**

Yes, such generalized causal abstractions can indeed be generated: Our meta-memory (abstracted over past trials in different tasks or environments) contains such abstractions. For example, from the causal rules (1) "Using the lighter on the metal pot should be necessary to heat the water" and (2) "Turning on the stove should be necessary to create heat", CLIN generates the abstract rule "Using a heat source (stove, lighter) on the container should be necessary to heat a substance" (see Figure 3). Empirically, we do see that meta-memory provides benefits in novel tasks, both immediately and after further learning (Figure 5a and 5b, the with-meta-memory red line is above the blue line).

**> More support showing CLIN can generate generalized insights where common rules may be hard to find useful for individual tasks such as ALFWorld.**

Indeed, tasks in ALFWorld require specific insights such as "going to fridge 1 is necessary to find the tomato." However, in terms of complexity and task type, the Pick&Place tasks in ScienceWorld subsume most of the ALFWorld tasks, and agents (e.g., ReAct) that perform well on ALFWorld perform considerably poorly in similar tasks in ScienceWorld. On the contrary, CLIN achieves perfect scores in such tasks during generalization.

As a prototype for generalized memory in ALFWorld, we accrue two successful trajectories of Pick&Place tasks, and generate a meta-memory using CLIN's causal abstraction for a target task: "Find two remotecontrols and put them in armchair": 

1. Going to the drawers may be necessary to find the remotecontrols.
2. Opening the drawers should be necessary to access the remotecontrols.
3. Taking the remote controls from the drawer should be necessary to obtain the remotecontrols.
4. Going to the armchair 1 should be necessary to place the remotecontrols.
5. Putting remotecontorls in/on armchair 1 should be necessary to complete the task.

We manually compared the correctness of such generalized insights with the underlying PDDL file that spawns the task and found that the memory captures most of the necessary (causal) information needed to solve the target task, indicating CLIN-style meta-memory will be effective in ALFWorld-like environments too.

Reviewer comment (Experiments):
More comparisons to related works can be added, such as [1, 2] which also improve iteratively via trials.

[1] AdaPlanner: Adaptive Planning from Feedback with Language Models. Sun et al.  
[2] Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiency. Liu et al.

Author response:
**> Related work: AdaPlanner and "Reason for the Future, Act Now"**

Thank you for pointing out these interesting works, we will add them! As points of contrast, AdaPlanner's primary strength is adaptation/self-refinement of a plan during execution, rather than continual learning between plans/tasks. It does have a skill memory of successful plans, but does not abstract over them nor "learn" from failures beyond a datapoint. "Reason for Future, Act for Now", a concurrent work, is also interesting: they propose a memory-based approach to algorithmically reason over possible hypotheses for the next goal and action over a longer trajectory. However, they also do not persist insights in one trajectory to generalize over unseen tasks or environment – a key difference between recent works and our work. 

As a prototype, we implemented a version of AdaPlanner for Pick&Place tasks in ScienceWorld, similar to those it has been predominantly successful in ALFWorld. On such tasks, AdaPlanner achieves an average score of 38.3, slightly lower than ReFlexion (40.7), whereas CLIN (our agent) achieves a perfect score of 100 in all such (Pick&Place) tasks.

Reviewer comment (Evaluation):
Although CLIN claims to have the ability of self-exploration, it primarily focuses on known information, which may affect its performance in new environments.

Author response:
**> Is CLIN's ability to self-explore limited by known information?**

CLIN's memory both guides CLIN towards a solution (via the "is necessary for" links), but also explicitly encourages exploration if earlier attempts fail (via the "does not contribute to" links). For example, if in trial T, action X did not contribute to goal Y, this will be noted in memory ("X does not contribute to Y"), causing CLIN to explore a different action X' in the next trial. In this way, CLIN maintains a balance between exploration and exploitation, depending on the success and failures in earlier trials.

Reviewer comment (Novelty):
Although CLIN presents some potential improvements over existing methods, it still does not have significant breakthroughs within the framework of current works.

Author response:
**> What are the significant breakthroughs in CLIN?**

There are two significant advances in CLIN:

1. While the idea of having a memory is not new, our novel contribution is to make this a **dynamic, evolving memory over time**, in contrast to the short-term "reflect, use, then discard" approach used in Reflexion and other memory-based agents. This allows CLIN to progressively build insights about the world over multiple experiences in different environments. It also allows CLIN to recover from bad or not-useful learnings in memory, as these will get dropped when they are found not to help as the memory is updated.
2. The **role** of the memory is itself novel, namely to help the agent mentally simulate the (learned) causal effects of actions, thus making more informed action choices in the future. This was missing in prior work, where memories were either task-specific factual instructions (e.g., "I should go to desk 1") or general hints ("Think about an object's usage when searching for it."). The idea of learning the causal effects of actions ("action models"), to help simulation/planning, has been previously explored in formal planning. Our work is the first to take this idea into the modern context of LLMs, and show its benefits.

We have updated the Introduction in the revised paper to clarify these better.

Reviewer comment (Experiments):
CLIN's textual memory seems to depend heavily on causal abstractions. How does the system handle tasks where the relations are not causal or obscure?

Author response:
**> How does the system handle tasks where the relations are not causal or obscure?**

In general, non-causal knowledge has a causal import, which we observe is then captured in the memory. For example, "the block is in the workshop" would be captured by the causal expression "moving to the workshop may be necessary to find the block." In general, causal abstractions indirectly capture factual knowledge relevant to the tasks being considered. 

There are definitely other kinds of relations that are required for ScienceWorld -- like retrieving property knowledge. An agent needs to know the melting points of objects, the needs of living things, the affordances of objects, etc., which are critical to solving the tasks. We empirically show that our agents are achieving perfect scores (add numbers) in many of these tasks requiring apparent non-causal knowledge. However, qualitative analysis reveals that CLIN is largely able to convert such knowledge with the structured causal import and effectively use it for completion of the tasks.

Reviewer comment (Evaluation):
How robust is the system when facing incorrect or misleading information stored in its memory? Is there any mechanism in place to correct or override these inaccuracies?

Author response:
**> How robust is the system when facing incorrect or misleading information stored in its memory? Is there any mechanism in place to correct or override these inaccuracies?**

Yes - in fact, a key contribution of CLIN is its ability to correct errors in its memory via its dynamic evolution over time, unlike prior work. The memory is always updated after each trial. If the latest trace indicates some memories are unhelpful or simply wrong, they will not be retained in the updated memory, given the LM is following its prompt-based instructions correctly. Please see our **new results** in the "[General Response to Reviewers](https://openreview.net/forum?id=d5DGVHMdsC&noteId=YGdT6et2uW)" (also Appendix C, Table 2 in the revised paper) on CLIN's recovery from wrong knowledge in its memory via continual learning.

Reviewer comment (Theory):
How does the system handle potential changes in state that are not immediately reacted to an action?

Author response:
**> How does the system handle potential changes in state that are not immediately reacted to an action?**

These do indeed occur in ScienceWorld, e.g., water changes to vapor multiple several steps after the action "activate stove" with a pot of water on it. In this case, the actions "wait" or "take the water temperature" may be executed. CLIN learns through experimentation that some tasks require such actions, and such learnings may then be added to memory (e.g., when growing plants, "waiting may be necessary to allow the plants to grow", Figure 10).

Reviewer comment (Theory):
Could you provide more insights on how the executor modifies an action when it doesn't align with valid actions? Specifically, how does it ensure that the modified action still aligns with the intended goal?

Author response:
**> How does the executor modify an action when it doesn't align with valid actions?**

If the executor generates an invalid (unexecutable) action, a simple check detects this and the executor is iteratively called again (with a note not to generate the invalid action) until a valid action is produced (or a max number of attempts is exceeded). While refinement, the executor is still conditioned on the intended goal provided by the controller. This utility essentially implements a self-refine technique [Madan et al., 2023](https://arxiv.org/pdf/2303.17651.pdf) to address this problem in the executor module.

Reviewer comment (Novelty):
Technical Novelty: the idea of having a memory of past experiences and learned summary of knowledge is not very new; while this work proposes that the content in the memory is important, i.e., using causal abstractions helps agent's learning, the method is not that different in terms of the general idea.

Author response:
**> How novel is CLIN? (the idea of having a memory of past experiences is not new)**

There are two key novelties:

1. Indeed, the idea of having a memory is not new. Our novel contribution is to make this a **dynamic, evolving memory over time**, in contrast to the short-term "reflect, use, then discard" approach used in Reflexion and other memory-based agents. This allows CLIN to progressively build insights about the world over multiple experiences in different environments. It also allows CLIN to recover from bad or not-useful learnings in memory, as these will get dropped when they are found not to help as the memory is updated.

2. The **role** of the memory is itself novel, namely to help the agent mentally simulate the (learned) causal effects of actions, thus making more informed action choices in the future. This was missing in prior work, where memories were either task-specific factual instructions (e.g., "I should go to desk 1") or general hints ("Think about an object's usage when searching for it."). The idea of learning the causal effects of actions ("action models"), to help simulation/planning, has been previously explored in formal planning. Our work is the first to take this idea into the modern context of LLMs, and show its benefits.

We will make this clearer in newer versions of the paper and have updated the Introduction to clarify these better.

Reviewer comment (Theory):
The memory entirely depends on the agent's own exploration trace, so if the agent cannot explore enough and find some useful information, the agent cannot gain the corresponding knowledge.

Author response:
**> Does the memory depend entirely on the agent's exploration trace? If the agent doesn't explore enough, can the agent gain the required knowledge?**

Like any agent, the agent learns from exploration, so some required knowledge (e.g., a fridge is in the kitchen) necessarily needs exploration. If CLIN doesn't explore enough (e.g., it fails to find a fridge in the living room), it can gain the required knowledge in the next iteration by adding a memory: "going to the living room does not contribute to finding a fridge". This "negative knowledge" biases CLIN to explore in a different direction in the future, helping it gain the required knowledge for the task, and balance exploration and exploitation.

Reviewer comment (Theory):
When generating causal abstractions, how does the LLM resolve the credit assignment problem? If the agent does several actions and results in several different results, how does it know what actions contribute (and not contribute) to what result?

Author response:
**> When generating causal abstractions, how does the LLM resolve the credit assignment problem?**

Good question: By informed trial and error, using commonsense. When initially generating causal abstractions, the LLM is biased towards proposing commonsense abstractions vs. nonsensical abstractions (e.g., it is more likely to propose "opening the kitchen door may be necessary to move to the kitchen" than "looking around may be necessary to move to the kitchen"). Second, when the memory is updated, regenerating a new set of causal abstractions based on the old ones and new experiences, we rely on the LLM to follow its prompt-based instructions correctly, in particular, dropping memories that are unhelpful or simply wrong. [Madan et al., 2023](https://arxiv.org/pdf/2303.17651.pdf]) show LLMs are good at refining their outputs, and [Ma et al., 2023](https://eureka-research.github.io/assets/eureka_paper.pdf) show that LLMs are powerful reward generators. Mirroring these two insights, we find similar phenomena: for example, our LLM-based memory generator can connect an action (activate stove) and an observation (stove is broken) using its prior commonsense to propose a rule that activating stove when it is broken does not contribute to boiling the water. Of course, such proposals may sometimes be wrong, in which case they should later be dropped in subsequent trials when they hurt or are unhelpful. Please see **our new results** in the "[General Response to Reviewers](https://openreview.net/forum?id=d5DGVHMdsC&noteId=YGdT6et2uW)" (also Appendix C, Table 2 in the revised paper) on evidence that CLIN is able to recover from wrong knowledge in its memory via continual learning.

Reviewer comment (Theory):
Whether the causal abstractions generation depends on LLM's known commonsense knowledge about the world, or depends on its own reasoning ability. What happens if the world follows some new logic, and does the causal abstraction generation still be correct?

Author response:
**> What happens if the world follows some new logic, and does the causal abstraction generation still be correct?**

This would be problematic! CLIN avoids thousands of trials (e.g., in RL) by leveraging the LLM's commonsense, so in a "nonsensical" world, this would not be effective. In fact, this is the key "breakthrough" insight of language-based agents: that with modern LLMs, commonsense can replace blind trial-and-error in real-world-like environments.

Reviewer comment (Theory):
If the environment dynamics change and old causal relationships are wrong, does the agent know when to update and unlearn the original knowledge?

Author response:
**> If the environment dynamics change and old causal relationships are wrong, does the agent know when to update and unlearn the original knowledge?**

Yes - the memory is updated after each trial, and as causal insights learned from past experiences no longer apply, or are found now to be incorrect, the LM will adapt or drop them based on the information provided in the latest trial. For example, CLIN learned that "activating the stove is necessary to boil a substance" but then encountered a trial with a broken stove. It subsequently updated that rule in the memory update to "activating the stove does not contribute to boiling a substance when the stove is broken." Again, this update process is not perfect, but our experiments show the memory dynamically evolves to a high-performing result, an exciting and novel finding.