Paper: Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control (ICLR-2024)

Reviewer comment (Experiments):
From the results in the MindAct task it seems that adding any trajectories helps but it does not matter which ones.

Author response:
**A1.** It seems that there are some misunderstandings in our Mind2Web results. Let us elaborate on the results here.

- In `SYNAPSE w/ state abstraction`, we simply adopt state abstraction principles to direct generation (with fewer top-ranked elements in clean observations), which outperforms the MCQ-formatted prompting used in MindAct. Here, we use the same exemplars as those used in the MindAct, but the format is changed from selecting an answer from multiple choices to directly generating the action, and we do not recursively select one answer from five candidates until only one is left in the top-50 elements. Instead, we directly generate the action based on abstracted states with the top-5 elements.
- The `SYNAPSE w/ state abstraction + TaE` variant integrates TaE prompting on top of state abstraction, where we replace the original prompting format with trajectories. The results show the benefits of using trajectories as exemplars. In both these two variants, we use **static** few-shot exemplars, i.e., the exemplars are the same for all tasks during evaluation. This is the standard paradigm of few-shot learning, where we provide examples of input and output for LLMs. The performance lift does not result from "adding any trajectories". It is due to the change in the exemplar format.
- In `SYNAPSE w/ state abstraction + TaE + memory`, we encode the training set as the memory and apply similarity search to match trajectories from similar tasks, which continues to improve performance. This also showcases that adding relevant trajectories as exemplars can be beneficial compared to "adding any trajectories".

We have updated the caption of Table 1 to clarify these details.

Reviewer comment (Experiments):
Have you studied the effect on performance on the MiniWob++ environment for different amount of tasks in the memory. The comparison to RCI does not seem fair at the moment (Figure 3)?

Author response:
**A2.** It is noteworthy that RCI solves 54 tasks with exemplars from 54 tasks, while Synapse solves 64 tasks with exemplars from 48 tasks, including challenging ones that RCI cannot solve, such as book-flight. The average number of exemplars per task for RCI and Synapse is 1.32 and 3.45, respectively. However, the number of exemplars for each task varies. For simple tasks, we use a similar number of exemplars as in RCI. To solve complex tasks (e.g., book-flight), we use more exemplars, which lowers the average data efficiency. For example, in book-flight, we include 5 exemplars for better robustness. Moreover, RCI relies on self-correction and our method does not. They proposed several self-correction modules (task/state/agent grounding), and their success rate will decrease by around 50% without either one of them, according to their paper [1]. Also, one of the benefits of our method is that we can leverage more exemplars in-context because of state abstraction.

Reviewer comment (Reproducibility):
It would be helpful to have an illustrating example of the result of the state abstraction procedure (i.e. a figure of a (state,observation)-pair) as well as an example prompt (maybe in the Appendix). Figure 1 only shows that some <body> and <h1> tag got removed but it's not yet clear to me what the result of the state abstraction procedure looks like. Its also not clear to me where the examples from the state abstraction prompt come from. Are they designed a-priori? If yes how and how difficult is that? Are the same examples reused for all states or also taken from the memory?

Author response:
**A3.** We have reorganized Sec. 3.2 to clarify the state abstraction procedure and included more examples in Appendix C for illustrations. Fig. 1 aims to show that raw states can be very long-winded, which we omitted in  `<h1>…` , while clean observations are much more concise with only task-relevant information, like `<input id="where" type="text">`.

In MiniWoB++, there are two types of state abstraction prompts:

- For scenarios where the context can handle multiple states, we utilize state-observation pairs ⟨state, observation⟩. These pairs are collected alongside the trajectories. The states are obtained from the environment, while the cleaned observations are provided by humans. To further reduce human efforts, we can also automatically infer cleaned observations given trajectories using LLMs [2].

- In scenarios with complex states where explicit abstraction is not applicable, we implicitly abstract states by pairing task descriptions and state-parsing code, denoted as ⟨task, code⟩. Similarly, the task descriptions are already given by the environment. The code here is collected via zero-shot sampling from LLMs (GPT-4) plus validation. This data collection pipeline can also be augmented with LLMs to reduce human effort.

The state abstraction prompts are retrieved from the memory, together with action generation prompts (trajectories).

In fact, one of the advantages of trajectory-as-exemplars over other methods is that it is simple and scalable to convert human demonstrations into exemplars (as they are already trajectories), without much human effort in designing promptings like multi-choice question-answering or high-level semantic plans in natural language.

Reviewer comment (Presentation):
The text states (last paragraph of 4.3) that failed trajectories are displayed in Appendix C, but Appendix C only refers to the supplementary material?

Author response:
**A4.** We have added trajectories of both successful (terminal and book-flight) and failed (text-transform and Mind2Web) cases in Appendix C.

Reviewer comment (Experiments):
In the original paper of MindAct, they used 3 in-context exemplars for GPT3.5 but Synapse uses 5, so the comparison favours Synapse. In general I would maybe add a sentence that clarifies that results are SOTA, conditioned on the fact that GPT3.5 was used as the underlying LLM.

Author response:
**A5.** The comparisons are fair because Synapse also uses three in-context exemplars, the same number of exemplars as MindAct. The $k=5$ refers to the number of elements after state abstraction. The reviewer's misunderstanding may come from our abuse of the notation $k$ for both top-$k$ memory retrieval and top-$k$ state abstraction. We have updated the notation for memory to top-$n$ to avoid misunderstanding.

To investigate how Synapse performs on other language models, we added the Mind2Web experiments on CodeLlama-7B in Table 1, where Synapse outperforms MindAct with a 2.5x average step success rate across three levels of generalization (cross-task, cross-website, and cross-domain). These results are consistent with the results with GPT-3.5 as the underlying LLM.

Reviewer comment (Experiments):
Have you considered building up the memory from scratch by adding successful trajectories over the course of training?

Author response:
**A6.** Thank you for this suggestion. It would be interesting to explore this aspect. However, in this work, we mainly investigate the design choices around few-shot exemplars, including state abstraction, trajectory-as-exemplar, and exemplar memory. Therefore, we initialize the memory with existing exemplars and fix it during experiments. In fact, Synapse complements existing methods that build up memory during the agent's interaction with the environment [3]. Specifically, our method can guide the collection of exemplars and the format to store them. It is definitely interesting to integrate trajectory exemplars into open-ended learning agents. Another direction for future work is to query human intervention and add their demonstrations into the memory when the similarity score is low, i.e., the agent is faced with unfamiliar situations.

[1] Kim et al. "Language models can solve computer tasks." NeurIPS 2023.

[2] He et al. "Annollm: Making large language models to be better crowdsourced annotators." *arXiv preprint arXiv:2303.16854* (2023).

[3] Wang et al. "Voyager: An open-ended embodied agent with large language models." arXiv preprint arXiv:2305.16291 (2023).

Reviewer comment (Experiments):
SYNAPSE uses state abstraction to make sure LLM can learn from compact information while ensuring the context length is well utilized. However, in the experiments, SYNAPSE sets k=5 for retrieval. How does the performance change if k grows? And why longer context is not helping here?

Author response:
**A1.** As suggested by the reviewer, we have added Fig. 7 in the appendix to ablate the performance with different $k$ (the number of elements in the observation after abstraction). The performance drops as $k$ grows, even though the recall@$k$ becomes higher (the probability of the correct element appearing in the observation). There are two possible explanations: i) more irrelevant information poses challenges in selecting correct ones [1,2], ii) current LLMs struggle to handle long context, e.g., the performance gets worse as the input context grows longer [3].

Reviewer comment (Experiments):
For Table 1 training set as memory experiments, do you add all the training data for 3 scenarios ("cross-task, cross-web, and cross-domain)? Or only the corresponding training data (e.g. cross-task but same web when evaluated on cross-task setting).

Author response:
**A3.** The Mind2Web dataset is split into a training set and three test sets: Cross-Task, Cross-Website, and Cross-Domain, to evaluate generalizability over tasks from the same websites, unseen websites from similar domains, and completely unseen domains in the training set, respectively. We only store the training set in the memory. All three cross-* test sets are not stored in the memory. We have added these details in Sec. 4.1, and provided detailed descriptions of these dataset splits in Appendix B.2.

Reviewer comment (Evaluation):
For tasks that need code to abstract state information, how can you examine if a code can successfully extract desired information? If examined from task success, it can be because of the suboptimal execution of policy.

Author response:
**A4.** If the code execution raises an error, we ask the LLM to perform zero-shot  
state abstraction, i.e., directly abstracting state information without any exemplars. For simplicity, we did not introduce any success detection or self-correction to examine what components caused the failure, which is not our core contribution. When the task fails, we directly stop and mark the task as a failure. This detail has been updated in Sec. 3.2.

Reviewer comment (Evaluation):
For the cross-domain tasks, retrieval doesn't seem to help, can you give a better explanation about this result? The retrieval number is set to 5, which is a small number. Is it because the retriever doesn't return the desired examples? Maybe showing some retrieved examples can be more intuitive.

Author response:
**A5.** After carefully examining the results, we found that there is a minor calculation error. The results have been updated in Table 1. However, the conclusion that memory does not help much in cross-domain tasks still holds. We also added experiments on CodeLlama-7B to validate the conclusion. We have added Table 2 in Sec. 4.5 to visualize the distance between retrieved exemplars and the tasks for cross-task, cross-website, and cross-domain test sets. As shown in Table 2, as the distance becomes larger, i.e., less similarity, the performance gain ($\Delta$Step SR) becomes worse. As discussed in Sec. 4.5, this might be attributed to the LLM being influenced by exemplars from completely unrelated domains.  

The number of few-shot exemplars is actually 3, the same as used in the baseline. We noticed that we abused the notation $k$ for both the top-$k$ parameter in memory retrieval and the top-$k$ parameter in Mind2Web state abstraction, and we have updated the notation for memory retrieval to top-$n$ to avoid misunderstanding. Intuitively, deciding the number of exemplars is a tradeoff between performance and context limits. Adding more exemplars can lead to better performance, but it also uses more tokens. As discussed in A1, current LLMs have limited context and struggle to perform well when the context is long. Three-shot learning is a suitable balance, ensuring a fair comparison with the baseline and taking into account both sides of the issue.

Reviewer comment (Writing):
"The action generation starts from prompting LLMs with successful trajectories to warm up LLMs with the dynamics of the current environment". Is it describing prompting with retrieved trajectories? Or that's some other warmup procedure? If it's cross-task/cross-web/cross-domain setting, how's current environment defined?

Author response:
**A6.** We only prompt with retrieved exemplary trajectories. There is no other warmup procedure. The environment here refers to the observation and action space. In cross-task/cross-website/cross-domain settings, these spaces remain consistent. To avoid misunderstanding, we have deleted this expression from the paper.

Reviewer comment (Presentation):
The same ideas (gaps in prior work, the three components and their descriptions, the benchmarks being used) are repeated several times throughout the paper. While some repetition is crucial to help a reader understand, in this case it is not just two or three times but four or five, or even if it's three, there is lots of detail every time. It feels as though the repetition is there to fill up space, even if that's not the case, and it gets tiring to read the same high-level concepts repeatedly.

Author response:
**A1.** As suggested by the reviewer, we have improved the presentation in the revised version to reduce repetition and include more details about our methods and experiments in response to the questions from reviewers.

Reviewer comment (Evaluation):
I don't quite understand what "step success rate" is as opposed to "success rate" — it is brought up several times as the metric for Mind2Act, but is not defined as far as I am aware.

Author response:
**A2.1.** In Mind2Web, each step of a task is evaluated separately against the human-annotated ground truth. The step success rate (Step SR) relates to how well the action at each step is performed, whereas the success rate (SR) reflects the success of the entire task, also known as the episode success rate. For example, to complete the task "Show me the reviews for the nearest auto repair business.", the agent has to execute several steps, including typing the search query and clicking through to find the reviews. If all steps in the task are successful, then the task is considered successful. The definition of these metrics has been added in Sec. 4.1.

Reviewer comment (Writing):
It doesn't seem to be made clear what exactly the state abstraction entails. In figure 1, the only difference between "raw state" and "clean state" is `<body><h1>…</h1></body>” --> <input id=”where” type=”text”>`. There aren't clear examples in the main text, but intuition about this would be very useful.

Author response:
**A2.2.** We have reorganized Sec. 3.2 to clarify the state abstraction procedure and included more examples in Appendix C for illustrations. Fig. 1 aims to show that raw states can be very long-winded, which we omitted in  `<h1>…` , while clean observations are much more concise with only task-relevant information, like `<input id="where" type="text">`.

Reviewer comment (Novelty):
It is not clear how SYNAPSE's history-in-prompt approach is significantly different from Mind2Act's (or many other works') to the point of being novel.

Author response:
**A2.3.** Directly providing trajectories (with multiple states-action pairs) results in very long contexts, as a single webpage state is already very complex. However, current LLMs can only process limited context, and the performance becomes worse as the context becomes longer. As a result, existing computer agents such as RCI and MindAct do not and **CANNOT** utilize **trajectories** as historical information. Instead, their history only consists of **actions**. In comparison, our state abstraction shrinks the length of each state, which is a prerequisite for trajectory-as-exemplar (TaE) prompting.

It is clear that TaE prompting **provides more information** for decision-making. More importantly, TaE prompting uses an interleaved observation-action format, which offers the following advantages over previous prompting schemes.

- **Temporal abstraction**. The interleaved prompt scheme implicitly prompts the LLM to generate multiple actions and query new states only when necessary, as shown in Fig. 2. This significantly improves the success rate of long-horizon tasks in MiniWoB++. In contrast, RCI and MindAct cannot prompt LLMs to generate temporally abstracted actions.
- **Lower cost and latency** (compared to MindAct). MindAct introduces accumulated error and much higher cost and latency. Specifically, it first filters the top-50 elements with a ranking model. After that, it recursively samples five elements and prompts the LLM to select one from the five until only one is left. As a result, MindAct uses an average of 10x more tokens compared to Synapse in the three test sets of Mind2Web.
- **Simplified action parsing**. TaE prompting is more effective in prompting LLMs to generate actions in a specific format, which allows easily parsing actions from LLM responses, leading to fewer action parsing errors than other prompting approaches. Additionally, we can set the stop token to "Observation" when querying actions so that the LLM does not generate other information.
- **Simplified data collection**. Compared to high-level semantic plans or multi-choice questions that are human-crafted in natural language, our approach makes data collection simpler and scalable. The exemplary trajectories can be converted directly from human demonstrations, and the state abstraction prompts can also be inferred from the action with minimal human effort.

In short, our approach is straightforward and very effective, rendering it universally applicable to current agents without much effort.

Reviewer comment (Presentation):
From both Fig 1 and the text, it's unclear when new trajectory exemplars and example state abstractions are generated — is it per task (as I'd expect) or per step?

Author response:
**A2.4.** The trajectory exemplars are used per task. We explained the pipeline of Synapse at the beginning of Sec. 3. Given a task, Synapse retrieves trajectories of similar tasks from memory, which will be fixed during the task execution. We have also added some illustrative trajectories in Appendix C.

Reviewer comment (Presentation):
In Fig 3, it might help to have a clearer visual distinction between SYNAPSE and the human baseline, especially since the colors aren't always visible. Lines, shading, or callout boxes could help.

Author response:
**A2.5.** Although we agree with the reviewer, it is difficult to adjust the plot. The box plot of Synapse is almost visible because our method is robust and has relatively low variance.

Reviewer comment (Novelty):
The paper spends considerable space discussing how it is unique compared to other prompting-based methods in helping the LLM understand the current state. However, MindAct gives trajectory-so-far back to the LLM when prompting for the next step. SYNAPSE gives observation as well, thereby informing the LLM in more detail about the current state. This is not a very original idea — it is an important step past MindAct, but not a big one, especially relative to how much it is discussed as a unique contribution.

Author response:
**A3.2.** While we appreciate the reviewer’s feedback, we respectfully have a different view. We believe our approach is a unique and original contribution to the field due to the reasons discussed in **A2.3**.

Reviewer comment (Writing):
Page 5, last paragraph: `Obseravtion` is a misspelling.

Author response:
**A4.** Thank you for pointing this out. This typo has been corrected.

Reviewer comment (Evaluation):
What exactly is step success rate vs. success rate?

Author response:
**A2.1.** In Mind2Web, each step of a task is evaluated separately against the human-annotated ground truth. The step success rate (Step SR) relates to how well the action at each step is performed, whereas the success rate (SR) reflects the success of the entire task, also known as the episode success rate. For example, to complete the task "Show me the reviews for the nearest auto repair business.", the agent has to execute several steps, including typing the search query and clicking through to find the reviews. If all steps in the task are successful, then the task is considered successful. The definition of these metrics has been added in Sec. 4.1.

Reviewer comment (Evaluation):
It is inherently interesting that this approach does not need self-correction unlike other prompting-based approaches, but does that offer any significant performance/efficiency gain?

Author response:
**A5.** Our non-self-correction approach outperforms self-correction baselines, which means that ours has better robustness and lower latency. In fact, Synapse is orthogonal to self-correction methods. We did not add self-correction because it is not our core contribution, and our approach already achieves SOTA performance without it. In contrast, RCI proposed several self-correction modules (task/state/agent grounding), and their average success rate will decrease by around 50% without either one of them, according to their paper [1].