Paper: You Only Look at Screens: Multimodal Chain-of-Action Agents (ICLR-2024)

Reviewer comment (Theory):
While the authors highlight the chain-of-action technique as a contribution, it appears to primarily concatenate the output actions, which can be confusing. It would be helpful to provide a more detailed explanation or clarification of how the chain-of-action technique enhances the decision-making process and contributes to the overall effectiveness of the Auto-UI approach.

Author response:
Chain-of-action (CoA) technique contributes two aspects instead of purely concatenating the output actions. It organizes a chain of previous action histories on the input side and a chain of future action plans on the output side, which encourages the agent to leverage action history and make future plans before predicting the current action. The working mechanism can be seen as bidirectional. 

CoA helps improve the decision-making process and contributes to the overall effectiveness by +5.74% accuracy. To investigate the effectiveness, we conduct additional analysis to ablate the CoA technique on category accuracy.

| Model                     | Overall | Action Type  | Click        | Scroll       | Text |
| ------------------------- | ------- | ------- | ------- | ------- | ------- |
| w/o Chain of Actions | 58.99 | 81.69 | 48.38 | 75.83 | 93.57 |
| w/ Chain of Actions | 68.24  (+9.25) | 87.03  (+5.34) | 58.34 (+9.96) | 82.74  (+6.91)  | 93.99 (+0.42) |

The results show that CoA helps improve the clicking and scrolling accuracy by a large margin (9.96 and 6.91), e.g., clicking accurate positions and scrolling in the correct direction, compared with the action type accuracy and text accuracy. The most possible reason is that CoA provides action history and future plans as the context for performing more accurate executions given that the model has already been effective in predicting the action type (over 81.69%).

Reviewer comment (Experiments):
The experiment section lacks an explanation for the rationale behind selecting specific baselines. It would be valuable to include a justification for choosing the particular baselines used in the evaluation. Additionally, providing information on the performance of a GPT4 model, if available, would offer a useful benchmark to compare the performance of the proposed Auto-UI approach.

Author response:
Yes. We have added a justification for the choice of the baselines in Section 4.2. The specialized UI agent is selected because it is the previous state-of-the-art approach in existing studies. The in-context learning LLMs are selected because they reflect the widely used paradigm of developing LLM agents. Fine-tuned LLMs are adopted to investigate the potential of leveraging open-source LLMs for our problem. The baselines encompass both the In-context Learning and fine-tuning paradigms, along with various backbone models of different sizes. These baselines were carefully chosen to ensure a thorough evaluation of our work's performance against existing approaches.

Following the reviewer’s comment, we have added the performance of the GPT-4V model. The main results are shown below.

| Model                     | Overall | General  |  Install  | GoogleApps  | Single  |
| ------------------------- | ------- | ------- | ------- | ------- | ------- |
| PaLM 2-CoT|  39.6 | -  |  - |  - |   -  | 
| ChatGPT  | 7.72  | 5.93  | 4.38  | 10.47  |  9.39  |  8.42  |
| GPT-4V  | 52.96   | 43.01   | 46.14   | 49.18      | 78.29  |
| Auto-UI | 74.27   | 68.24   | 76.89   | 71.37      | 84.58  |

The results show that GPT-4V achieves much better performance than the PaLM and ChatGPT models. Auto-UI is still better than GPT-4V.

Reviewer comment (Reproducibility):
The paper does not describe much about the actual training details, in that sense, to me, the proposed method is still a kind of BC, where the target decoding is optimized towards mimicking the golden action sequences. (Unless some RL or other mechanism is used here, which is not described.) In my opinion, the novelties here mainly lie in the multimodal representations (both modality taken into account) and the format of the action performed.

Author response:
> W1: Training details & Novelty.

The training details are provided in Section 4.4. We agree that our approach is still a kind of BC. However, the novelty of Auto-UI lies within (i) a multimodal agent for autonomous UI control that can directly interact with the screens, thus circumventing the constraints of environment parsing and application-specific API access, and (ii) a chain-of-action technique that leverages the previously executed actions and future action plans to help the agent decide what action to execute at each step. Therefore, our work represents a significant departure from prior studies in the field. In addition, Auto-UI achieves state-of-the-art performance with an action type prediction accuracy of 90% and an action success rate of 74%. Notably, Auto-UI can infer an action as fast as within less than one second.

Reviewer comment (Experiments):
I’m a bit skeptical about the ICL baseline, first of all more details (e.g., how actions are represented, how OCRed results are used) of that baseline need to be described, at least in the appendix. Secondly, it also needs to be evaluated at the action plan level, my guess is that this method should be quite accurate on those but might fail more on the lower-level executions. Thirdly, it is indeed unfair simply because the model is not taking the images into account, which could be the key towards the success of the proposed method in this work. So, at least a multimodal version of it needs to be taken into consideration, or, a better spatial representation of the html syntax is required. (HTML can be many times too coarse to represent a spatial layout.)

Author response:
> W2: About the ICL baseline.

Firstly, the ICL baseline is implemented following the prior study [1]. To help readers understand the details, we have exactly provided examples of the ICL baseline in Appendix A.3 with represented actions and OCRed results. 

Secondly, we have conducted additional analysis on the action plan level. The result is shown below.

| Model                     | Overall | Action Type  | Click        | Scroll       |
| ------------------------- | ------- | ------- | ------- | ------- |
| ChatGPT | 5.93    | 41.72       | 8.50  | 4.00   |
| Auto-UI | 68.24   | 87.03       | 58.34 | 82.74  |

Indeed, we see that the ICL method (ChatGPT) is quite accurate at predicting the action type (41.72%) but fails at lower-level executions, e.g., clicking positions (8.5%) and scrolling directions (4.0%).

Thirdly, this is exactly the motivation of our work. Previous work commonly uses HTML syntax while we argue that parsing the visual environment into textual elements may be prone to error propagation or information loss and the parsed elements generate lengthy inputs, thus leading to inference inefficiency. Following the reviewer’s comment, we add the results of a strong multimodal ICL baseline (GPT-4V). The results show that our proposed approach is still much better than the baseline.

| Model                     | Overall | General  |  Install  | GoogleApps  | Single  |
| ------------------------- | ------- | ------- | ------- | ------- | ------- |
| GPT-4V  | 52.96   | 43.01   | 46.14   | 49.18      | 78.29  |
| Auto-UI | 74.27   | 68.24   | 76.89   | 71.37      | 84.58  |

Reviewer comment (Experiments):
Similar to above, the third baseline, fine-tuning LLMs, need to have a version with multimodal inputs.

Author response:
> W3: “Similar to above, the third baseline, fine-tuning LLMs, need to have a version with multimodal inputs.”

Yes. We have added the results of fine-tuned LLMs with multimodal inputs (LLaVA) [1]. This can be seen as a variant of our framework by changing the backbone modules. We see that LLaVA is able to achieve slightly better performance gains and our proposed chain-of-action technique is also effective at enhancing the performance further.

| Model                     | General  |
| ------------------------- | ------- |
| T5 backbone            | 68.24 |
| LLaVA  backbone      | 68.70   |
| w/o chain of actions | 58.40   |

Reviewer comment (Evaluation):
An error analysis is required both on the quantitative and qualitative sides, what are the major errors that these models exhibit?

Author response:
> W4 & Q2: “An error analysis is required both on the quantitative and qualitative sides, what are the major errors that these models exhibit?”

We have conducted error analysis in Section 5.1 and find that the major errors lie within the click region and scroll direction predictions. Although the model is able to predict the right action most of the time, it tends to click the wrong place or scroll in the wrong direction. The result reveals a future direction of improving the model’s ability to understand the screen layouts, e.g., using more advanced vision features.

Reviewer comment (Evaluation):
What are the main types of errors observed by the proposed framework? And, does the framework provide good insights on how to assign these errors to specific modules? I.e., where should the improvements be?

Author response:
> W4 & Q2: “An error analysis is required both on the quantitative and qualitative sides, what are the major errors that these models exhibit?”

We have conducted error analysis in Section 5.1 and find that the major errors lie within the click region and scroll direction predictions. Although the model is able to predict the right action most of the time, it tends to click the wrong place or scroll in the wrong direction. The result reveals a future direction of improving the model’s ability to understand the screen layouts, e.g., using more advanced vision features.

Reviewer comment (Presentation):
The Figure 1 in this paper is somewhat not clear enough, making it difficult to understand the two paradigms in (a) and (b).

Author response:
We provide a detailed explanation of the two paradigms from the third and fifth paragraphs in the introduction. We elaborate on it as follows:

*(a) sandbox paradigm*: depends on the intermediate transformation between environments and agents

- input side: relies on external tools such as optical character recognition (OCR) and icon detectors to parse the environment into textual elements 

- output side: requires accessing internal APIs to interact with the environment, e.g., using a JavaScript element selection on a webpage or a Python interpreter to execute actions. 

*(b) first principles thinking paradigm*: allows direct interactions on the screen without needing access to intermediate environment parsing or interval application-dependent APIs.

Reviewer comment (Writing):
The author does not provide a specific explanation of the Sandbox Paradigm and the First Principles Thinking Paradigm, which is confused.

Author response:
We provide a detailed explanation of the two paradigms from the third and fifth paragraphs in the introduction. We elaborate on it as follows:

*(a) sandbox paradigm*: depends on the intermediate transformation between environments and agents

- input side: relies on external tools such as optical character recognition (OCR) and icon detectors to parse the environment into textual elements 

- output side: requires accessing internal APIs to interact with the environment, e.g., using a JavaScript element selection on a webpage or a Python interpreter to execute actions. 

*(b) first principles thinking paradigm*: allows direct interactions on the screen without needing access to intermediate environment parsing or interval application-dependent APIs.

Reviewer comment (Writing):
The authors don't explain exactly what touch_point, lift_point, etc. mean in the first place, causing some confusion.

Author response:
Touch_point and lift_point simply mean the axis of touching and lifting. As the notation is widely accepted in related studies and we believe it is also commonsense, we provide examples when discussing coordinate normalization in Section 3.3. Even so, following the reviewer’s comment, we have added an explanation in the first place (Figure 1).

Reviewer comment (Experiments):
The authors do not provide a specific example between Auto UI and other baselines in Section 5, which is not clear to understand the effectiveness of the provided Auto UI.

Author response:
Section 5 is about the analysis of our approach, which follows the same model architecture. We believe the reviewer means Section 4 (please correct us if it is wrong). We have provided a detailed description of the baselines in Section 4.2. Actually, the example of the specialized UI agent and fine-tuned LLMs is illustrated in Figure 1 while the example of the in-context learning LLMs is provided in Appendix A3.

Reviewer comment (Evaluation):
In Section 4.3, why do you use 14% instead of other number to evaluate the correction of a click action, could you provide some references?

Author response:
The choice basically follows the previous study [1] which releases the dataset and defines the evaluation criteria. We follow the setting for a fair comparison.

[1] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A large-scale dataset for Android device control. arXiv preprint arXiv:2307.10088, 2023.

Reviewer comment (Experiments):
A potential weakness is where is the gain from? It looks PaLM and ChatGPT are pretty low on this dataset, while they only take text input, and BC models and Auto-UI models take image screen as input, and get very high results, it is unclear where is the gain from? image encoder? or a chain of action input?

Author response:
Thanks for your insightful review and constructive feedback.

> W & Q: Where is the gain from?

The performance gain mainly comes from two aspects, multimodal perception and chain-of-action technique. Following the reviewer’s suggestion, we have conducted additional experiments to ablate the performance of the model without the image encoder (w/o multimodal perception) but use the parsed HTML layout instead.

| Model                     | Overall |
| ------------------------- | ------- |
| Auto-UI                   | 74.27   |
| w/o multimodal perception |   62.70      |
| w/o chain of actions      | 68.53   |

The key takeaways are: 

(i) Multimodal perception is critical compared with existing baseline language agents built purely on large language models. The result shows that our model out of first principles thinking can serve as a strong autonomous agent.

(ii) The chain-of-action technique helps improve the ability of our multimodal agent further, by leveraging the previously executed actions and future action plans to help the agent decide what action to execute at each step. As a result, the memory and planning ability can be enhanced.

It is reasonable that the baseline results are low due to the lack of vision encoders. As a result, they may suffer from error propagation or information loss (as discussed in the introduction section). To have a deeper understanding of the performance, we have conducted additional analysis on the performance regarding the action type and action execution (i.e., clicking accurate positions and scrolling in the correct direction). 

| Model                     | Overall | Action Type  | Click        | Scroll       |
| ------------------------- | ------- | ------- | ------- | ------- |
| ChatGPT | 5.93    | 41.72       | 8.50  | 4.00   |
| Auto-UI | 68.24   | 87.03       | 58.34 | 82.74  |

We find that the ICL method (ChatGPT) is quite accurate at predicting the action type (41.72%) but fails at lower-level executions, e.g., clicking positions (8.5%) and scrolling directions (4.0%).

Reviewer comment (Experiments):
I try to understand the setting of the experiments, and why the strong PaLM and ChatGPT baselines are so low. Based on the main Table 2, it looks the most gain is from the image encoder, right? Since PaLM-CoT and ChatGPT-CoT only take text input, and their performance is pretty low, and also similarly for Llama 2.

Author response:
Thanks for your insightful review and constructive feedback.

> W & Q: Where is the gain from?

The performance gain mainly comes from two aspects, multimodal perception and chain-of-action technique. Following the reviewer’s suggestion, we have conducted additional experiments to ablate the performance of the model without the image encoder (w/o multimodal perception) but use the parsed HTML layout instead.

| Model                     | Overall |
| ------------------------- | ------- |
| Auto-UI                   | 74.27   |
| w/o multimodal perception |   62.70      |
| w/o chain of actions      | 68.53   |

The key takeaways are: 

(i) Multimodal perception is critical compared with existing baseline language agents built purely on large language models. The result shows that our model out of first principles thinking can serve as a strong autonomous agent.

(ii) The chain-of-action technique helps improve the ability of our multimodal agent further, by leveraging the previously executed actions and future action plans to help the agent decide what action to execute at each step. As a result, the memory and planning ability can be enhanced.

It is reasonable that the baseline results are low due to the lack of vision encoders. As a result, they may suffer from error propagation or information loss (as discussed in the introduction section). To have a deeper understanding of the performance, we have conducted additional analysis on the performance regarding the action type and action execution (i.e., clicking accurate positions and scrolling in the correct direction). 

| Model                     | Overall | Action Type  | Click        | Scroll       |
| ------------------------- | ------- | ------- | ------- | ------- |
| ChatGPT | 5.93    | 41.72       | 8.50  | 4.00   |
| Auto-UI | 68.24   | 87.03       | 58.34 | 82.74  |

We find that the ICL method (ChatGPT) is quite accurate at predicting the action type (41.72%) but fails at lower-level executions, e.g., clicking positions (8.5%) and scrolling directions (4.0%).

Reviewer comment (Experiments):
Probably needs a baseline/ablation to see the performance of model without image encoder.

Author response:
Thanks for your insightful review and constructive feedback.

> W & Q: Where is the gain from?

The performance gain mainly comes from two aspects, multimodal perception and chain-of-action technique. Following the reviewer’s suggestion, we have conducted additional experiments to ablate the performance of the model without the image encoder (w/o multimodal perception) but use the parsed HTML layout instead.

| Model                     | Overall |
| ------------------------- | ------- |
| Auto-UI                   | 74.27   |
| w/o multimodal perception |   62.70      |
| w/o chain of actions      | 68.53   |

The key takeaways are: 

(i) Multimodal perception is critical compared with existing baseline language agents built purely on large language models. The result shows that our model out of first principles thinking can serve as a strong autonomous agent.

(ii) The chain-of-action technique helps improve the ability of our multimodal agent further, by leveraging the previously executed actions and future action plans to help the agent decide what action to execute at each step. As a result, the memory and planning ability can be enhanced.

It is reasonable that the baseline results are low due to the lack of vision encoders. As a result, they may suffer from error propagation or information loss (as discussed in the introduction section). To have a deeper understanding of the performance, we have conducted additional analysis on the performance regarding the action type and action execution (i.e., clicking accurate positions and scrolling in the correct direction). 

| Model                     | Overall | Action Type  | Click        | Scroll       |
| ------------------------- | ------- | ------- | ------- | ------- |
| ChatGPT | 5.93    | 41.72       | 8.50  | 4.00   |
| Auto-UI | 68.24   | 87.03       | 58.34 | 82.74  |

We find that the ICL method (ChatGPT) is quite accurate at predicting the action type (41.72%) but fails at lower-level executions, e.g., clicking positions (8.5%) and scrolling directions (4.0%).