Paper: Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation API (ICLR-2024)

Reviewer comment (Reproducibility):
The work is a bit over-claimed by stating that the framework does not require any additional information, while it indeed still requires the ground truth bounding boxes of the elements.

Author response:
**Q1: Is it a bit over-claimed to state that the framework does not require any additional information?**  
**A1:** Actually, it's not. This statement refers to that our framework does not require any additional information ***during inference***. We still require the ground truth bounding boxes of UI elements as the supervisions for training. In contrast, previous work require such information ***in both training and inference stages*** since they take this information as one of the model inputs, thus limiting the practicability especially when the additional information is not available or noisy. Thanks for your question, we will further clarify this in our revision.

Reviewer comment (Experiments):
More details on the baselines of existing grounding modules, such as GroundingDINO [1] and GLIP [2], are needed. I.e., how the inputs are handled, how the tasks are adapted for their settings, training details, etc. I’m a bit skeptical about the results being that much lower than the proposed module where after all they are all using the same amount of output supervisions for training. (Afterall, the author proposed method is also not benefiting much from non-provided metadata, such as OCRs, so that explanation is not convincing.) These grounding models may have been simply under-tuned.

Author response:
**Q2: Does the low performance of traditional grounding methods (such as Grounding DINO [1] and GLIP [2]) on our targeted UI tasks result from them being simply under-tuned?**  
**A2:** We believe that it's not due to this reason, but because ***their design purposes do not align with our targeted UI tasks***. Actually, we have kept the data and training configurations for these traditional grounding methods with ours to make their comparison as fair as possible. The root of their performance gaps on our targeted UI tasks lies in their different pre-training designs. The effectiveness of these traditional methods designed for object detection can be attributed to their pre-training on large-scale ***object-level*** image-text pairs ***in the form of contrastive learning***, wherein obeject-corresponded bounding boxes are required. Two points need to be known: 
1) We experimentally find that contrastive learning based pre-training does not perform well on the images densely populated with texts, e.g., documents, screenshots, for two reasons. One is that the elements (analogous to objects) in such images are too dense, making contrastive learning extremely difficult. The other is that different elements may contain one or more the same wolds, leading to a certain degree of ambiguity in the discrimination of positive and negative samples.  
2) The document data used for pre-training our model is not applicable for conducting contrastive learning, due to the lack of element-wise (object-level) text annotations. The annotations of our pre-training document data correspond to all texts on the entire page, arranged in a reading order (i.e., at the page-level), without element-wise bounding boxes provided. Details can be found in this [open-source repo](https://github.com/clovaai/donut).    

Considering these, we think Pix2Seq framework is more suitable for visual grounding on the images densely populated with texts. Thus, our contributions lie not only in applying Pix2Seq to this task, but also in addressing the limitations in the training of the Pix2Seq framework. We will provide more in-depth analysis and insights in our revision.

Reviewer comment (Experiments):
Following up on the previous point, the proposed method adopts pre-trained weights from document understanding tasks that the conventional grounding modules do not have access to. A more fair comparison is to pre-train these grounding modules at least with these document understanding data (perhaps only regressing the boxes of texts and/or contents, the actual text recognition is not so important), as their pre-training domains are far from these structural textual contents.

Author response:
**Q3: About pre-training traditional grounding methods on document data.**  
**A3:** This is actually not feasible because of the second point detailed in our A2 above. Please note that we have pre-trained them using our UI data. But the reaults were not satisfactory due to the first point analyzed in A2 above.

Reviewer comment (Experiments):
At a first glance, I thought the model can really omit any metadata for supervisions, including even the ground truth boxes. That is to say, this may be the advantage of web UI problems as the predicted boxes may not need to be 100% correct but at least “touch” the ground truth ones (and then in reality it should be clickable). In this sense, one can really design an RL framework that relies solely on “successfully clicked” the right place in the UI as the reward function, and this should be much easier to collect from the data curation point of view. Why not consider this setting as an upgraded version of the framework, and really showcase the power of RL here?

Author response:
**Q5: Why not design an RL framework that relies solely on "successfully clicked" the right place in the UI as the reward function?**  
**A5:** It seems much easier in terms of data collection at first glance, but it actually is not. Because this requires us to not only record user instructions and the clicked positions but also determine whether the click has been successful. In terms of labor costs, this is not any easier than our current practice of generating image-instruction pairs and extracting their corresponding ground truthes from the metadata of UIs. In fact, the offline extraction of metadata is cost-effective for developers. But we cannot guarantee that metadata can always be available online when used as a generic executor for practical UI task automation. This motivates us to employ the information in the metadata as supervisions to train a UI visual grounding model which does not need it during inference.

Reviewer comment (Experiments):
The two learning objectives in Eq.2 and Eq.3 are used in parallel, but another straightforward idea is to combine the two and scale the losses for coordinate tokens by rewards. Will this work better?

Author response:
**Q1: How about combining Eq.2 and Eq.3 and scaling the losses for coordinate tokens by rewards?**  
**A1:** Thanks for this insightful question. We conduct an experiment for this configuration (denoted by "ScaleLoss") and report the comparison results with ours as below.

| Models                      | App Seen (mIoU \| Acc) | App Unseen (mIoU \| Acc) | Web Seen (mIoU \| Acc) | Web Unseen (mIoU \| Acc) |
|:----------------------------|:---------------:|:---------------:|:---------------:|:---------------:|
| Baseline                    | 0.52 \| 72.23%  | 0.42 \| 65.03%  | 0.45 \| 48.69%  | 0.41 \| 46.46%  |
| ScaleLoss                   | 0.54 \| 73.87%  | 0.43 \| 66.59%  | 0.43 \| 47.78%  | 0.41 \| 47.09%  |
| RUIG (Ours)                 | 0.62 \| 81.16%  | 0.48 \| 73.92%  | 0.51 \| 61.78%  | 0.49 \| 59.03%  |

We can find this configuration delivers close performance with the baseline model, with slight improvements sometimes. It does not work better than ours. Here, we could understand this experimental observation from two aspects: 1) It does not integrate coexisting relationships among different tokens into their supervisions. 2) It can be understood as a special case of the proposed policy gradient based modelling in Eq.3, since it is actually equivalent to estimating the expectation term in Eq.3 by always taking the token with the highest probability value from the distribution $P(\cdot)$ as the Monte Carlo sampling results, with other tokens ignored. However, the token with the highest reward may not necessarily the one with the highest probability value in the current distribution $P(\cdot)$, so it is prone to under-optimization.

Reviewer comment (Writing):
Although the Monte Carlo sampling is commonly adopted in the RL community for computing the expectation value of rewards, I’d suggest the authors briefly describe its formulation or core ideas for the paper’s completeness.

Author response:
**Q2: Add the formulation or core ideas of Monte Carlo sampling.**  
**A2:** Thanks for this valuable suggestion. The core idea of Monte Carol sampling is to complete a numerical estimation based on the numerical results obtained by repeated random sampling according to a given probability distribution. We have added it in our revision and marked the corresponding part in red.

Reviewer comment (Novelty):
In Sec 3.1, the authors claim that some of the existing grounding methods require the bounding boxes of all UI elements as priors and that limits their generic use in practice. However, the proposed methods also need the coordinates of the bounding boxes as the labels for computing the rewards. In this case, what are the advantages of the proposed methods in terms of practice using?

Author response:
**Q3: What are the adavantages of the proposed methods in terms of practical using?**  
**A3:** Sorry for this confusion. The existing methods do not model the task of localizing UI elements upon instructions as a **visual** grounding problem, while our this work is the first one to cast this task as a **visual** grounding problem. They commonly require the bounding boxes from the metadata as the model inputs, thus limiting their practical using when the metadata is not avaliable or noisy. In contrast to these model, our advantage in terms of practical using lies in that we do not rely on this information during inference since we do not take it as the model inputs.

Reviewer comment (Experiments):
In the comparisons to existing methods in Table 5, why not use web data?

Author response:
**Q4: In the comparisons to existing methods in Table 5, why not use web data?**  
**A4:** This is because other methods we compare are proposed for handling mobile UI tasks. They haven't reported their results on web data.

Reviewer comment (Novelty):
It is not very clear what is the key difference between the UI task and general object grounding tasks [1][2][3]? A follow-up question is that is the proposed reinforced learning method similar to [4]?

Author response:
**Q1: The key difference between UI tasks and general object grounding tasks [1][2][3]?**  
**A1:** Thanks for this question. The UI tasks differ from general object (visual) grounding tasks [1][2][3] in: 1) The UI screenshots are domain-specific images as they are densely populated with texts. This characteristic requires the visual grounding model for UI tasks to have sufficient OCR capability and stronger instance (instead of category) descrimination capability relative to those for general object grounding tasks. This is why we propose to adopt a Pix2Seq paradigm to align the output format of OCR (document understanding) and grounding for the utilization of pre-training. 2) Another slight difference is that the language inputs for UI tasks are user instructions which include but are not limited to element captions (with actions and other components also included), while those for general object grounding are mainly object captions. 

In additon, we would like to further clarify that it is only a design choice for UI tasks to be modeled as a visual grounding problem. Besides the screenshots, the metadata containing comprehensive information of UI elements (including their names, types, bounding boxes, relationships to other elements, etc.) can also serve as the model inputs for UI element localization. In fact, most previous works cast the UI element localization task as a classification problem, and take metadata or partial information of metadata as the model inputs for learning to select the target element upon given instructions. We are the first of its kind to show the feasibility and advantages of modelling this task as a visual grounding problem.

**Q2: Is the proposed reinforced learning method similar to [4]?**  
**A2:** They are similar in their high-level roles but different in the addressed problems. The similarity lies in that they both apply policy gradients based RL to enable the optimization of a global objective, but with distinct purposes in different problems. The method [4] is to enable the optimizaiton with the mAP metric over multiple objects for a vision model. Ours is designed to address the limitation issue on ignoring token-combinational semantics in the sequential decoding process for a language model.

Reviewer comment (Novelty):
In my understanding, introducing Reinforcement Learning is the main contribution of this paper, instead of multimodal large language model, since a lot of papers have been proposed to use multimodal large language model to solve tasks during the past months. However, the authors only use half of the page to illustrate the reinforced pixel-to-sequence model. Is there no detailed introduction or well-curated design?

Author response:
**Q3: Why do we only use half of the page to illustrate the reinforced pixel-to-sequence model?**  
**A3:** The reason for this is that we would like to benefit more readers by bringing their attentions to two takeaways of this work. The first one is for the research community on UI tasks, that is the feasibility and benefits of casting instruction-following UI task as a visual grounding problem with an improved pixel-to-sequence model. The other is about the improvement of pixel-to-sequence method itself for a broader research community. For the latter one, we think the analysis on the limitation of vanilla pixel-to-sequence model is as important as how to reinforce it in specific. A balanced solution is what we present currently.

Reviewer comment (Experiments):
In Section 4.3, when comparing with traditional grounding approaches, it is not surprising the traditional grounding approaches and UI-tailored grounding approaches are not good at understanding UI data, since their language models have less capacity than Large Language Models. Have the recently-proposed LLM-based multimodal models [5][6][7] been tested on UI data?

Author response:
**Q4: Have the recently-proposed LLM-based multimodal models [5][6][7] been tested on UI data ?**  
**A4:** Yes, we also test them with the corresponding results reported below. Note that for this kind of models, we are not sure whether the tested Apps or Webs are seen or unseen in their training. Therefore, there are no obvious gaps between the seen and unseen settings.
| Models                      | App Seen (mIoU \| Acc) | App Unseen (mIoU \| Acc) | Web Seen (mIoU \| Acc) | Web Unseen (mIoU \| Acc) |
|:----------------------------|:---------------:|:---------------:|:---------------:|:---------------:|
| Kosmos-2 [5]                | 0.03 \| 5.41%   | 0.03 \| 5.48%   | 0.03 \| 5.19%   | 0.03 \| 5.29%   |
| Shikra [6]                  | 0.02 \| 4.26%   | 0.02 \| 4.89%   | 0.02 \| 3.26%   | 0.02 \| 3.47%   |
| MiniGPT-v2 [7]              | 0.04 \| 4.93%   | 0.03 \| 4.43%   | 0.05 \| 6.85%   | 0.04 \| 5.58%   |
| RUIG (Ours)                 | 0.62 \| 81.16%  | 0.48 \| 73.92%  | 0.51 \| 61.78%  | 0.49 \| 59.03%  |

Reviewer comment (Experiments):
It is unclear if the benefit of the reward loss is due to the lack of model capacity in the LLM or if it would scale with the model size (e.g. LLaMA-2) and the model knowledge (e.g. more training data). It is nice that the authors conduct the scaling experiments in Table 6, but it is not necessarily at the scale of the recent large language models and has not undergone large-scale pretraining. Despite LLMs use the same loss at token level, which also has the "combinational semantics" issue, they are able to achieve complex reasoning capabilities as they scale up.

Author response:
Thanks for positive comments and valuable questions. We response to your questions in detail below.

**Q: How about scaling up to the scale of the recent LMMs and comparing to them?**  
**A:** Thanks for your valuable questions for inspiring us to further investigate the potential of our proposed approach. Actually, our proposed model functions as a low-level (single-step) command executor for UI task automation. In terms of its pratical demands and design purposes, it was never anticipated to be scaled up to such an extent, in consideration to its industrial requirements (e.g., inference speed). To study its technical potential, we scale our baseline and the proposed model to the 7B scale on the current data for collecting more experimental evidences. Specifically, to obtain the results promptly, we conduct the comparison experiment between the baseline and our proposed model at this scale by employing a frozen vision encoder and fine-tuning the Vicuna-7B (an open-source improved version of LLaMA-7B) on our UI data with LoRA. They are denoted by *Baseline (Vicuna-7B)* and *RUIG (Vicuna-7B)*, respectively, as below. Besides, we also test LLaVA out of box on our UI data. All results are reported in the following table.
| Models                      | App Seen (mIoU \| Acc) | App Unseen (mIoU \| Acc) | Web Seen (mIoU \| Acc) | Web Unseen (mIoU \| Acc) |
|:----------------------------|:---------------:|:---------------:|:---------------:|:---------------:|
| LLaVA                       | 0.05 \| 5.12%   | 0.04 \| 4.82%   | 0.04 \| 4.77%   | 0.05 \| 5.49%   |
| Baseline (Vicuna-7B)        | 0.39 \| 49.92%  | 0.33 \| 38.91%  | 0.30 \| 36.43%  | 0.28 \| 35.58%  |
| RUIG (Vicuna-7B)            | 0.46 \| 58.35%  | 0.37 \| 51.01%  | 0.39 \| 49.14%  | 0.36 \| 46.21%  |

With the constraints in time and computational resources, we have not yet tuned the baseline and our proposed model to their best during this rebuttal. Based on the current experiment results as above, we have the following observations: 1) Our proposed method consistently delivers improvements by a clear margin when scaled up to 7B. 2) LLaVA cannot work well when tested on our task right out of box, as it is designed for general object tasks. We believe that our proposed approach is still beneficial for its SFT on UI-domain data since LLaVA can be generally considered as a Pix2Seq model with an autoregressive decoding process. And our proposed approach addresses the limitation (lacking the awareness on the "combinational semantics") of current autoregressive decoding process without changing the decoding mechanism itself, as commented by Reviewer 3sqL. Therefore, we believe that, at least, it can facilitate the model optimization on a larger scale.  

We are looking forward to your feedbacks on our rebuttal. Please feel free to let us know if you have new questions or suggestions. Thanks!

Reviewer comment (Novelty):
How do you compare the original Pix2Seq model, and the recent instruction-tuned multimodal models like LLaVA [1]? Can we think of them as Pix(image)2Seq(language)? -- bounding box outputs is a special case. If LLaVA is finetuned to (1) understand the text; (2) predict the bounding box for objects using datasets like COCO, would the proposed approach still be beneficial compared with SFT on these datasets? Given that this is a concurrent work, I am putting this in the questions section instead of a weakness.

Author response:
Thanks for positive comments and valuable questions. We response to your questions in detail below.

**Q: How about scaling up to the scale of the recent LMMs and comparing to them?**  
**A:** Thanks for your valuable questions for inspiring us to further investigate the potential of our proposed approach. Actually, our proposed model functions as a low-level (single-step) command executor for UI task automation. In terms of its pratical demands and design purposes, it was never anticipated to be scaled up to such an extent, in consideration to its industrial requirements (e.g., inference speed). To study its technical potential, we scale our baseline and the proposed model to the 7B scale on the current data for collecting more experimental evidences. Specifically, to obtain the results promptly, we conduct the comparison experiment between the baseline and our proposed model at this scale by employing a frozen vision encoder and fine-tuning the Vicuna-7B (an open-source improved version of LLaMA-7B) on our UI data with LoRA. They are denoted by *Baseline (Vicuna-7B)* and *RUIG (Vicuna-7B)*, respectively, as below. Besides, we also test LLaVA out of box on our UI data. All results are reported in the following table.
| Models                      | App Seen (mIoU \| Acc) | App Unseen (mIoU \| Acc) | Web Seen (mIoU \| Acc) | Web Unseen (mIoU \| Acc) |
|:----------------------------|:---------------:|:---------------:|:---------------:|:---------------:|
| LLaVA                       | 0.05 \| 5.12%   | 0.04 \| 4.82%   | 0.04 \| 4.77%   | 0.05 \| 5.49%   |
| Baseline (Vicuna-7B)        | 0.39 \| 49.92%  | 0.33 \| 38.91%  | 0.30 \| 36.43%  | 0.28 \| 35.58%  |
| RUIG (Vicuna-7B)            | 0.46 \| 58.35%  | 0.37 \| 51.01%  | 0.39 \| 49.14%  | 0.36 \| 46.21%  |

With the constraints in time and computational resources, we have not yet tuned the baseline and our proposed model to their best during this rebuttal. Based on the current experiment results as above, we have the following observations: 1) Our proposed method consistently delivers improvements by a clear margin when scaled up to 7B. 2) LLaVA cannot work well when tested on our task right out of box, as it is designed for general object tasks. We believe that our proposed approach is still beneficial for its SFT on UI-domain data since LLaVA can be generally considered as a Pix2Seq model with an autoregressive decoding process. And our proposed approach addresses the limitation (lacking the awareness on the "combinational semantics") of current autoregressive decoding process without changing the decoding mechanism itself, as commented by Reviewer 3sqL. Therefore, we believe that, at least, it can facilitate the model optimization on a larger scale.  

We are looking forward to your feedbacks on our rebuttal. Please feel free to let us know if you have new questions or suggestions. Thanks!