Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding (2024)

Abstract: Graphical User Interfaces (GUIs) are central to our interaction with digital devices and growing efforts have been made to build models for various GUI understanding tasks. However, these efforts largely overlook an important GUI-referring task: screen reading based on user-indicated points, which we name the Screen Point-and-Read (ScreenPR) task. Currently, this task is predominantly handled by rigid accessible screen reading tools, in great need of new models driven by advancements in Multimodal Large Language Models (MLLMs). In this paper, we propose a Tree-of-Lens (ToL) agent, utilizing a novel ToL grounding mechanism, to address the ScreenPR task. Based on the input point coordinate and the corresponding GUI screenshot, our ToL agent constructs a Hierarchical Layout Tree. Based on the tree, our ToL agent not only comprehends the content of the indicated area but also articulates the layout and spatial relationships between elements. Such layout information is crucial for accurately interpreting information on the screen, distinguishing our ToL agent from other screen reading tools. We also thoroughly evaluate the ToL agent against other baselines on a newly proposed ScreenPR benchmark, which includes GUIs from mobile, web, and operating systems. Last but not least, we test the ToL agent on mobile GUI navigation tasks, demonstrating its utility in identifying incorrect actions along the path of agent execution trajectories. Code and data: https://screen-point-and-read.github.io.

## introduction graphical user interfaces (guis), dominate our digital interactions with visually rich screenshots featuring colors, icons, texts, and spatial layouts. recognizing that screenshots are more accessible and intuitive for depicting visual cues, there are growing efforts to build ai agents to interpret guis visually @xcite @xcite @xcite . existing works, however, overlook the point-based screen reading task, where the input is the coordinate of a user's indicated point on the screen with the corresponding screenshot, and the output is a descriptive interpretation of the screen region pointed. this task is critical for accessible technology, providing valuable assistance to users with visual impairments and we refer to such a task as the screen point-and-read (screenpr) task. aiming to solve the screenpr task, we introduce the tree-of-lens (tol) agent. taking advantage of the generalizability of the advanced multimodal large language models (mllms), the tol agent is built for gui screenshots across domains with a point anywhere on the screen as input. the output from the tol agent is natural language descriptions for both the content of the indicated region and the related information about the screen layout, as shown in figure 1 . especially, we find that it is critical to articulate the layout between elements in the screenshot for users to gain a comprehensive understanding of the interface and to avoid ambiguity. for example, with only content description, one cannot distinguish the two identical "tumbler pack" shown in figure 1 . our tol agent employs a tol grounding mechanism, constructing a hierarchical layout tree for each input screenshot to depict its underlying structure. the nodes in the tree represent regions of varying scales. we build this tree using an object detection model trained on our newly collected android screen hierarchical layout (ashl) dataset, which contains 50k bounding boxes of hierarchical screen regions from android screenshots. this model enables an automatic extraction of local and global regions, forming the hierarchical layout tree. after constructing the tree, we extract the target path based on the region of interest, producing lenses as visual prompts for the mllm, simulating a human-like, gradually refined focus. to rigorously evaluate our tol agent against other baselines, we introduce the screen point-and-read (screenpr) benchmark with the screenpr task that requires the model to generate descriptions based on user-indicated points. this benchmark consists of 650 screenshots from web, mobile, and operating system guis, manually annotated with 1,500 target points and regions. with the screenpr benchmark, we show that our tol agent achieves the best performance compared with the baselines of other general and gui-specialized mllms, with over 15% and 30% improvements respectively in terms of the accuracy of content and layout descriptions compared with vanilla textprompting gpt-4o. in addition, we also test our tol agent with trajectories from a mobile gui navigation agent. by applying the tol agent to describe the point of each action taken in the trajectory, we demonstrate its utility in identifying incorrect actions along the execution path. this capability is pivotal for refining the development of mobile agents, as it provides clear, actionable feedback on navigation decisions. this application not only validates the tol agent's effectiveness but also opens up possibilities for strengthening the performance of mobile agents with sophisticated, layout-aware gui understanding capabilities. in conclusion, the contributions of our paper are as follows: in our work, we focus on the screen point-and-read (screenpr) task, a referring task in which inputs are point locations on the screen with the corresponding screenshots, and develop a specific benchmark to evaluate the model's performance in such tasks. also, inspired by big challenges that mllms are facing in understanding images with complex layout information @xcite , we propose the tree-of-lens (tol) agent that generates layout-aware description, which distinguishes from all other gui agents with gui referring ability. ## experiments baselines we adopt several baselines, including generalist mllms: the proprietary gpt-4o (ope-nai, 2024) and the open-source llava-next @xcite . we also test the cogagent @xcite , a mllm specifically tuned for gui tasks, and a visual prompting method scaffolding @xcite that is specially developed to enhance the spacial reasoning ability of mllms. the inputs for baselines are the text prompt with the input point coordinate p i and the screenshot s i . we keep prompts for baselines in the same style as the tol agent, except that we have to simplify it for cogagent as we find it necessary for cogagent to follow the instructions. detailed prompt design is shown in appendix d. ## conclusion in this work, we propose a novel tree-of-lens (tol) grounding method and build a tol agent for the screen point-and-read (screenpr) task, an important gui referring task that is critical for accessible technology. our tol agent outputs layoutaware descriptions for pointed regions on screenshots. with our newly proposed screen pointand-read (screenpr) benchmark, we show that our tol agent outperforms all other baselines in terms of content and layout description accuracy, demonstrating great potential in assisting visually impaired users when using digital devices. last but not least, in our experiment, we also apply our tol agent to identify incorrect actions performed by a mobile navigation agent, yielding promising results.