S creen QA : Large-Scale Question-Answer Pairs Over Mobile App Screenshots (2025)

Abstract: We introduce ScreenQA, a novel benchmarking dataset designed to advance screen content understanding through question answering. The existing screen datasets are focused either on low-level structural and component understanding, or on a much higher-level composite task such as navigation and task completion for autonomous agents. ScreenQA attempts to bridge this gap. By annotating 86k question-answer pairs over the RICO dataset, we aim to benchmark the screen reading comprehension capacity, thereby laying the foundation for vision-based automation over screenshots. Our annotations encompass full answers, short answer phrases, and corresponding UI contents with bounding boxes, enabling four subtasks to address various application scenarios. We evaluate the dataset’s efficacy using both open-weight and proprietary models in zero-shot, fine-tuned, and transfer learning settings. We further demonstrate positive transfer to web applications, highlighting its potential beyond mobile applications.

## introduction recent advancements in machine learning, especially visual large language models (vlms) or multimodal llms (mllms), have catalyzed numerous applications centered on mobile screens. these applications range from personal assistants enabling hands-free or eyes-free interaction to code generation from ui design mock-ups, adaptive device interfaces, automated ad creation, and mobile app testing. all of these require reliable screen content understanding as the foundation to ensure quality applications. mobile app screenshots have been analyzed using machine learning from multiple perspectives. these include pixel-level understanding, such as layout structural analyses, ui issue detection and correction @xcite , ui element semantics like icon recognition, button action prediction @xcite , to even higherlevel functional analyses such as accessibility support @xcite , screen description @xcite , and screen type classification @xcite . however, the specific aspect of understanding screen contents from pixels remains comparatively understudied. screen contents encompass diverse information, from restaurant ratings to system settings and chat messages. this capacity is crucial because: 1) screens primarily function to present information, and 2) autonomous agents and task completion systems require precise screen content understanding to achieve reasonable success rates for multi-step processes. in this work, we propose using pixels as the sole representation of ui screens, to eliminate dependency on unreliable structural representations such as view hierarchies (vhs) @xcite and maximize applicability to various visual understanding tasks. to this end, we annotated the rico dataset @xcite with 85,984 questionanswer pairs (section 4) and defined four tasks with evaluation metrics (section 3), creating screenqa foot_0 . rico remains the preferred screenshot dataset due to its 1) extensive collection of 66k screenshots, 2) diverse app distribution spanning 9.3k apps across 27 categories, 3) numerous benchmarking tasks built upon rico that enable crossreferencing and comparison foot_1 @xcite @xcite @xcite , and 4) widespread use as a training dataset in recent document/screen understanding and vlm research @xcite @xcite . despite the availability of newer datasets, rico's comprehensive nature and established presence in the field make it an optimal choice for this work. see this work offers the following contributions: ## related work closest to screenqa is work that tackles multimodal understanding and question answering, as discussed below. ## 22 object detection motif @xcite 62 4.7 navigation aitw @xcite 5,690 † 715 navigation * a subset of the 66k images from rico @xcite as described in section 4 † limited app and screenshot diversity as described in actionability is related to task completion and autonomous agents. motif @xcite , vi-sualwebarena @xcite , and android-inthe-wild @xcite provide interactive app environments for evaluating visually grounded screen agents. this work is focused on the information retrieval aspect. visual document understanding visual document understanding concerns understanding scanned or photographed documents. docvqa @xcite uses an extractive qa format for span/segment extraction. textvqa @xcite and several other domain-specific datasets @xcite @xcite @xcite @xcite relate the 2d arrangement of texts to semantic meanings. beyond text alone, infographics understanding @xcite @xcite @xcite focus on charting with text around. ## data annotation we performed a five-stage process to annotate screenqa: 1) prefiltering (section 4.1), 2) question annotation (section 4.2), 3) answer annotation (section 4.3), 4) not-answerable question annotation (section 4.4), and 5) short answer generation (section 4.5). see data statistics in ## experiments and baselines we conducted three sets of experiments: zero-shot, fine-tuning, and transfer learning, each of which is explained below. ## conclusion we introduced screenqa, a rich dataset that enables training and evaluating models on questionanswering tasks on screen content. we described the annotation process, statistics of the collected dataset, which contains 85,984 question-answer pairs. in addition to answers, our dataset contains extensive annotations of the ui elements, enabling the ability to train or probe models for their holistic understanding of the screen, a necessary capability for high-level reasoning and automation using ui interfaces. compared to other vision-language tasks, such as document understanding or visualquestion answering, the four constructed tasks on the screenqa dataset pose unique challenges: rich in text, diverse in mobile applications, blended with icons and symbols. the tasks not only evaluate content quality, but also ui element identification quality. furthermore, we conducted a diverse set of experiments, including zero-shot, fine-tuned, and transfer learning, on a series of open-weight and proprietary models to assess the capacity of our screenqa dataset. we encourage the community to tackle screen content understanding challenges present in our benchmark, fostering new technologies and user experiences.