Mobile-Bench: An Evaluation Benchmark for LLM -based Mobile Agents (2024)

Abstract: With the remarkable advancements of large language models (LLMs), LLM-based agents have become a research hotspot in human-computer interaction.However, there is a scarcity of benchmarks available for LLM-based mobile agents.Benchmarking these agents generally faces three main challenges:(1) The inefficiency of UI-only operations imposes limitations to task evaluation.(2) Specific instructions within a singular application lack adequacy for assessing the multi-dimensional reasoning and decision-making capacities of LLM mobile agents.(3) Current evaluation metrics are insufficient to accurately assess the process of sequential actions. To this end, we propose Mobile-Bench, a novel benchmark for evaluating the capabilities of LLM-based mobile agents.First, we expand conventional UI operations by incorporating 103 collected APIs to accelerate the efficiency of task completion.Subsequently, we collect evaluation data by combining real user queries with augmentation from LLMs.To better evaluate different levels of planning capabilities for mobile agents, our data is categorized into three distinct groups: SAST, SAMT, and MAMT, reflecting varying levels of task complexity. Mobile-Bench comprises 832 data entries, with more than 200 tasks specifically designed to evaluate multi-APP collaboration scenarios.Furthermore, we introduce a more accurate evaluation metric, named CheckPoint, to assess whether LLM-based mobile agents reach essential points during their planning and reasoning steps. Dataset and platform will be released in the future.

## introduction interacting with mobile devices using natural language is a long-standing pursuit in humancomputer interaction @xcite ., <@xmath0; @xcite . with the remarkable advancements in large language models (llm) @xcite @xcite @xcite , llm-driven agents are at the forefront, yet their reasoning capability to navigate mobile application functionalities lags behind their proficiency with web pages on pcs @xcite . to faithfully replicate a typical mobile environment, it's imperative to incorporate a diverse set of applications and leverage authentic data, moving beyond the limitations of purely simulated scenarios. the development challenges in the mobile domain stem from a trio of core issues: a limited understanding of mobile interfaces, a scarcity of application variety, and a lack of real-world data. due to google's breakthrough @xcite in ui interface representation, llm agent's understanding of ui pages becomes easier, leading to the creation of ui platforms such as android-env @xcite and mobile-env @xcite , which tasks are defined within individual games or search engines. however, these works collectively face the following challenges: the content of graphical buttons or images which can lead to wrong actions. a single api action might be equivalent to dozens of ui steps, leading to ui's inefficiency. (2) their tasks are far removed from real-world task scenarios encountered in daily use, which require cooperation between multiple applications, with user commands being ambiguous and not specifying target applications. in fact, voice assistants on mobile phones can meet most of the users' daily needs, yet they do not interact directly with ui interfaces but operate by invoking the apis @xcite behind applications. as shown in figure 1 , in mobile applications, apis are more efficient than ui interfaces; a single api call can be equivalent to multiple ui operations to achieve the same outcome. however, a single api is insufficient for more complex tasks, especially when user commands are unclear, necessitating reliance on llms to interpret user intent. therefore, an agent capable of utilizing both ui and apis would be best suited for the job. simultaneously, it requires developing a strategy for the selection and order of the application usage, with human oversight merely focusing on reviewing the outcomes. this is a function that voice assistants currently lack @xcite . to this end, we develop a combination of api and ui actions to circumvent the limitations of ui interfaces, each action can be chosen between ui interactions and api calls; all tasks begin from the mobile home page rather than from the launch page of a specific application, enabling the agent to determine single or multiple applications it will use; queries in the task are gathered from real users, and instruction generation is only applied to some complex ones which undergo rigorous manual review; we draw inspiration from objective metrics in software automation testing, named checkpoint, and have made necessary adjustments to accommodate the unpredictable semantic outputs of llms. above all, we propose a mobile phone environment that includes a platform supporting both api and ui interactions, and a corresponding dataset with multi-app tasks. our contributions are summarized as follows: ## conclusion in this work we have proposed an agent capability testing environment that supports api and ui interaction on mobile phone. this holds significant importance for exploring how llms can be integrated with mobile operating systems. additionally, it can serve as a valuable reference for developing testing platforms for operating systems to evaluate the capabilities of llm agents. we collected and released a test dataset containing tasks for multiple apps, ensuring its quality through human verification. based on this data set and environment, we tested the planning, decision-making and execution of various llm-based agents. please refer to the section 6 for the limitations of our benchmark.