CRAB : Cross-environment Agent Benchmark for Multimodal Language Model Agents (2025)

Abstract: The development of autonomous agents increasingly relies on Multimodal Language Models (MLMs) to perform tasks described in natural language with GUI environments, such as websites, desktop computers, or mobile phones. Existing benchmarks for MLM agents in interactive environments are limited by their focus on a single environment, lack of detailed and generalized evaluation methods, and thecomplexities of constructing tasks and evaluators. To overcome these limitations, we introduce CRAB, the first cross-environment agent benchmark framework, incorporating a graph-based fine-grained evaluation method and an efficient task generation method. Our framework supports multiple devices and can be easily extended to any environment with a Python interface. Leveraging CRAB, we develope CRAB Benchmark-v0 comprising 120 tasks in computer desktop and mobile phone environments. We evaluated 6 advanced MLMs using different single and multi-agent system configurations on this benchmark. The experimental results demonstrate that the single agent with GPT-4o achieves the best completion ratio of 38.01%.

## introduction in recent years, multimodal language models (mlms) based gui agents have emerged as a significant area of research, aiming to develop intelligent systems capable of interacting with digital environments autonomously, including desktop os @xcite , websites @xcite , smartphones @xcite , and games @xcite . a crucial aspect of advancing these agents is the evaluation process, which directly impacts their effectiveness and evolution. there are two distinct types of gui agent benchmarks: one-step benchmarks @xcite @xcite and taskcompletion benchmarks @xcite . one-step benchmarks typically provide a screen-shot, a current instruction, and an action space, along with a correct action as the label. in this setting, the agent is required to select a single correct action without access to prior actions or full environment information. these benchmarks are primarily used to evaluate ui grounding capabilities. in contrast, task-completion benchmarks involve one or more interactive environments and require the agent to complete a task based on a given instruction. existing task completion benchmarks are typically evaluated on a single platform, either web, android, or desktop os @xcite @xcite . however, real-world applications often involve tasks that span multiple platforms, such as smartphone-pc collaboration or distributed server interactions. traditional benchmarks do not support such cross-platform tasks, limiting their applicability to practical scenarios. to address these limitations, we introduce a novel framework crab, cross-environment agent benchmark framework. it leverages multiple environments, enabling agents to interact with real-world digital systems independently. another major drawback of existing benchmarks lies in two key areas: task construction and evaluation methodology. task construction often requires significant human labor to manually design and write tasks, limiting scalability. to overcome this, we propose a graph-based task construction approach capable of generating diverse and dynamic tasks efficiently. additionally, traditional evaluation paradigms used in the one-step benchmarks, particularly those designed for original large language models (llms), which do not align with the interactive nature of agents. many existing benchmarks either focus solely on final task completion, which overlooks intermediate progress, or enforce predefined solution paths, restricting the flexibility of different correct approaches. our proposed evaluation framework seeks to overcome these limitations by capturing both final outcomes and intermediate milestones, figure 1 : architecture of the crab demonstrating the task generation process and benchmarking workflow for a multi-agent system. a task is generated by selecting subtasks from a task pool to form a graph, which is then used to construct the graph evaluator and generate instructions. the benchmarking workflow progresses through a cycle where the main agent observes, plans, and instructs the sub-agents, who then execute actions within their respective environments. the graph evaluator monitors the status of tasks within the environments, continuously updating and outputting the task completion metrics. using the aforementioned crab framework, we propose a benchmark crab benchmark-v0 with two cooperated environments that include an android emulator and an ubuntu desktop virtual machine. we have developed a total of 120 real-world tasks. these tasks address a wide array of common real-world applications and tools, including but not limited to calendars, email, maps, web browsers, and terminals, and facilitate common interactions between smartphones and desktops. considerable time has been invested in verifying the accuracy and comprehensiveness of the instructions for subtasks, as well as the generalization and correctness of their evaluators. we test 6 popular mlms, including gpt-4 turbo, gpt-4o, claude 3 pro, gemini 1.5 pro, pixtral-8b, and llava-onevision-72b across different structures of single-agent and multi-agent systems, totaling 12 different agent settings in our benchmarks. the experimental results show that the single agent structure with gpt-4o model achieves the best overall completion ratio of 38.01%, underscoring the necessity for ongoing development of more effective autonomous agents. our proposed metrics successfully distinguish between different methods better than previous metrics. we further analyze the different termination reasons that reflect the problems inherent in the communication within the multi-agent system. interactive environment multimodal observation crossplatform evaluation task construction # of apps or websites miniwob++ (shi et al., 2017) web ✓ ✗ goal-based handmade 1 webshop (yao et al., 2022) web ✓ ✗ goal-based template 1 metagui (sun et al.) ✗ ✗ ✗ trajectory-based handmade 6 gaia (mialon et al.) ✗ ✗ ✗ goal-based handmade n/a mind2web (deng et al., 2023) ✗ ✗ ✗ goal-based llm-inspired 137 agentbench (liu et al., 2024) multi-isolated ✗ ✗ multiple handmade n/a intercode (yang et al.) code ✗ ✗ goal-based handmade n/a webarena (zhou et al.) web ✓ ✗ goal-based template 6 omniact (kapoor et al.) ✗ ✗ ✗ trajectory-based handmade 60+ vwebarena (koh et al.) web ✓ ✗ goal-based template 4 androidarena (xing et al., 2024) android ✓ ✗ trajectory-based llm-inspired 9 osworld (xie et al.) linux / windows ✓ ✗ goal-based template 9 mobile-env (zhang et al., 2024) android ✓ ✗ intermediate-reward template 13 gui-world (chen et al., 2024) ✗ ✓ ✗ llm-as-a-judge llm-inspired not present androidworld (rawles et al., 2024) android ✓ ✗ goal-based template 20 waa (bonatti et al., 2024) windows ✓ ✗ goal-based handmade 6 gui-odyssey (lu et al., 2024) ✗ ✓ ✗ trajectory-based llm-inspired 201 crab linux & android ✓ ✓ graph-based subtask composition 25 @xcite , additional system components @xcite @xcite @xcite , and novel training methods @xcite @xcite @xcite @xcite , a robust and widely adopted benchmark for evaluation remains necessary. recent gui agent systems @xcite @xcite have also begun to model gui tasks across different platforms from a unified perspective, highlighting the need for benchmarks that focus on cross-platform capabilities. benchmarks for gui agents recently, various benchmarks have been applied to evaluate the capabilities of gui agents across different platforms. early works @xcite @xcite @xcite focus on evaluating the automation capabilities of web-based agents, which were designed using the xml architecture of webpages. with the rise of multi-modal web agents, subsequent works (koh et al.; @xcite @xcite @xcite have been built upon multimodal inputs for web tasks. researchers have also expanded the focus from web-based platforms to mobile systems. some studies @xcite @xcite @xcite @xcite rely solely on visual information, while others @xcite @xcite @xcite introduce additional inputs, such as xml data or control codes. in addition, efforts @xcite @xcite @xcite have been made to develop benchmarks for gui control tasks in computer operating systems, covering various environments such as ubuntu @xcite , windows @xcite , and other systems. these contributions have significantly advanced gui automation. these benchmarks can be broadly categorized into two types based on evaluation methods: goal-based benchmarks @xcite @xcite and trajectory-based benchmarks @xcite . mobile-env @xcite introduces a new perspective by incorporating immediate rewards into the evaluation process. existing benchmarks for gui agents generally ignore the potential graph-like relationships between sequential actions, where some steps can be reordered without affecting task success. this makes evaluations less realistic, as tasks in the real world don't always have a fixed order of steps. although our benchmark does not include the largest number of apps, it uniquely emphasizes crossenvironment evaluation, a significantly more complex and underexplored benchmarking scenario. in contrast to one-step benchmarks, which cannot accurately evaluate the agent performance in real world scenario, our approach involves fully interactive environments and introduces a graph-based evaluator. this requires greater implementation effort but provides increased flexibility and extensibility for future research. ## conclusion we propose the crab framework, which introduces the cross-environment automatic taskperforming problem, featuring advanced graphbased task generation and evaluation methods that reduce manual effort in task design and provide more dynamic and accurate agent assessments. based on this framework, we present crab benchmark-v0, a set of high-quality crossenvironment tasks in smartphone and desktop environments. we tested various backend models and agent system structures on this dataset. the results reveal preferences for different agent settings.