MMI n A : Benchmarking Multihop Multimodal I nternet Agents (2025)

Abstract: Autonomous embodied agents live on an Internet of multimedia websites. Can they hop around multimodal websites to complete complex user tasks? Existing benchmarks fail to assess them in a realistic, evolving environment for their embodiment across websites. To answer this question, we present MMInA, a multihop and multimodal benchmark to evaluate the embodied agents for compositional Internet tasks, with several appealing properties: ***1) Evolving real-world multimodal websites.*** Our benchmark uniquely operates on evolving real-world websites, ensuring a high degree of realism and applicability to natural user tasks. Our data includes 1,050 human-written tasks covering various domains such as shopping and travel, with each task requiring the agent to extract multimodal information from web pages as observations autonomously. ***2) Multihop web browsing.*** Our dataset features naturally compositional tasks that require information from or actions on multiple websites to solve, to assess long-range reasoning capabilities on web tasks. ***3) Holistic evaluation.*** We propose a novel protocol for evaluating an agent’s progress in completing multihop tasks. We experiment with both standalone (multimodal) language models and heuristic-based web agents. Extensive experiments demonstrate that while long-chain multihop web tasks are easy for humans, they remain challenging for state-of-the-art web agents. We identify that agents are more likely to fail on the early hops when solving tasks of more hops, which results in lower task success rates. To address this issue, we propose a simple memory augmentation approach replaying past action trajectories to reflect. Our method significantly improves the performance of both the single-hop and multihop web browsing abilities.

## introduction building embodied agents capable of autonomous behaviors navigating in various environments has been a longstanding and intricate challenge in the realm of artificial intelligence research @xcite @xcite . one common scenario that necessitates automation involves the interaction with digital interfaces @xcite , with a particular emphasis on the automation of actions performed on rich internet websites @xcite @xcite . real-world web tasks are inherently compositional, requiring multihop actions across multiple websites. to accomplish this, agents must possess both long-range planning and multimodal reasoning capabilities. this includes understanding high-level instructions from user inputs, planning multihop actions across the web browser environment by leveraging html content and visual cues, and making informed predictions based on observations. while web agents perform well on singlehop tasks that require interactions with only one website, according to existing benchmarks @xcite @xcite @xcite , we observe that most agents struggle with multihop web tasks, which are prevalent in real-world scenarios where users must gather information from or take actions across multiple websites to accomplish a high-level task (tab. 2). this gap motivates us to establish a multihop web browsing benchmark to assess the usefulness of internet agents in natural multihop tasks. another gap in web agent research is multimodality. existing benchmarks pose autonomous agent tasks that rely solely on textual information @xcite @xcite . however, in real-world scenarios, visual information often plays an indispensable role and cannot be disregarded. for example, consider the figure 1 : an example task from mmina. to evaluate an internet agent's ability to carry out complex tasks, we make it navigate through a variety of websites to gather information and execute actions. in our proposed holistic evaluation protocol, each phase of the compositional task (defined as a hop) and the overall task are assessed for performance. our benchmark includes 1,050 varied human-written multimodal tasks that require an average of 2.85 hops between websites and 12.9 actions to complete. the longest compositional task takes 10 hops. to address the two issues above, we present mmina, a novel benchmark designed to advance multihop and multimodal internet task solving. mmina operates on real-world, evolving websites (fig. 1 ), offering high realism and applicability to natural scenarios (tab. a3 ). it focuses on realistic tasks users commonly perform, such as navigating e-commerce platforms, extracting information from content-rich sites like wikipedia, and conducting comparative analysis across web sources. our 1,050 human-written tasks challenge agents to process multimodal inputs across multiple website hops and execute complex, multi-step reasoning, surpassing the simpler tasks found in existing benchmarks. our experiments with state-of-the-art models as agents' reasoning backbones show that while progress has been made in handling simple textual tasks, the integrated and sequential nature of tasks in mmina remains a significant challenge. for example, gpt-4v @xcite achieves a 21.8% success rate, which improves upon textual baselines but falls short of human performance (96.3%). agents often fail early in multi-hop tasks (tab. 3 ), leading to lower success rates. these results highlight the complexity of real-world web navigation and decision-making, underscoring the need for further advancements in multimodal and multihop reasoning. to bridge this gap, we propose a memory augmentation approach that replays past action trajectories, significantly improving both single-hop and multihop performance. this modelagnostic technique can be applied to other large multimodal models in the future. in summary, our contributions are as follows: web agents webshop @xcite builds a simulated e-commerce environment featuring 1.18 million real-world products, complemented by 12,087 crowdsourced textual instructions, while mind2web @xcite , cogagent @xcite , and seeact @xcite try to construct a generalist web agent; webvoyager @xcite can automatically identify interactive elements based on the types of webpage. recently, webarena @xcite deploys a standalone set of multicategory websites in an interactive environment, where visualwebarena @xcite is a subsequent project that built upon we-barena, introducing the reliance on visual cues into the benchmark's design. however, we found that the tasks of existing benchmarks are oversimplified whose completions requiring a single website, which is highly diverged from the natural web browsing tasks and should originally be designed for multihop over a long-horizon setting. ## conclusion, challenges, and outlook we present mmina, a benchmark with three key features: 1) it benchmarks agents on real-world websites with 1,050 multimodal multihop tasks across 14 diverse websites, including experiments with state-of-the-art llms and lmms, as well as human baselines; 2) it introduces a novel holistic evaluation method for multihop tasks, assessing both task-level and hop-level success rates; 3) it proposes a flexible memory-augmented approach to enhance agents' performance by improving their procedural memory.