X - W eb A gent B ench: A Multilingual Interactive Web Benchmark for Evaluating Global Agentic System (2025)

Abstract: Recently, large language model (LLM)-based agents have achieved significant success in interactive environments, attracting significant academic and industrial attention. Despite these advancements, current research predominantly focuses on English scenarios. In reality, there are over 7,000 languages worldwide, all of which demand access to comparable agentic services. Nevertheless, the development of language agents remains inadequate for meeting the diverse requirements of multilingual agentic applications. To fill this gap, we introduce X-WebAgentBench, a novel multilingual agent benchmark in an interactive web environment, which evaluates the planning and interaction performance of language agents across multiple languages, thereby contributing to the advancement of global agent intelligence. Additionally, we assess the performance of various LLMs and cross-lingual alignment methods, examining their effectiveness in enhancing agents. Our findings reveal that even advanced models like GPT-4o, when combined with cross-lingual techniques, fail to achieve satisfactory results. We hope that X-WebAgentBench can serve as a valuable benchmark for multilingual agent scenario in real-world applications.

## introduction large language models (llms) have demonstrated remarkable success across various natural language processing (nlp) tasks @xcite @xcite @xcite , particularly in the development of language agents @xcite @xcite . these agentic systems employ various approaches to interact effectively with the real world. typically, react @xcite combines reasoning traces with task-specific actions to mitigate error propagation during chain-of- @xcite , while the multilingual task scores are obtained through evaluation on our own benchmark. thought reasoning and interaction, enhancing the reliability of decision-making processes. furthermore, reflexion @xcite incorporates feedback to induce better interaction. moreover, metagpt @xcite integrates human workflows to decompose complex tasks in software programming, demonstrating efficiency and scalability in collaborative environments. to better evaluate and understand the capabilities of agents, it emerges a series of research to provide diverse interactive benchmarks to evaluate the ability of language agents in complex environments, such as mind2web @xcite , osworld @xcite , and webarena @xcite . despite the significant advancements in current language agent benchmarks, most developments remain centered on english-language environments. actually, with globalization acceler-benchmark # language # tasks interactivity multilingual instruction multilingual environment miniwob++ @xcite 1 100 ✔ ✘ ✘ russ @xcite 1 80 ✔ ✘ ✘ websrc @xcite 1 400k ✔ ✘ ✘ webshop @xcite 1 12,087 ✔ ✘ ✘ webshop-core @xcite 1 500 ✔ ✘ ✘ mind2web @xcite 1 2,350 ✘ ✘ ✘ mind2web-live @xcite 1 542 ✔ ✘ ✘ webarena @xcite 1 821 ✔ ✘ ✘ omniact @xcite ating, the real-world application of language agents increasingly involves multilingual contexts @xcite . for example, as illustrated in figure 1 , users engaging in global e-commerce, such as shopping on amazon, tend to rely on their native languages for searching, selecting, and purchasing products. yet, when mainstream language agents are employed in these scenarios, a critical issue emerges: current research often overestimates their effectiveness. specifically, as illustrated in figure 1 , multilingual performance lags behind english-only performance by over 20%. insights gained from english environments are challenging to generalize to multilingual contexts, which blocks the development of multilingual agents. moreover, there is a notable absence of standardized benchmarks for evaluating language agents in multilingual scenario. without such benchmarks, identifying gaps, limitations, and constraints in existing multilingual agentic systems remains challenging, hindering their global development. motivated by this, to fill this gap, we introduce x-webagentbench, a multilingual interactive website environment benchmark comprising 14 languages, 2,800 instructions, and 589,946 products. in comparison to previous benchmarks (see moreover, to investigate the performance and relevant drawbacks of language agents in multilingual scenarios, we conduct comprehensive experiments on x-webagentbench, yielding the following key findings: (1) for larger llms, advanced cross-lingual alignment methods can significantly enhance multilingual performance. (2) for smaller llms, translating multilingual environments into english can effectively mitigate the limitations of their multilingual capabilities. (3) the simple integration of existing agent-based and cross-lingual strategies fails to adequately address the challenges posed by multilingual agent systems. in summary, our contributions can be summarized as: to facilitate the further research, our code will be available at https://github.com/ wpengxs/x-webagentbench . ## x-webagentbench construction this section describes the construction of x-webagentbench, including data preparation ( § 2.1), multilingual instruction construction ( § 2.2), multilingual envrionment construction ( § 2.3), and quality check ( § 2.4). further, we will provide the detailed data statistics ( § 2.5). ## analysis in this section, we analyze the findings from both the multilingual( § 4.1 and § 4.2) and agentic perspectives( § 4.3 and appendix § h) as follows: ## exploration following @xcite , we select 6 representative languages based on language statistics in com-moncrawl, including high-(de, ru), mid-(tr, vi), and low-resource (bg, hi) languages for exploration. our study examines the effects of multilingual factors ( § 5.1, § 5.2, § 5.3) and agentic factors ( § 5.4, § 5.5, and appendix f). ## related work to evaluate the interaction capabilities of llms, researchers focus on benchmarks in interactive environments @xcite @xcite . earlier, miniwob++ @xcite proposes a suite of web-browser tasks with over 100 web interaction environments. web-src @xcite further integrates web pages and instructions for structural reading comprehension evaluation. androidenv @xcite define custom tasks on the android operating system, enabling realistic simulation of an android device. after that, webshop @xcite emphasizes interactive tasks based on given instructions in e-commerce environments. furthermore, mind2web @xcite extends agentic evaluation to 2,000 tasks across 137 websites. webarena @xcite constructs a highly realistic and reproducible environment for language agents, which aims to boost the development of robust agents. however, current interactive environments are primarily in english, which limits the development of multilingual agents. we are the first attempt to construct a multilingual interactive benchmark, x-webagentbench, which quantifies the interaction performance of language agents. ## conclusion to address the multilingual evaluation of language agents, we introduce x-webagentbench, a benchmark comprising 14 languages, 2,800 text instructions, and 589,946 products. through extensive experimentation on x-webagentbench, we derive several novel insights into multilingual language agentic systens. we hope that x-webagentbench can serve as an effective benchmark, providing valuable reference for future research.