MASSIVE -Agents: A Benchmark for Multilingual Function-Calling in 52 Languages (2025)

Abstract: We present MASSIVE-Agents, a new benchmark for assessing multilingual function calling across 52 languages. We created MASSIVE-Agents by cleaning the original MASSIVE dataset and then reformatting it for evaluation within the Berkeley Function-Calling Leaderboard (BFCL) framework. The full benchmark comprises 47,020 samples with an average of 904 samples per language, covering 55 different functions and 286 arguments. We benchmarked 21 models using Amazon Bedrock and present the results along with associated analyses. MASSIVE-Agents is challenging, with the top model Nova Premier achieving an average Abstract Syntax Tree (AST) Accuracy of 34.05% across all languages, with performance varying significantly from 57.37% for English to as low as 6.81% for Amharic. Some models, particularly smaller ones, yielded a score of zero for the more difficult languages. Additionally, we provide results from ablations using a custom 1-shot prompt, ablations with prompts translated into different languages, and comparisons based on model latency.

## introduction large language models (llms) and foundation models (fms) are now commonly used as agents, where the term agent is used to describe a software module that has access to real-world tools and can determine if using a tool is necessary to achieve the user's goal. the agent must choose from a list of available tools, generate the correct syntax to successfully invoke the tool, and handle the tool's response @xcite @xcite . the agent also often then formulates human-friendly responses. existing benchmarks for agentic performance are often limited to evaluating agents with queries only in english, such as the berkeley function calling leaderboard (bfcl) @xcite , api-bank @xcite , nexus function calling (team, 2023), and τ -bench @xcite . to help address this gap, we adapted a portion of the massive dataset @xcite into a new benchmark we call massive-agents. our contributions include the following: ## method building upon the massive @xcite benchmark published in 2022, we created the massive-agents multilingual natural language understanding dataset designed for evaluating fms agentic capabilities. in the following, we first provide more details on our dataset specification and the original massive dataset. subsequently, we describe the filtering and transformations we conducted on the massive dataset to create massive-agents . ## benchmarking in this section, we present a comprehensive evaluation of various language models performance on the massive-agents benchmark. specifically, we assess fms ability to generate accurate function calls across multiple languages, test splits, and different prompting strategies. limiting our analysis to the family of non-thinking models available in amazon bedrock at the time of writing, our evaluation encompasses open-source models ranging from 1b to 405b parameters @xcite , and proprietary models, including releases such as claude 3.5 v2 sonnet @xcite , and nova models @xcite . we examine both zero-shot and one-shot scenarios across all languages, using either an initial english-only prompt or prompts translated into different target languages. in sec. 3.1, we detail our experimental setup, including our prompting strategies and evaluation metrics. sec. 3.2 presents our main results, analyzing performance across different languages and models. finally, in sec. 3.3, we conduct ablation studies to understand the impact of different prompting strategies and their interaction with various languages. ## conclusion when discussing artificial general intelligence, emphasis is often given to a model's performance across a breadth of tasks and domains. we contend that full multilingualism across all of the earth's languages should also be a requirement. mas-sive and subsequently massive-agents are unique in that the languages are sampled across a range of typologies and language resource levels. we hope that this new benchmark will prove valuable to language model researchers worldwide toward the goal of greater multilinguality of future foundation models.