P ro B ench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks (2025)

Abstract: Solving expert-level multimodal tasks is a key milestone in general intelligence. As the capabilities of multimodal large language models (MLLMs) continue to evolve, evaluation of frontier multimodal intelligence becomes necessary yet challenging. In this work, we introduce ProBench, a benchmark of open-ended user queries encapsulating professional expertise and advanced reasoning. ProBench consists of 4,000 high-quality samples independently collected from professionals based on their productivity demands. It spans across 10 fields and 56 sub-fields, including science, arts, humanities, coding, mathematics, and creative writing. Experimentally, we evaluate and compare 24 latest models using MLLM-as-a-Judge. Our results reveal that although the best open-source models rival the proprietary ones, they all face significant challenges in visual perception, textual understanding, domain knowledge, and advanced reasoning. Our benchmark is publicly accessible at TBC .

## introduction solving expert-level multimodal tasks with multimodal large language models (mllms) represents an important milestone toward achieving humanlevel general intelligence. however, these tasks require mllms to possess strong user query understanding, domain-specific knowledge, and advanced reasoning abilities. ensuring their reliability before deployment necessitates rigorous eval-uation. to address it, we introduce probench, a challenging and automatic evaluation benchmark leveraging mllm-as-a-judge. probench consists of 4,000 queries from professional users, covering diverse productivity demands to assess mllm capabilities in open-ended scenarios (fig. 1 ). one common benchmark to evaluating mllm performance with expert knowledge is mmmu @xcite . while effective for automatic evaluation using predefined answers, these closedended visual question answering benchmarks fail to capture mllm capabilities in open-ended user interactions. specifically, they do not adequately assess mllm ability to follow user instructions or align with human preferences, both of which are fundamental for real-world applications @xcite @xcite . similar limitations apply to other benchmarks, such as mmmu-pro @xcite , mmbench @xcite , and others @xcite @xcite . alternatively, mllm-as-a-judge is employed to automatically evaluate model performance in openended scenarios. however, existing benchmarks fail to rigorously assess mllms on expert-level professional tasks. some @xcite are artificially constructed by a small group of experts, limiting their ability to reflect real-world user interactions. the remaining benchmarks @xcite , such as wildvision, are mostly set in general chat environments and require much less domain knowledge to solve. to fill this gap, in this paper, we aim to design an open-ended benchmark that requires expert-level knowledge for multimodal tasks. our probench is created from high-quality interactions within 100k real-world, professionally crowdsourced multimodal conversations for productivity scenarios. specifically, samples are collected by encouraging users to ask questions related to their daily professional work, which usually require significant expert-level knowledge. this distinction sets our benchmark apart from prior works like wildvision @xcite (fig. 2 ). for a comprehensive evaluation, probench includes three tracks: singleround, multi-round, and multi-linguistic conversations. they respectively span 10 task fields and 56 sub-fields, support 17 languages, and support conversations with up to 13 conversation turns. an overview of probench is presented in fig. 3 . leveraging mllm-as-a-judge (e.g., gpt-4o), we assess 24 leading mllms on probench. our evaluation reveals several key limitations in stateof-the-art mllms: i) current mllms struggle in visual perception, textual understanding, domain knowledge, and advanced reasoning, suffering from tasks like mathematics and planning; ii) multi-linguistic understanding and long-context reasoning during multi-round interaction remain challenging for most existing mllms. our main contributions are summarized as follows: ## probench preliminary. the probench dynamically ranks mllms by employing the elo rating system, implemented through statistical modeling based on direct pairwise model comparisons. in the following, we provide an overview. for further details, please refer to @xcite . given n mllms, an online elo rating system compares model i with rating r i and model j with rating r j using the probability p (y i,@xmath0. here, y i,j denotes the binary outcome, where y i,@xmath1, and y i,@xmath2. the probability is calculated by taking model i as an example, the rating is updated according to the following rule: similarly, k is a constant determining the magnitude of rating adjustments, commonly set to @xmath3. the term s i,j is a scalar representing the actual outcome: 0 for a loss, 0.5 for a tie, and 1 for a win. this updating rule encourages that a higher-rated model gains fewer points for a win, and loses more points for a defeat, while a lowerrated model experiences the opposite effect. however, when using mllm-as-a-judge, the comparison results can be sensitive to model presentation order and confounded by response style variations @xcite . to address these challenges, the probench incorporates the bradley-terry model @xcite as an additional layer atop the elo system. for n mllms and m pairwise comparisons, each round 1 ≤ @xmath4. we have x win @xmath5, while x sty @xmath6.g., word counts, and use of markdown). the bradley-terry model then refines the rating of model i as overview. our paper aims to establish a comprehensive and challenging benchmark for evaluating mllms. the resulting probench is built on two primary components: i) curating high-quality conversations from a crowdsourced dataset, categorized into single-round, multi-linguistic, and multiround tracks; ii) employing mllm-as-a-judge to compare and rank mllms. in total, 3000, 500, and 500 conversations are selected for the single-round, multi-linguistic, and multi-round tracks, respectively, from an initial pool of 100k crowdsourced user-mllm conversations. an overview is presented in fig. 4 . ## related work the evolution of mllm-as-a-judge is largely inspired by the concept of llm-as-a-judge @xcite @xcite , which aims to automatically measure the alignment between mllms and human preferences. while pairwise comparison @xcite is considered as most preferred, it suffers from biases introduced by factors such as the presentation order of mllm outputs, verbosity, and markdown styles. to mitigate these issues, style control has been proposed @xcite , using statistical modeling to de-bias these confounding effects, thereby improving the mllm judges. other approaches, such as few-shot judging, have also been explored, but they face challenges such as reliance on the few-shot example selection and increased evaluation costs @xcite . existing mllm-as-a-judge leaderboards can be specified to @xcite @xcite . however, these often focus on a narrow scope of mllm capability dimensions @xcite , or rely on artificially posed evaluations by a limited number of human experts @xcite , making them inadequate for assessing mllms on professional tasks. consequently, they fail to capture the dynamic nature of real-world human and mllm interactions for a comprehensive assessment of mllm capabilities. in contrast, this work introduces a challenging benchmark, probench, curated from large-scale crowdsourced datasets reflecting real-world professional productivity scenarios. it features three distinct evaluation tracks: single-round, multi-round, and multi-linguistic conversations, across various task fields, offering a robust framework for evaluating mllm performance in real-world scenarios. ## conclusion this paper introduces the probench, which features single-round, multi-round, and multi-linguistic tracks to enable a comprehensive and challenging assessment of the alignment between mllms and human preferences across diverse professional productivity demands. by employing mllm-asa-judge, the benchmark evaluates mllm pairwisely, achieving 79.9% agreement with human expert judgments, and underscoring its reliability. through benchmarking 24 leading mllms, our results reveal significant shortcomings of existing mllms, particularly in visual perception and reasoning. furthermore, models often struggle with multi-linguistic and multi-round tracks, highlighting the challenges of diverse language requirement and complex interactions. it reveals valuable insights for future mllm developments. we hope it inspires successors.