All-in-one: Understanding and Generation in Multimodal Reasoning with the MAIA Benchmark (2025)

Abstract: We introduce MAIA (Multimodal AI Assessment), a native-Italian benchmark designed for fine-grained investigation of the reasoning abilities of visual language models on videos. MAIA differs from other available video benchmarks for its design, its reasoning categories, the metric it uses, and the language and culture of the videos. MAIA evaluates Vision Language Models (VLMs) on two aligned tasks: a visual statement verification task, and an open-ended visual question-answering task, both on the same set of video-related questions. It considers twelve reasoning categories that aim to disentangle language and vision relations by highlighting the role of the visual input. Thanks to its carefully taught design, it evaluates VLMs’ consistency and visually grounded natural language comprehension and generation simultaneously through an aggregated metric revealing low results that highlight models’ fragility. Last but not least, the video collection has been carefully selected to reflect the Italian culture, and the language data are produced by native-speakers.Data available at *[GitHub](https://github.com/Caput97/MAIA-Multimodal_AI_Assessment.git).*

## introduction vision and language models entered the computer vision and nlp scenes more than a decade ago pushed by theoretical (e.g., @xcite and application-oriented (e.g., @xcite motivations. their success on combining image and text has been monitored and summarized in various surveys, from the earlier ones on visual question answering @xcite to the more recent ones focusing on visually grounded llms (e.g., @xcite . researchers have always felt the need to target vision and language models (vlms) shortcomings, developing carefully designed benchmarks consisting of a suit of vl tasks to evaluate a variety of 1 data available at github. capabilities @xcite . the minimal pair task, contrasting a caption with its foil @xcite has been applied to large-scale linguistic phenomena @xcite and extended to highlight weakness of vlms on video qa @xcite . this trend focuses on visually grounded natural language understanding @xcite . today vlms are trained to generate text and are known to excel at it. we argue that the evaluation of natural language generation (nlg) and natural language understanding (nlu) competence should always be pursued together: an agent that can answer questions about an event must understand it too. yet, existing benchmarks tend to treat comprehension and generation separately, often relying on independent datasets and evaluation protocols. this fragmented design prevents an assessment of a model's robustness, in other words, its ability both to understand and generate visually grounded text. moreover, success in nlu should be claimed only through evaluation regimes that monitor models' consistency across answers, showing they are insensitive to surface variations. to address these limitations, we present maia: a competence-oriented benchmark consisting of two paired tasks -multiple-choice visual statement verification (vsv) and open-ended visual question answering (oevqa) -containing aligned datapoints. for instance, in the example in figure 1 , for the video showing a pizza cooking in a wood-fired oven, the vsv contains the true and false statements (ts and tf) a: there is no cake in the video and b: there is a cake in the video, and the oevqa contains the (aligned) question what is the cake made of?. such interleaved data let us evaluate models' robustness: we evaluate a model positively, only if it performs correctly both on the visual statement verification task (nlu) -it selects there is no cake in the video -and on the open-ended vqa (nlg) by generating something like there is a pizza, not a cake; while we evaluate it negatively, if it performs correctly only on one of the two tasks. moreover, the vsv is organized in pools containing 8 ts-fs pairs that differ only on the surface, letting us evaluate the model's consistency. we categorize questions based on the reasoning they elicit; for instance, the question in figure 1 is out-of-scope. maia spans twelve reasoning categories, helping highlight the role of language and visual modalities across them. finally, it implements an all-in-one evaluation philosophy that lets us evaluate both models' robustness and consistency through its aggregate metric. last but not least, maia is based on native italian videos with language data obtained through human annotations and complemented by semi-automatic data augmentation. to the best of our knowledge, this is the first benchmark for the italian language on videos. contributions. in the paper, we: (i) present maia, the first italian benchmark specifically designed to assess the reasoning abilities of vlms on video data; (ii) evaluate multiple vlms, highlighting how their performance and their reliance on linguistic or visual cues varies across reasoning categories; (iii) demonstrate the importance of evaluating models from multiple perspectives, not only in terms of their competencies, but also their robustness and consistency; @xcite propose a novel metric for evaluating visually grounded comprehension and generation simultaneously. video reasoning benchmarks. widely used benchmarks, such as agqa @xcite and mvbench @xcite focus on explicit visual elements (e.g., entity, action, and the spatio-temporal reasoning involving them), instead maia's categories focus on the interplay between language and vision, especially when this relation is implicit or must be inferred, a dimension largely neglected in prior video qa benchmarks. @xcite includes multiple-choice and open-ended data points from entirely independent data sets with different origins and content, and reports the average performance across distinct tasks. instead, maia's nlu and nlg data points are aligned, a unique feature of maia framework, as such it introduce an aggregate accuracy metric specifically designed to ensure that the performance of the model is evaluated consistently across multiple-choice and open-ended questions derived from the same underlying data. there are few other video-text benchmarks including both these formats (e.g., @xcite ; peng et al. ( 2024 )), however, to the best of our knowledge, none of them attempt to define a unifying metric, as we do in maia. ## experimental setting we run several experiments to test modern vlms on the maia benchmark in a zero-shot setting. to capture different vlm behaviors, strengths and limitations, we defined two tasks, aligned on the same datapoints: a multiple choice task, visual statement verification (vsv), and a generative task, open-ended visual question answering (oevqa). ## aggregating understanding and generation a distinctive feature of maia is the alignment between the vsv and oevqa tasks: both are grounded in the same question and the same video. while section 5 reported their results separately, we now combine them into a unified evaluation framework that jointly tests comprehension and generation abilities in vlms. we introduce aggregate accuracy (agg-acc), a metric rewarding models that: (i) consistently select the correct statement (ts) over all 8 ts-fs pairs in task 1, and (ii) generate a correct answer to the same question in task 2, according to our llm-as-a-judge evaluation. the idea is to capture overlapping abilities (i.e. knowledge and reasoning) for the two aligned tasks, evaluating models' robustness. aggregate accuracy is defined as follow: ## conclusion we introduce maia, a benchmark designed for fine-grained investigation of the reasoning abilities of vlms on videos. maia has two aligned tasks: a visual statement verification task (nlu), and a open-ended visual question answering task (nlg), both on the same set of video related questions. first, we provided a in-depth analysis of the two tasks independently, showing the importance of evaluating model with answer-pools to account for model consistency. then, we couple comprehension and generation in an aggregated evaluation framework, arguing that the aggregated "all-in-one" understanding&generation task is a challenging and natural setting toward a more objective vlm evaluation, as it reveals inconsistencies within the same task and a lack of robustness across aligned tasks, even in larger models. as for the future, it would be interesting to see whether our framework promote models that undergo learning paradigms tightly integrating these two capabilities, as in gul and artzi (2024).