NUMINA : A Natural Understanding Benchmark for Multi-dimensional Intelligence and Numerical Reasoning Abilities (2025)

Abstract: Recent advancements in 2D multimodal large language models (MLLMs) have significantly improved performance in vision-language tasks. However, extending these capabilities to 3D environments remains a distinct challenge due to the complexity of spatial reasoning. Nevertheless, existing 3D benchmarks often lack fine-grained numerical reasoning task annotations, limiting MLLMs’ ability to perform precise spatial measurements and complex numerical reasoning. To address this gap, we introduce NUMINA, the first Natural Understanding benchmark for Multi-dimensional Intelligence and Numerical reasoning Abilities to enhance multimodal indoor perceptual understanding. NUMINA features multi-scale annotations and various question-answer pairs, generated using NUMINA-Flow, an automated annotation pipeline that integrates LLM rewriting and rule-based self-verification. We evaluate the performance of various state-of-the-art LLMs on NUMINA following the Chat-Scene framework, demonstrating that current LLMs struggle with multimodal numerical reasoning, particularly in performing precise computations such as distance and volume estimation, highlighting the need for further advancements in 3D models. The dataset and source codes can be obtained from https://github.com/fengshun124/NUMINA.

## introduction multimodal perception has gained increasing attention in recent years. leveraging large language models (llms), 2d multimodal llms (mllms) have progressed rapidly, leading to groundbreaking achievements in image classification @xcite @xcite and reasoning @xcite . to extend the applications of mllms to 3d domains, researchers have strived to unlock the potential of mllms in spatial intelligence to enable practical functionalities, such as autonomous driving @xcite , indoor navigation @xcite , and route planning @xcite . nevertheless, understanding accurate spatial information in 3d environments remains a significant challenge due to the exponential increase in structural complexity introduced. for instance, when positioning a cabinet in a living room, one must consider not only the availability of space in the front and back but also factors such as excessive vertical space occupancy and efficient space utilization. consequently, equipping mllms with spatial perception capabilities, including precise measurement and localization, is critical for improving their real-world applicability. achieving precise spatial reasoning necessitates well-annotated 3d vision-language datasets. however, existing benchmarks offer only coarse, global annotations, which limit their applicability for finegrained reasoning. datasets like scanrefer @xcite , @xcite , and scanqa @xcite advance 3d tasks by supporting object localization, navigation, and question answering, but they lack multimodal nu-merical reasoning incorporating spatial precision. unlike simpler spatial understanding tasks, numerical reasoning involves complex scene interpretation, requiring detailed annotations of object distances, sizes, and positions, which current datasets do not provide. additionally, these benchmarks predominantly focus on textual outputs, limiting the diversity of question formats and response types, and consequently, their utility in developing models for sophisticated multimodal reasoning. this highlights the need for a more comprehensive, numerically enriched 3d vision-language benchmark. to this end, we present numina, the first natural understanding benchmark for multi-dimensional intelligence and numerical reasoning abilities. as shown in figure 1 , numina stands out by focusing on fine-grained spatial understanding and numerical reasoning in indoor environments, featuring 74,526 question-answer (qa) pairs across three response types with increasing difficulty: fact validation (judgment questions), prompt matching (multiple-choice questions), and numerical inference (value-output questions). each category is evaluated using tailored metrics and can be further subdivided. these tasks, spanning from simple verification to complex numerical reasoning, ensure that progress on numina reflects genuine advances in 3d multimodal reasoning rather than overfitting to a single question format. compared with existing benchmarks, as shown in name #pairs question types non-num. num. sub-cat. to generate high-quality data samples with numerical ground truth. it also incorporates diverse qa templates, adopts the convex hull distance measurement that aligns with human intuition, and ensures a balanced data distribution to prevent models from exploiting shortcuts or biases. to rigorously assess the utility of numina and establish a baseline, we utilize several mainstream open-source llms, such as vicuna @xcite and @xcite , to serve as decoders within the chat-scene @xcite framework, enabling the simultaneous processing of 3d point clouds, 2d images, and textual inputs. experimental results reveal substantial challenges of existing llms in 3d numerical inference and spatial reasoning, with accuracy falling below 3% under a 5% error threshold accuracy (ta@5) in distance and volume estimation tasks. while the models perform well in non-numerical evaluations, their accuracy declines considerably in distancerelated tasks, achieving only about 54%, close to random selection. notably, no single model consistently outperforms across all task categories, highlighting diverse strengths depending on the specific evaluation criteria. ## numina and numina-flow this section describes in detail the automatic construction pipeline numina-flow for the numina dataset. as illustrated in figure 2 , numina-flow generates numerical data by extracting numerical ground truth (ngt) from scannet @xcite . it leverages gpt-4o @xcite to create question-answer templates, followed by rule-based and manual verification. the generated templates are validated by human evaluators to ensure clarity and ambiguity, making them comprehensible to both humans and models. additionally, variations in expression and syntactic structure are incorporated to promote generalization across different reasoning patterns. to enhance question diversity, non-numerical questions are further generated using qwen2.5-72b (yang et al., 2024). ## conclusion we introduce numina, a natural understanding benchmark for multi-dimensional intelligence and numerical reasoning abilities, which is designed to advance multimodal indoor perceptual understanding. numina provides a diverse dataset with multi-scale annotations and a variety of questionanswer pairs spanning multiple task types. this dataset is generated using numina-flow, an automated annotation platform that leverages llm rewriting and rule-based self-verification. following the chat-scene paradigm, we replace llms within the architecture and jointly train and evaluate all tasks. experimental results indicate that current llms exhibit significant limitations in multimodal numerical reasoning, particularly in tasks requiring fine-grained computational precision, such as quantity, distance, and volume estimation. in the future, we will develop further advancements in numerical and spatial reasoning for multimodal ai systems.