Revisiting 3 D LLM Benchmarks: Are We Really Testing 3 D Capabilities? (2025)

Abstract: In this work, we identify the “2D-Cheating” problem in 3D LLM evaluation, where these tasks might be easily solved by VLMs with rendered images of point clouds, exposing ineffective evaluation of 3D LLMs’ unique 3D capabilities. We test VLM performance across multiple 3D LLM benchmarks and, using this as a reference, propose principles for better assessing genuine 3D understanding. We also advocate explicitly separating 3D abilities from 1D or 2D aspects when evaluating 3D LLMs.

## introduction recent advances in large language models @xcite have led to the development of vision language models (vlms) @xcite . to overcome their lack of grounding in the real 3d physical world @xcite , researchers developed 3d llms @xcite for 3d processing. given the extreme scarcity of 3d training data @xcite , many approaches leverage existing llms and vlms to generate annotations for 3d data @xcite @xcite . this reliance on 2d and language (1d) priors raises a fundamental question: what capabilities do 3d llms possess that truly differentiate them from 2d vlms? to explore this, we revisit some benchmarks used for 3d llm evaluation. current 3d llms are primarily evaluated on q&a or captioning tasks @xcite , rather than specific downstream tasks like object detection @xcite , as llms provide a general-purpose language interface @xcite . as shown in figure 1 , we find some tasks can be easily solved by 2d vlms, indicating that * equal contribution. they fail to effectively assess the unique capabilities of 3d models. we refer to this phenomenon as 2d-cheating. specifically, we render point clouds into images to test the performance of vlms on 3d llm benchmarks. experimental results indicate that vlms can significantly exceed the state-of-theart (sota) performance of 3d models on certain benchmarks. however, on other benchmarks, even with enhanced information, they still fail to achieve the best performance. we argue that tasks where vlms perform poorly compared to 3d llms are those that truly involve unique 3d capabilities. based on analysis, we (1) propose several principles for designing benchmarks that effectively evaluate true 3d capabilities, and (2) advocate for separating the evaluation of 3d capabilities from 2d and 1d aspects when assessing 3d llms. ## principles for effective 3d evaluation based on the analysis, we propose the following principles for effective 3d capabilities evaluation of 3d llms. * we also apply these principles to redesign a task, as shown in figure 6 . ## conclusion this paper identifies the 2d-cheating problem in current 3d llm evaluation, compares vlm performance with 3d llms across various benchmarks, and provides principles for assessing real 3d capabilities in 3d llms. however, this does not imply that 3d llms can ignore the ability to solve tasks vulnerable to 2d-cheating. therefore, we emphasize the need to deliberately decouple the evaluation of 3d capabilities from 1d and 2d capabilities to ensure that core 3d evaluations are not confused or overlooked.