3 D -Aware Vision-Language Models Fine-Tuning with Geometric Distillation (2025)

Abstract: Vision-Language Models (VLMs) have shown remarkable performance on diverse visual and linguistic tasks, yet they remain fundamentally limited in their understanding of 3D spatial structures.We propose Geometric Distillation, a lightweight, annotation-free fine-tuning framework that injects human-inspired geometric cues into pretrained VLMs without modifying their architecture.By distilling (1) sparse correspondences, (2) relative depth relations, and (3) dense cost volumes from off-the-shelf 3D foundation models (e.g., MASt3R, VGGT), our method shapes representations to be geometry-aware while remaining compatible with natural image–text inputs.Through extensive evaluations on 3D vision-language reasoning and 3D perception benchmarks, our method consistently outperforms prior approaches, achieving improved 3D spatial reasoning with significantly lower computational cost.Our work demonstrates a scalable and efficient path to bridge 2D-trained VLMs with 3D understanding, opening up wider use in spatially grounded multimodal tasks.

## introduction vision-language models (vlms) (e.g., clip @xcite , align @xcite , and blip @xcite ), trained on large-scale image-text datasets, have demonstrated competitive performance on diverse multimodal tasks @xcite @xcite . despite their progress, these models struggle with understanding 3d spatial structures @xcite @xcite @xcite @xcite . specifically, vlms remain limited in grounded spatial reasoning tasks such as depth ordering, occlusion, or object layout in a scene @xcite @xcite . this limitation stems from their reliance on 2d projections, which lack depth cues and multiview supervision @xcite @xcite . it is illustrated in figure 1 , where features of standard vlms like clip incorrectly predict relative depth due to their limited 3d awareness. these shortcomings greatly hinder applications requiring spatial reasoning, including navigation, scene understanding, and robotic planning @xcite @xcite . to address this, recent work has explored injecting 3d priors into vlms. fit3d @xcite reconstructs 3d scenes from multi-view images using gaussian splatting @xcite , and then aligns vlm's features with those rendered 3d views. multiview equivariant finetuning (mef) @xcite improves 3d equivariance by reinforcing feature consistency across rendered views of the same object. spatialvlm @xcite improves its spatial reasoning abilities by generating billions of synthetic spatial question-answer pairs to train vlms. despite these advancements, existing methods suffer from notable drawbacks. fit3d incurs a high computational cost and suffers from semantic degradation due to its reliance on explicit 3d reconstruction. mef depends on 3d objectcentric datasets, which restricts its generalizability to real-world scenes. spatialvlm requires extensive synthetic data generation and task-specific tuning, making it resource-intensive and less flexible. these limitations motivate the need for more efficient and generalizable approaches to endow vlms with robust 3d awareness. we propose geometric distillation, a lightweight and glanannotation-free fine-tuning framework that enriches 3d spatial understanding in vlms. our approach introduces supervision signals aligned with human perceptual strategies, derived from pretrained 3d foundation models such as mast3r @xcite and vggt @xcite as in figure 2 (a) -(d). first, we supervise the vlm to align features at sparse correspondences that are visually stable and semantically meaningful regions, such as object corners or room boundaries, derived from pretrained 3d foundation models without any explicit 3d annotations. these locations provide strong geometric anchors across views, and featurelevel matching at these points encourages the model to learn consistent and viewpoint-invariant representations. second, we supervise relative depth reasoning through ordinal comparisons both within and across views. this reflects the human tendency to reason in relative terms and aligns with the way spatial relationships are expressed in language @xcite . lastly, we incorporate dense cost volume alignment, which captures soft correspondences across views by fully exploiting the geometric priors and warping relationships @xcite provided by 3d foundation models, thereby enabling the model to learn fine-grained geometric consistency. these signals collectively reshape the visual representations into a geometry-aware space that better supports grounded spatial reasoning and improves vlm performance on 3d-aware tasks as shown in figure 2 (e), (f). additionally, since our approach operates without modifying the vlm's architecture and retains compatibility with natural image-text inputs, it preserves the strong generalization capabilities of the original model. to overcome these limitations, we draw inspiration from human 3d spatial perception. humans infer depth and structure from sparse relational cues such as occlusions, relative size, and perspective, rather than absolute measurements @xcite @xcite . in addition, spatial relationships are often expressed in language using relative terms (e.g., "next to the table", "behind the sofa") rather than absolute metric units, suggesting that the reasoning is both perceptually and linguistically grounded. these observations suggest that incorporating human-inspired geometric cues into vlm can enhance their spatial reasoning abilities. our approach enhances the model's ability to infer spatial relations, such as object proximity, without explicit 3d labels or costly reconstruction. we demonstrate consistent improvements across a range of 3d-aware tasks, including semantic correspondence, depth estimation, and 3d visual question answering. our method outper-forms strong baselines on benchmarks such as pf-pascal, tap-vid, and scanqa, illustrating both the effectiveness and scalability of our approach. ## proposed method we propose a geometric knowledge distillation framework that transfers 3d spatial understanding from high-performance 3d foundation models such as mast3r @xcite and vggt @xcite into a pretrained visionlanguage model (vlm) @xcite without requiring any ground truth 3d annotations. inspired by human perception, which infers spatial structure by integrating visual cues from multiple viewpoints, our method uses paired images, {i v 1 , i v 2 }, of the same scene captured from different perspectives v 1 and v 2 . from these image pairs, we extract geometric signals including sparse correspondences, ordinal depth relations, and viewpoint-induced disparities, which guide the vlm to learn geometry-aware representations. an overview of our framework is illustrated in figure 3 . our framework obtains these geometric cues using a teacher model that generates pseudo-3d supervision from image pairs. specifically, we utilize the following information provided by 3d foundation models: (i) sparse correspondences for matching 3d points across views, (ii) estimated depth maps dv 1 , dv 2 for each viewpoint, and (iii) a dense cost volume, c v 1 →v 2 , representing patch-level features similarity between two viewpoints. these heterogeneous signals serve as supervision for three complementary objectives: sparse correspondence matching, relative depth learning using both intraview and inter-view comparisons, and alignment of dense feature similarity. combined, they enrich the model's multimodal representations and facilitate 3d-aware reasoning in complex scenes. ## conclusion we present geometric distillation, a lightweight and annotation-free framework that enhances 3d spatial awareness and reasoning in vlms. by distilling rich geometric signals such as multiview correspondences, relative depth relations, and dense cost volumes from high-capacity 3d foundation models like mast3r and vggt, our method equips pretrained 2d vlms with robust 3d perception. without requiring architectural modifications or explicit 3d annotations, our approach improves state-of-the-art results across diverse spatial reasoning tasks, including semantic correspondence, depth estimation, and 3d visual question answering. extensive experiments demonstrate that our method consistently outperforms prior approaches while offering greater scalability and generalization to real-world scenes. our work highlights an effective pathway to bridge the gap between 2d vision-language understanding and 3d perception.