From Grounding to Manipulation: Case Studies of Foundation Model Integration in Embodied Robotic Systems (2025)

Abstract: Foundation models (FMs) are increasingly applied to bridge language and action in embodied agents, yet the operational characteristics of different integration strategies remain under-explored—especially for complex instruction following and versatile action generation in changing environments. We investigate three paradigms for robotic systems: end-to-end vision-language-action models (VLAs) that implicitly unify perception and planning, and modular pipelines using either vision-language models (VLMs) or multimodal large language models (MLLMs). Two case studies frame the comparison: instruction grounding, which probs fine-grained language understanding and cross-modal disambiguation; and object manipulation, which targets skill transfer via VLA finetuning. Our experiments reveal trade-offs in system scale, generalization and data efficiency. These findings indicate design lessons for language-driven physical agents and point to challenges and opportunities for FM-powered robotics in real-world conditions.

## introduction natural language is emerging as a universal interface for embodied robotics. advances in foundation models (fms) enable robots to follow free-form instructions across perception, reasoning, and motor control, offering the promise of language-grounded autonomy. these models include vision-language models (vlms) @xcite @xcite , multimodal large language models (mllms) @xcite @xcite , and vision-language-action (vla) models @xcite @xcite . however, realizing the promise of languagegrounded autonomy in deployable systems remains highly challenging. robots must (i) map ambiguous instructions to the physical world (instruction grounding), (ii) execute reliably across novel objects, scenes, and morphologies (generalizable execution), and (iii) achieve these goals with limited data (efficient adaptation). how effectively different fm integration strategies address these competing requirements remains under-explored (fig. 1 ). this work offers an empirical study of three integration paradigms: end-to-end vlas mapping language and vision to actions, mllm agents orchestrating perception and control, and modular vlm pipelines pairing perception-specialist fms with task-specific planners (fig. 2 ; table1). we evaluate these paradigms through tabletop case studies that expose their complementary strengths and limitations. two task categories are considered: complex instruction grounding, probing fine-grained understanding and cross-modal disambiguation (sec.3); and object manipulation, assessing skill transfer after vla fine-tuning under distribution shifts, complemented by comparative and ablation analyses (sec.4). our instruction grounding experiments reveal distinct trade-offs across integration strategies. vlm pipelines emphasize interpretability and efficiency, but fall short of peak performance. they struggle with complex instruction grounding yet achieve moderate object grounding-using less than 1% of the parameters required by mllms. in contrast, mllms generalize better on complex instructions but incur substantially higher inference costs, with smaller reasoning-focused models at times outperforming larger ones. vlas, with tightly coupled perception-to-action pathways, streamline action generation but make it difficult to isolate and assess grounding performance. to probe this gap, we evaluate their perception heads as reference points, which also surfaces open questions for vla architecture design. while mllms perform well on grounding tasks, their scale limits practical deployment on robotic platforms. quantization is often adopted as a simple remedy, yet instruction-dependent behaviors emerge, with certain reasoning capabilities degrading disproportionately under naïve compression. this underscores the need for fine-grained quantization strategies. taken together, our findings clarify the trade-offs between model scale and performance and provide practical guidance for building robotic systems under real-world constraints. our object manipulation experiments investigate the skill-adaptation capabilities of vlas in both real-world and simulated settings. we examine how these models transfer manipulation skills under distribution shifts, evaluating their training dynamics, generalization, and robustness. results reveal fragile training and slow adaptation in generalist policies, such as openvla @xcite , compared to task-specific counterparts, underscoring the challenges of real-world deployment." in summary, our main contributions are: • we analyse three fm integration paradigms on shared embodied tasks designed to probe the capabilities and trade-offs of fms. concerning how fms are integrated into robot systems, we identified the following three types of integration strategies (fig. 2 ). in the following, we briefly describe each strategy along with its respective advantages and limitations. ## case studies on instruction grounding natural language instruction grounding translates user intent into actionable goals within a visual scene. our case study evaluates grounding performance under challenging cross-modal disambiguation, highlighting trade-offs and offering guidance for efficient deployment. ## case studies on robotic manipulation now we shift the focus to skill adaptation. in an ideal deployment scenario, a pretrained vlaalready endowed with broad visuomotor skillsshould be retargeted to a new manipulation task with minimal data and fast convergence. we use fine-tuning, the standard practice for adaptation, as a probing lever to evaluate how the state-of-the-art vla models adapt to new tasks and deployment conditions. our experiments highlight the performance of vla models in different settings, offering practical suggestions for practitioners who have to adapt large vlas under tight data, time and compute budgets. ## constraints and future directions despite the promise of foundation models (fms) for enabling embodied agents to perform daily tasks, several critical constraints still hinder their reliable deployment: data scarcity. unlike natural language data that abundantly available on the internet, robotic datasets are costly to collect due to hardware wear, safety risks, and labor-intensive demonstrations. a key direction is improving data efficiency through parameter-efficient adaptation, imitation from unlabeled interaction, and self-supervised pretraining. complementary approaches include leveraging high-fidelity simulators and developing robust sim-to-real transfer pipelines to reduce reliance on large real-world collections.