V ehicle W orld: A Highly Integrated Multi-Device Environment for Intelligent Vehicle Interaction (2025)

Abstract: Intelligent vehicle cockpits present unique challenges for API Agents, requiring coordination across tightly-coupled subsystems that exceed typical task environments’ complexity. Traditional Function Calling (FC) approaches operate statelessly, requiring multiple exploratory calls to build environmental awareness before execution, leading to inefficiency and limited error recovery. We introduce VehicleWorld, the first comprehensive environment for the automotive domain, featuring 30 modules, 250 APIs, and 680 properties with fully executable implementations that provide real-time state information during agent execution. This environment enables precise evaluation of vehicle agent behaviors across diverse, challenging scenarios. Through systematic analysis, we discovered that direct state prediction outperforms function calling for environmental control. Building on this insight, we propose State-based Function Call (SFC), a novel approach that maintains explicit system state awareness and implements direct state transitions to achieve target conditions. Experimental results demonstrate that SFC significantly outperforms traditional FC approaches, achieving superior execution accuracy and reduced latency. We have made all implementation code publicly available on GitHub.

## introduction api agents represent a paradigm shift in intelligent interaction by combining large language models' cognitive capabilities with external tools' execution capabilities @xcite @xcite . these agents distinguish themselves through autonomous decision-making, sophisticated reasoning abilities, and seamless tool interaction, transcending tradi-user: the weather is hot, turn on the air conditioner, then play my music collection. ## vehicleworld to simulate realistic intelligent cockpit environments, we introduce vehicleworld, the first comprehensive environment for the automotive domain that provides real-time state information during agent execution and supports precise evaluation of vehicle agent behaviors. ## state-based function call through the precise modeling of system states in vehicleworld, we are able to analyze how the system evolves before and after function calls. we observe that in many scenarios, tasks can be accomplished by generating short state transitions instead of full api call sequences. based on this insight, we propose state-based function call (sfc), which predicts the target system state and generates ... @classmethod def set_volume(): """set environment volume.""" ... @classmethod def search_module(): """get available modules.""" ... @classmethod def search_api(modules): """get the module's api.""" ... ## vehicle benchmark to evaluate model performance in vehicleworld, we developed a comprehensive benchmark comprising natural, diverse, and challenging intelligent cockpit scenarios. our approach includes a specialized data generation pipeline that maintains state continuity across interactions and an evaluation methodology that analyzes state transition patterns. please create a scenario. init() @xmath0.init()</@xmath1.init() radio_play() whether the query is executable given the state whether the api call is aligned with the query whether the query results in any modification of the @xmath2.init()</@xmath3: true • "temperature": 20 door • "is_locked": false • "status": "closed" ## experiments 6.1 experimental setup methods. we adopted three prompting-based decision-making strategies in our experiments under both the function call (fc) and state-based function call (sfc) paradigms: react, react + reflection, and react without examples. detailed descriptions are provided in appendix a.2. models. we analyzed a range of open-source and commercial models, detailed in appendix d. ## conclusion we introduced vehicleworld, the first comprehensive multi-device environment for intelligent vehicle interaction that accurately models the complex, interconnected systems in modern cockpits. this environment enables precise evaluation of agent behaviors by providing real-time state information during execution. building on this foundation, we constructed the first benchmark for evaluating vehicle agents, establishing standardized metrics for comparing different llms. our analysis revealed a critical insight: directly predicting environment states proves more effective than predicting function calls in complex, state-dependent systems. this observation led to our state-based function call (sfc) approach, which maintains explicit awareness of system state and implements direct state transitions. experimental results demonstrate that sfc significantly outperforms traditional function calling, enhancing models' ability to interpret user intentions while reducing erroneous function calls. furthermore, we identified the complementary strengths of both paradigms: sfc excels at device selection due to its global environmental percep-tion, while fc's high-level apis are more efficient for complex state transitions. this led us to propose a hybrid fc+sfc approach, which leverages sfc for device selection before using fc for api execution. our experiments confirm that this hybrid method achieves the highest end-to-end accuracy. these advancements establish a foundation for future research in intelligent cockpit systems and offer valuable insights for agent design in other complex, multi-device environments.