A gent T hink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Driving (2025)

Abstract: Vision-Language Models (VLMs) show promise for autonomous driving, yet their struggle with hallucinations, inefficient reasoning, and limited real-world validation hinders accurate perception and robust step-by-step reasoning. To overcome this, we introduce \textbf{AgentThink} , a pioneering unified framework that, for the first time, integrates Chain-of-Thought (CoT) reasoning with dynamic, agent-style tool invocation for autonomous driving tasks. AgentThink’s core innovations include: \textbf{(i) Structured Data Generation} , by establishing an autonomous driving tool library to automatically construct structured, self-verified reasoning data explicitly incorporating tool usage for diverse driving scenarios; \textbf{(ii) A Two-stage Training Pipeline} , employing Supervised Fine-Tuning (SFT) with Group Relative Policy Optimization (GRPO) to equip VLMs with the capability for autonomous tool invocation; and \textbf{(iii) Agent-style Tool-Usage Evaluation} , introducing a novel multi-tool assessment protocol to rigorously evaluate the model’s tool invocation and utilization. Experiments on the DriveLMM-o1 benchmark demonstrate AgentThink significantly boosts overall reasoning scores by \textbf{53.91\%} and enhances answer accuracy by \textbf{33.54\%} , while markedly improving reasoning quality and consistency. Furthermore, ablation studies and robust zero-shot/few-shot generalization experiments across various benchmarks underscore its powerful capabilities. These findings highlight a promising trajectory for developing trustworthy and tool-aware autonomous driving models.

## introduction recent advances in foundation models have opened new opportunities for autonomous driving @xcite , where pretrained large language models (llms) @xcite and vision-language models (vlms) @xcite @xcite are increasingly employed to enable high-level scene understanding, commonsense reasoning, and decision making. these models aim to transcend traditional perception pipelines, which rely on hand-crafted components such as object detection @xcite , motion prediction @xcite , and rule-based planning @xcite by providing richer semantic representations and broader generalization grounded in web-scale knowledge. many recent approaches recast autonomous driving tasks as visual question answering (vqa) problems, applying supervised fine-tuning (sft) to foundation vlms with task-specific prompts for object identification, risk prediction, or motion planning @xcite @xcite @xcite . this paradigm are widely used in offline pipelines to mine rare driving scenarios, such as identifying intersections with a typical traffic signals or dense pedestrian activities. however, as shown in fig. 2 (a), these models typically treat reasoning as a static input-to-output mapping, neglecting the uncertainty, and verifiability essential to real-world decision making. consequently, they often suffer from poor generalization, hallucinated outputs, and limited interpretability @xcite . to improve robustness and transparency, recent works have explored incorporating chain-ofthought (cot) reasoning into vlms, as shown in fig. 2 (b) . some approaches adopt rigid cot templates @xcite , promoting structured logic at the expense of flexibility. others use open-ended reasoning formats @xcite , but may overfit to token patterns and exhibit shallow or redundant reasoning. moreover, most existing methods rely purely on imitation learning from curated trajectories, lacking the ability to detect knowledge uncertainty or invoke tools for intermediate verification @xcite . these challenges lead to a pivotal question: how can a vlm truly function as a decision-making agent-cognizant of its knowledge boundaries, proficient in verification, and capable of learning from tool-guided feedback? inspiration comes from experienced human drivers who will consult aids like mirrors or gps to refine their judgment when uncertain. similarly, a capable autonomous agent must not only reason explicitly but also recognize its limitations and dynamically employ tools, such as object detectors or motion predictors, to steer its reasoning and decision-making. therefore, we present agentthink, a unified framework for vlms in autonomous driving that models reasoning not as a static output, but as an agent-style process-in which the model learns to utilize tools to generate tool-augmented reasoning chains, verify intermediate steps, and refine its conclusions. as illustrated in fig. 2 (c ), rather than blindly mapping inputs to outputs, agentthink dynamically decides when and how to use tools during inference to support or revise reasoning paths. to enable this behavior, we create a data-trainingevaluation pipeline. first, we construct a structured dataset of tool-augmented reasoning traces. then, we introduce a two-stage training pipeline: (i) sft to warm up reasoning capabilities, and (ii) grpo @xcite , a reinforcement learning (rl)based strategy that refines reasoning depth and tooluse behavior through structured rewards. finally, we propose a comprehensive evaluation protocol beyond answer correctness to assess tool selection, integration quality, and reasoning-tool alignment. as shown in fig. 1 , experiments on the advanced drivelmm-o1 benchmark @xcite demonstrate that agentthink achieves new state-of-the-art performance in both answer accuracy and reasoning score, surpassing existing mod-els. the effectiveness of our approach in cultivating dynamic, tool-aware reasoning is further substantiated by comprehensive ablation studies and robust generalization capabilities across multiple benchmarks. these collective results strongly suggest that empowering vision-language agents with learned, dynamically invoked tool use is pivotal for creating more robust, interpretable, and generalizable systems for autonomous driving. generally, our contributions are as follows: ## methodology our framework is designed to address three key challenges: (1) how to generate reasoning data that incorporates tool usage? (2) how to equip the vlm with the capability to effectively utilize tools? and (3) how to enable reasoning-driven tool invocation during inference, and how to evaluate the model's ability to leverage tools for solving driving-related vision-question-answering tasks? fig. 3 illustrates agentthink's three key components: (i) a scalable pipeline for generating structured, tool-augmented reasoning data; (ii) a twostage training pipeline with sft and grpo to improve reasoning and tool-use abilities; and (iii) a new evaluation methodology focused on assessing the model's effective tool utilization and its impact on reasoning. ## experiments in this section, we conduct extensive experiments to validate the effectiveness of agentthink. our experiments are designed to answer the following core questions: ## conclusion we present agentthink, the first unified framework that tightly fuses cot reasoning with agent-style tool invocation for autonomous driving. with a scalable tool-augmented dataset and a two-stage sft with grpo pipeline, agentthink raises drivelmm-o1 reasoning score from 51.77 to 79.68 and answer accuracy from 37.81% to 71.35%, outperforming the strongest prior model by +5.9% and +9.0%. beyond improved performance, agentthink demonstrates stronger interpretability by making each reasoning step grounded in tool outputs. notably, as a driving-scene vqa system, agentthink aligns with industrial practices where such models are used off the control loop, for example, for mining corner cases or providing high-level feedback in a dual-system setup, and thus operating under relaxed latency constraints compared to real-time planning. results validate that coupling explicit reasoning with learned tool use is a promising path toward safer, more robust language-model-centric driving tasks. we believe this framework lays the foundation for building trustworthy vlm-based agents capable of generalizing to complex, dynamic real-world driving environments.