V is TRA : Visual Tool-use Reasoning Analyzer for Small Object Visual Question Answering (2025)

Abstract: This study proposes VisTRA (Visual Tool-use Reasoning Analyzer), a framework for analyzing how Visual Language Models (VLMs) utilize tools in VQA tasks involving small objects in high-resolution images. While tools like object detection and zoom functionality are essential for small object VQA, their potential errors necessitate careful verification of outputs. Our framework provides systematic evaluation of VLMs’ tool-use capabilities through analysis of verification patterns. Using the V* bench dataset, we find that direct acceptance of tool outputs correlates with decreased VQA accuracy, while lower-performing models exhibit higher frequencies of cyclic verification loops. These findings offer insights for improving tool verification mechanisms in VLM architectures focused on small object detection tasks.

## introduction a well-known challenge in vqa is handling high-resolution images containing small objects @xcite ). the conventional approach of feeding entire images into vqa systems faces an inherent limitation: when processing images holistically, small objects that occupy only a minimal portion of the total pixels become practically indistinguishable in the global representation. this fundamental constraint of whole-image processing leads to system-1 * these two authors contributed equally to this work. atic failures in capturing small objects, resulting in degraded response accuracy. several approaches have been proposed to address the small object challenge, including analyzing magnified portions of images @xcite , additional training with small object-focused datasets @xcite , and multi-scale learning that preserve native resolution while extracting features from numerous image sub-crops @xcite . while these methods improve recognition accuracy, they introduce new challenges such as increased architectural complexity and limitations in detection targets. a promising direction for addressing these limitations has emerged from recent advances in llm capabilities. modern llms demonstrate agentic reasoning capabilities, autonomously planning and utilizing various external tools to solve complex tasks @xcite . this tool-use paradigm has been successfully applied to enhance llms' capabilities across various domains @xcite @xcite , leading to its application in vqa tasks @xcite @xcite . in small object vqa, systems utilizing detection and sliding window tools can identify objects that the core vlm might miss. however, these tool-augmented approaches face a critical challenge: vlms often exhibit direct acceptance of tool outputs without proper verification @xcite , making them vulnerable to false positives and missed detections @xcite . to address this issue, vlms need to not only verify tool outputs but also adaptively refine their approach based on the verification results @xcite . when initial tool outputs prove unreliable, effective re-planning-such as adjusting detection parameters or switching to alternative analysis methods-becomes crucial. while studies demonstrate improved accuracy through iterative refinement with feedback @xcite , current vlms struggle to consistently implement such adaptive reasoning and strategic re-planning @xcite . in this study, we propose vistra (visual tooluse reasoning analyzer), a systematic framework for analyzing and classifying vlms' behavioral patterns in tool utilization, with particular focus on the relationship between vqa accuracy and two critical aspects: direct acceptance of tool outputs and re-planning capabilities. leveraging the v* bench dataset @xcite , which features abundant small objects, we conduct a comprehensive analysis of how vlms process external vision tool outputs and handle potential misrecognitions. the main contributions of this paper are: our analysis reveals that direct acceptance patterns strongly correlate with decreased vqa accuracy, while lower-performing vlms tend to exhibit ineffective re-planning through cyclic verification loops. ## vistra: visual tool-use reasoning analyzer this section introduces and details the visual tool-use reasoning analyzer (vistra), our pro-posed framework for systematically analyzing how visual language models (vlms) utilize external tools and respond to tool errors during agentic reasoning. focusing on vqa tasks involving small objects, we analyze how vlms employ tools such as detection and slidingwindow, and how they handle tool misrecognition (through direct acceptance, verification, or re-planning). our study utilizes the sketchpad framework proposed in visualsketchpad @xcite and evaluates it on v* bench @xcite , a benchmark for small object vqa. the vistra framework records and categorizes each step of a vlm's reasoning process, providing detailed analysis of reasoning transitions. the following subsections detail our three-stage methodology: first, we collect development and validation data through v* bench; second, we establish a systematic tagging system and transition patterns; and finally, we automate the annotation process using llms. through this framework, we aim to provide quantitative insights into how vlms utilize tools and handle potential errors in their reasoning processes, particularly in challenging scenarios involving small object recognition in highresolution images. ## discussion our analysis using vistra revealed two critical insights about visual language models' reasoning capabilities. first, while more capable models like gpt-4.5-preview can effectively utilize direct reasoning paths, strategic re-planning remains valu-able for improving performance in models with similar architectural capabilities. this suggests that both inherent model capacity and reasoning strategy contribute to overall vqa performance, with the relative importance of each factor varying by model scale. our analysis also revealed how training influences tool usage patterns. fine-tuning can dramatically reshape a model's approach to tool selection and verification, even with minimal training data. we observed the emergence of a hierarchical tool usage pattern, where certain tools serve as primary diagnostic instruments while others function as fallback options. this suggests that models develop tool-specific confidence levels that guide their verification strategies. these findings point to several important directions for future research. first, we need more sophisticated frameworks for distinguishing between genuine direct acceptance and internal verification processes, particularly in high-capability models. second, the relationship between model scale and optimal verification strategy deserves further investigation -there may be scale-dependent patterns that could inform more efficient training approaches. finally, understanding how models develop tool-specific confidence levels could lead to more effective training strategies for improving tool use in visual reasoning tasks. ## conclusion this work introduces vistra, a systematic framework for analyzing vlms' tool utilization patterns in small object vqa tasks. through automated reasoning step annotation, we quantitatively demonstrated that direct acceptance of tool outputs correlates with decreased vqa accuracy across all models, while lower-performing vlms exhibit ineffective re-planning through cyclic verification loops. our analysis revealed that more capable models like gpt-4.5-preview show reduced accuracy degradation with direct acceptance, and that fine-tuning can significantly reshape tool preferences and verification patterns, even with minimal training data. these findings provide insights into the relationship between model capacity and reasoning strategies in visual tool utilization.